Infrastructure / Platform Engineer (ML, Data)

Responsibilities:

• Productionize research prototypes. Take model code from researchers (PyTorch, CUDA, exotic dependency stacks) and deliver production pipeline stages: reproducible Docker images, pinned GPU/CUDA/cuDNN environments, clean I/O contracts, retries, idempotency, and observability. Real examples of this work from our pipeline:
• A researcher's stereo depth-estimation model with a GPU batch stage with a locked CUDA base image, a standardized S3 input/output layout, and per-clip cost tracking.
• A fused MediaPipe + WiLoR hand-pose prototype with a single orchestrated labelling stage with well-defined intermediate artifacts and failure isolation per clip.
• A monocular depth model + a stereo model and a three-stage fuse pipeline where resolution-alignment invariants are enforced by the platform, not by tribal knowledge.
• Own the orchestration layer. Design and evolve step functions, state machines, AWS Batch compute environments and job queues, Lambda glue, and selective stage re-execution (re-run just one stage across a fleet of clips without redoing everything).
• Own the infrastructure as code. All of it lives in Terraform modules, per-environment stacks, ECR, IAM, and networking. You'll extend and harden this, not click around a console.
• Drive cost efficiency as a first-class feature. Spot capacity strategies, right-sizing GPU instance families, eliminating GPU idle time (we've measured it, we hunt it), storage lifecycle policies on multi-TB S3 datasets, and batching strategies that keep expensive GPUs saturated.
• You should be the person who can say what a pipeline run costs per clip and then make that number go down.
• Make it reliable at scale. Structured logging, metrics, and alerting across stages; dead-letter handling and automatic retries for flaky clips; data-quality gates so bad inputs fail fast and loudly instead of silently poisoning downstream datasets.
• Manage the container fleet. A dozen-plus GPU images with heavy, conflicting ML dependencies. Keep builds fast, images slim, CUDA stacks consistent, and breakage (e. g., an upstream wheel disappearing from an index) fixed within hours, not weeks.
• Move fast with researchers. Sit close to the research loop: prototype, deploy to staging, run on real fleet data, iterate on feedback, and promote to production.

Requirements:

• 5+ years in infrastructure/platform with real ownership of production systems.
• Strong Python. Not just scripting: you write clean, tested, maintainable pipeline and tooling code that other engineers build on.
• Deep AWS experience in compute, networking, IAM, and storage, and strong opinions about cost.
• A track record of taking rough prototypes (ideally ML/research code) to production.
• Hands-on Docker/containerization depth: you debug CUDA base-image conflicts and dependency hell without flinching; ECS/EKS or other orchestration experience; Terraform (or equivalent IaC) is used seriously, in a team, across environments.
• Working GPU knowledge: what saturates a GPU, what leaves it idle, and how instance choice and batching change the bill.
• Cost-optimization instinct: spot strategies, right-sizing, storage tiering, and the discipline to measure before and after.
• Systems thinking and bias to ship: you'd rather run it on real data today and iterate than perfect it in isolation.

Our Stack:

• Languages: Python (primary; you must be genuinely strong here), Bash; Go/Rust a plus.
• Orchestration: AWS Step Functions, AWS Batch, Lambda.
• Containers: Docker, ECR; multi-stage GPU image builds; container orchestration concepts (ECS/EKS experience welcome).
• IaC: Terraform (modules, multi-environment).
• GPU/ML runtime: NVIDIA CUDA/cuDNN, PyTorch deployment environments, GPU instance families on AWS, spot vs. on-demand economics.
• Data: S3 at multi-TB scale, structured artifact layouts, dataset versioning, high-throughput transfer.
• Observability: CloudWatch logs/metrics/alarms, cost attribution, and reporting.

Nice to Have:

• Experience with ML labelling/inference pipelines, video, or multimodal sensor data.
• Exposure to computer vision workloads (depth estimation, pose estimation, SLAM/VIO).
• EKS/Kubernetes at scale; Ray or other distributed-compute frameworks.
• CI/CD for container-heavy repos (CodeBuild, GitHub Actions).