MLOps Engineer

Responsibilities:

• Productionize research prototypes. Take model code from researchers (PyTorch, CUDA, exotic dependency stacks) and deliver production pipeline stages: reproducible Docker images, pinned GPU/CUDA/cuDNN environments, clean I/O contracts, retries, idempotency, and observability. Real examples of this work from our pipeline:
• A researcher's stereo depth-estimation model: a GPU Batch stage with a locked CUDA base image, standardised S3 input/output layout, and per-clip cost tracking.
• A fused MediaPipe + WiLoR hand-pose prototype: a single orchestrated labelling stage with well-defined intermediate artefacts and failure-isolation per clip.
• A monocular depth model + a stereo model: a three-stage fuse pipeline where resolution-alignment invariants are enforced by the platform, not by tribal knowledge.
• Own the orchestration layer. Design and evolve Step Functions state machines, AWS Batch compute environments and job queues, Lambda glue, and selective stage re-execution (re-run just one stage across a fleet of clips without redoing everything).
• Own the infrastructure as code. All of it lives in Terraform modules, per-environment stacks, ECR, IAM, networking. You'll extend and harden this, not click around a console.
• Drive cost efficiency as a first-class feature. Spot capacity strategies, right-sizing GPU instance families, eliminating GPU idle time (we've measured it, we hunt it), storage lifecycle policies on multi-TB S3 datasets, batching strategies that keep expensive GPUs saturated.
• You should be the person who can say what a pipeline run costs per clip and then make that number go down.
• Make it reliable at scale. Structured logging, metrics, and alerting across stages; dead-letter handling and automatic retries for flaky clips; data-quality gates so bad inputs fail fast and loudly instead of silently poisoning downstream datasets.
• Manage the container fleet. A dozen-plus GPU images with heavy, conflicting ML dependencies.
• Keep builds fast, images slim, CUDA stacks consistent, and breakage (e. g., an upstream wheel disappearing from an index) fixed within hours, not weeks.
• Move fast with researchers. Sit close to the research loop: prototype, deploy to staging, run on real fleet data, iterate on feedback, promote to production.

Requirements:

• 5+ years in infrastructure/platform with real ownership of production systems.
• Strong Python. Not just scripting: you write clean, tested, maintainable pipeline and tooling code that other engineers build on.
• Deep AWS experience: compute, networking, IAM, storage and strong opinions about cost.
• A track record of taking rough prototypes (ideally ML/research code) to production.
• Hands-on Docker/containerization depth: you debug CUDA base-image conflicts and dependency hell without flinching; ECS/EKS or other orchestration experience.
• Terraform (or equivalent IaC) used seriously, in a team, across environments.
• Working GPU knowledge: what saturates a GPU, what leaves it idle, how instance choice and batching change the bill.
• Cost-optimisation instinct: spot strategies, right-sizing, storage tiering, and the discipline to measure before and after.
• Systems thinking and bias to ship: you'd rather run it on real data today and iterate than perfect it in isolation.

Nice to Have:

• Experience with ML labelling/inference pipelines, video or multimodal sensor data.
• Exposure to computer-vision workloads (depth estimation, pose estimation, SLAM/VIO).
• EKS/Kubernetes at scale; Ray or other distributed-compute frameworks.
• CI/CD for container-heavy repos (CodeBuild, GitHub Actions).

Our Stack:

• Languages: Python (primary; you must be genuinely strong here), Bash; Go/Rust a plus.
• Orchestration: AWS Step Functions, AWS Batch, Lambda.
• Containers: Docker, ECR; multi-stage GPU image builds; container orchestration concepts (ECS/EKS experience welcome); IaC: Terraform (modules, multi-environment).
• GPU/ML runtime: NVIDIA CUDA/cuDNN, PyTorch deployment environments, GPU instance families on AWS, spot vs. on-demand economics.
• Data: S3 at multi-TB scale, structured artefact layouts, dataset versioning, high-throughput transfer.
• Observability: CloudWatch logs/metrics/alarms, cost attribution and reporting.