Data Engineer

Requirements:

• 4+ years in a data engineering role with end-to-end pipeline ownership.
• Strong Python async patterns, subprocess management, API clients, and data processing at scale.
• Hands-on AWS Athena, Glue, S3, DynamoDB, and Lake Formation; production-grade, not just familiarity.
• Apache Hudi or Delta Lake schema evolution, partition strategies, and upsert patterns.
• SQL proficiency: able to write and optimize complex analytical queries.
• Experience with Airflow or an equivalent workflow orchestrator.
• Demonstrated experience with large-scale media data pipelines, audio/video format conversion, metadata extraction, and chunking, e. g., egocentric or multimodal datasets (hard requirement, not a plus).
• Understanding of how data format and I/O design affect downstream GPU compute workloads (WebDataset, sharded Parquet, tfrecord, or equivalent).

Nice to Have:

• Direct experience designing data handoff for GPU clusters (AWS Batch GPU instances, Ray, or SLURM).
• Familiarity with ML training data formats and dataset standards used by AI labs (Hugging Face datasets, WebDataset, dataset cards).
• Experience with rclone, large-scale file transfer, or cloud-to-cloud sync pipelines.
• Exposure to data lineage or provenance tooling (OpenLineage, DataHub, or custom metadata schemas).