Data Engineer

What you will do

• Build extraction from our transactional systems and our Athena lake; land, transform, and write the curated Parquet datasets
• Implement the de-identification transforms, with tests that assert on the handling class of every single column - not just that the job completed
• Own orchestration, scheduling, retries, backfills, alerting and data-quality checks
• Contribute CI/CD and infrastructure as code alongside our platform team
• Populate catalog and lineage, and document every dataset the pipeline produces
• Support what you build in production, including handling schema drift in upstream systems

What we are looking for

• Strong Python and SQL, and distributed processing with Spark or an equivalent
• You have built and operated batch pipelines in production on AWS - not just authored them, but been on the hook when they broke
• Orchestration with Airflow, Dagster, Step Functions or similar, including what you actually do when a run fails at three in the morning
• Real software engineering discipline: version control, code review, automated tests, and an opinion about idempotency and backfills that came from getting it wrong once
• Comfort working from a written technical standard rather than from a conversation
• For the senior role: you have mentored engineers, owned a system end to end including its on-call, and you will push back on a design you think is wrong

Nice to have

• Change data capture tooling such as Debezium or AWS DMS
• Iceberg, Delta Lake or Hudi in anger
• Any exposure to data masking, tokenization or privacy tooling
• dbt or a comparable transformation framework
• Having built datasets that fed model training, and knowing how that consumer differs from a reporting one