Lead Engineer — Data Platform & Acquisition

We are looking for Data Engineer.

Key Responsibilities

▪   Architect high-throughput pipelines processing petabytes of text and multimodal data.

▪   Implement deduplication, quality filtering and benchmark decontamination protocols.

▪   Lead licensing negotiations for proprietary datasets.

▪   Manage external vendors for large-scale digitisation and crowdsourcing.

▪   Own data provenance and the audit trail behind every corpus entering a training run.

Essential Skills & Experience

▪   Experience in data engineering or data acquisition at scale.

▪   Proven ownership of large-scale data infrastructure — Spark, Ray, Airflow, Kafka or equivalent.

▪   Practical deduplication and data-quality work on very large corpora.

▪   Strong vendor management and commercial negotiation ability.

▪   Python at production depth.

Preferred

▪   Web crawling and corpus construction at petabyte scale.

▪   Experience negotiating dataset licensing agreements.