Director of Data Engineering

The Director of Data Engineering will architect and lead the development of NeuroDiscovery's core data platform, powering our NeuroLLM and RWE capabilities. The candidate will design petabyte-scale data pipelines that ingest, process, and serve multimodal healthcare data while ensuring HIPAA/GDPR compliance, data quality, and real-time analytics capabilities. This is a hands-on leadership role reporting to the CTO/CEO, overseeing a team of 20+ data engineers while personally architecting mission-critical systems.

The candidate will have responsibilities across the following functions:

Strategic Leadership:

• Define and execute NeuroDiscovery's 3-year data engineering roadmap aligned with clinical research and RWE business goals.
• Partner with Chief Scientific Officer, VP AI/ML, and clinical partners to translate research requirements into scalable data architectures.
• Build and mentor a high-performing data engineering team (hiring, coaching, career development).
• Establish data engineering best practices, CI/CD pipelines, and observability standards.

Technical Architecture:

• Design petabyte-scale data platforms handling multimodal healthcare data (MRI/CT, EHR, genomics, claims, clinical notes, PACS/RIS).
• Architect real-time data pipelines for extracting EHR and Claims data to start with, and then PACS, EEG, EMG and OMICS data.
• Lead HIPAA-compliant data ingestion from US provider practices (EHR systems: Epic, Cerner, athenahealth, Veradigm, eClinicalWorks, etc).
• Build observability-driven data quality frameworks with lineage, freshness, and schema validation.
• Implement federated learning infrastructure for privacy-preserving AI model training across partner datasets.
• Design healthcare data integration across disparate systems (EHR, imaging, genomics) with standardised data models.

Execution Excellence:

• Deliver production data pipelines processing 100TB+ daily with 99.99% uptime.
• Lead cloud native infrastructure migration (AWS, Azure, GCP) with Kubernetes orchestration.
• Implement advanced data orchestration using (Airflow, Prefect) for complex ML/RWE workflows.
• Drive cost optimisation while scaling infrastructure 10x annually.

Requirements:

• 12+ years in data engineering, with 5+ years leading engineering teams at healthtech companies.
• Mandatory 5+ years of hands-on experience with EHR/EMR ingestion, transformation, and engineering.
• Proven track record building petabyte-scale data platforms for healthcare/ML use cases.
• Deep expertise in Python, Spark, SQL optimisation, and healthcare data standards (HL7 DICOM, FHIR, ICD-10/11 HCPCS).
• Experience with HIPAA/GDPR compliant data pipelines and federated learning architectures.
• Hands-on architect of production ML data platforms (experience with similar healthcare LLMs a plus).
• Cloud platform mastery (AWS, Azure, GCP; Kubernetes, Terraform).
• Orchestration expertise (Airflow, dbt, Prefect) at enterprise scale.
• Team-building success: scaled data engineering teams to 520+ people.

Technical Must Haves:

• Data Processing: Spark/Dask/Ray (distributed processing at PB scale).
• Data Integration and ETL: Talend or equivalent tool (core requirement), healthcare data pipeline design.
• Cloud: AWS (EC2 S3 RDS, EMR, Glue), Azure Data Lake, GCP BigQuery/Dataflow, Kubernetes.
• Healthcare Standards and Systems: HL7/DICOM (PACS/RIS), FHIR/OMOP, EHR system integration (Epic, Cerner, athenahealth), imaging data handling (CT/MRI), genomics data.
• Search and Analytics: Elasticsearch for clinical data indexing and querying.
• Orchestration: Apache Airflow (primary), dbt, Prefect.
• ML Infrastructure: Feature stores (Feast), model serving (Seldon/TFX), MLOps, vector databases, knowledge graphs.
• Observability: Data lineage (OpenLineage), data quality (Great Expectations), monitoring (Datadog/Prometheus).

Preferred Experience:

• Domain expertise in neurology clinical research, RWE generation, or clinical trial matching.
• Multimodal data experience (MRI/CT processing with DICOM, genomics/OMICS, time-series EHR, text data from clinical notes).
• Leadership in Talend or similar ETL platforms at scale.
• Vector databases and knowledge graph architecture (supporting NeuroLLM integration).
• PACS/RIS system integration and healthcare imaging workflows.
• SQL AST/parsing experience (sqlglot, column-level lineage).
• Contributed to open-source data tools or healthcare data standards.