Data Engineer

Role & responsibilities

About the Role

We are looking for a skilled Data Engineer who can design and maintain robust data pipelines, work with large-scale structured and semi-structured pharma datasets, and collaborate closely with Data Scientists and Python Developers to deliver clean, validated, production-grade data infrastructure.

Pipeline Design & Development

• Design and build scalable ETL/ELT pipelines for ingesting, transforming, and loading pharmaceutical data from multiple sources
• Process indication-level market forecast data from Excel workbooks into structured JSON and DB-ready formats for frontend visualisation
• Build and maintain data extraction pipelines from clinical trial registries, conference abstracts, and drug databases
• Implement multi-step validation workflows to ensure pipeline output accuracy and completeness before downstream handoff

Data Modelling & Architecture

• Design normalised relational schemas in SQL Server to support pharmaceutical analytics use cases
• Collaborate with Data Scientists on data preparation and feature engineering pipelines
• Maintain data dictionaries, schema documentation, and lineage records for all active pipelines
• Contribute to architecture decisions on data storage, processing strategies, and API integrations

Quality, Validation & Reliability

• Own data quality across assigned pipelines implement automated checks, alerting, and audit trails
• Investigate and resolve silent data errors including missing fields, incorrect labels, zero-value nodes, and schema mismatches
• Write thorough unit and integration tests for all pipeline components

Collaboration & Standards

• Work within an agile delivery framework participate in sprint planning, estimation, reviews, and retrospectives
• Follow and contribute to team-wide GitHub standards: branching strategy, PR reviews, naming conventions, and documentation
• Actively use AI coding tools (GitHub Copilot, Claude, etc.) as part of everyday development
• Produce clear technical documentation and handoff notes for every delivery

Preferred candidate profile

• Python (3+ years, production pipelines)
• SQL Server (primary)
• SQL complex queries, stored procedures, optimisation
• Git / GitHub (branching, PRs, code review)
• ETL/ELT design and implementation
• Pandas, SQLAlchemy, Pydantic
• Data modelling and schema design
• Excel / openpyxl for source data handling
• REST API integration and data extraction
• Docker or containerised environments (desirable)
• Data validation and testing practices
• Selenium / web scraping tools (desirable)
• JSON / XML data processing
• Airflow or equivalent orchestration (desirable)