Data Engineer (SQL Python PySpark ETL Azure Da)

We are seeking a skilled Data Engineer with strong expertise in SQL, Python, PySpark, ETL development, and Azure Databricks to design, build, and optimise scalable data pipelines. The ideal candidate should have hands-on experience working with pharmaceutical commercial datasets, including claims, prescription (Rx), patient, sales, and payer data, and be proficient in processing large-scale healthcare data to support analytics and business intelligence initiatives.

Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, and SQL.
• Build and optimise data processing workflows in Azure Databricks.
• Develop and maintain data ingestion frameworks from multiple structured and unstructured data sources.
• Process and transform pharmaceutical commercial datasets, including claims, prescription, payer, sales, formulary, and patient-level data.
• Optimise Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, validation rules, and monitoring processes.
• Collaborate with business analysts, data scientists, and commercial analytics teams to deliver reliable datasets.
• Develop reusable data engineering components and automation frameworks.
• Troubleshoot data pipeline issues and ensure high availability of data platforms.
• Participate in code reviews and follow best practices for version control and CI/CD.
• Ensure compliance with healthcare data governance, privacy, and security standards.

Requirements:

• Strong SQL programming and query optimisation skills.
• Proficiency in Python for data engineering and automation.
• Hands-on experience with PySpark and Apache Spark.
• Experience developing ETL/ELT pipelines.
• Strong experience with Azure Databricks.
• Experience with Azure Data Lake Storage (ADLS).
• Knowledge of Azure Data Factory (ADF) for orchestration.
• Familiarity with Delta Lake, Parquet, and other big data storage formats.
• Experience with Git or Azure DevOps for source control.
• Good understanding of data modelling and data warehousing concepts.
• Strong analytical and p.