Technical Lead-Data Engg

Key Responsibilities
-
- Design and develop scalable ETL/ELT pipelines using PySpark, Spark SQL, Python, and Databricks.
- Develop complex transformation logic across Bronze, Silver, and Gold layers.
- Implement reusable transformation frameworks and common data processing utilities.
- Optimize transformation workloads for performance, scalability, and cost.
- Apply data quality, validation, and business-rule checks throughout the pipeline.
- Enterprise Data Modelling
- Design scalable data models using:
- Dimensional modelling
- Star/Snowflake schemas
- Data Vault 2.0
- Canonical data models
- Develop curated data products and analytical datasets using Databricks SQL and Delta Lake.
- Support enterprise-wide data harmonization across clinical, regulatory, safety, and commercial domains.
- Experience supporting data platforms across:
- Clinical trials - EDC, CDMS, CTMS
- Regulatory and submission systems
- Pharmacovigilance and safety
- Commercial analytics
- Real-World Evidence (RWE)
- Ensure adherence to relevant regulatory and data governance requirements including GxP, 21 CFR Part 11, HIPAA/GDPR, and ALCOA+ principles.
Required Qualifications
- 10-15 years of experience in Data Engineering and enterprise data platforms.
- Strong hands-on experience with Databricks, Apache Spark, PySpark, Python, and SQL.
- 4+ years of experience with Databricks/Spark-based data engineering preferred.
- Strong experience in:
- Data migration and modernization
- ETL/ELT development
- Batch and streaming data pipelines
- Delta Lake
- Databricks Workflows
- Unity Catalog
- Data modelling
- Data quality and reconciliation
- Experience with cloud platforms - AWS/Azure/GCP.
- Strong understanding of CI/CD, Git, Jenkins, and DevOps practices.
- Experience working in Life Sciences/Pharma environments preferred.