Data Engineer

Education:
- Bachelor's degree in Computer Science, Computer Engineering, Data Engineering, or a relevant field.

Required Experience:
- 5-8+ years of solid understanding in data analytics concepts (data ingestion, data warehouse, data lakes, reporting, etc.).
- 5+ years of experience delivering production ETL pipelines.
- 2+ years of hands-on experience with Databricks (Azure Databricks preferred).
- Built, optimized, and maintained ETL pipelines using Databricks for large-scale data processing.
- Strong experience with PySpark and SQL; proficiency in Python.
- Strong experience in Azure cloud and data platform services including Data Factory, Databricks, databases, ETL/ELT concepts, IAM, and security controls.
- Strong understanding of cloud technologies, data management, big data solutions, machine learning, and AI frameworks.
- Experience with ADF for orchestration and operational monitoring.
- Experience implementing CI/CD practices using GitHub Actions.
- Knowledge/experience in DevOps and Agile methodologies.
- Collaborate with cross teams, infrastructure, and security teams for best and timely delivery.

Preferred Qualifications:
- Experience with Spark/Delta optimization (partitioning, file sizing, performance tuning).
- Experience in healthcare/Life Sciences is preferred

Technical Skills:
- Azure Databricks (workspace management, cluster configuration, notebooks, jobs)
- PySpark and Spark SQL for large-scale data processing
- Python for scripting, automation, and data engineering tasks
- Unity Catalog for data governance, access controls, and asset organization
- Azure Data Factory (ADF) for pipeline orchestration, scheduling, and monitoring
- Lakehouse architecture patterns (Bronze/Silver/Gold data layers)
- Delta Lake format and optimization techniques
- CI/CD pipelines using GitHub Actions
- Git-based version control and collaborative development workflows
- Strong SQL skills for data transformation and validation
- Knowledge of data quality and testing frameworks
- Knowledge of DevOps and Agile methodologies

Knowledge, Skills & Abilities:
- Strong analytics and problem-solving skills with experience across a broad spectrum of analytics
- Strong written and verbal communication skills
- Strong team player working on cross-functional deliverables