Data Engineer

Job Summary:

The Data Engineer will design, build and support reliable data pipelines and lakehouse solutions on Microsoft Fabric and Azure. The role will integrate ERP, database and file-based sources; prepare governed data for analytics; and use practical DevOps methods to release changes safely across environments. The successful candidate will work with platform, architecture, BI and business teams to deliver reusable, monitored data products

Role & responsibilities :

• Design and develop ingestion and transformation pipelines in Microsoft Fabric using Data Factory pipelines, Lakehouse, OneLake, Delta tables, notebooks and Spark. • Integrate SQL Server, Oracle/ERP, APIs and file-based sources using suitable Azure services such as Azure Data Factory, ADLS Gen2 and Azure Key Vault. • Implement batch and incremental loads, watermark handling, schema evolution, reconciliation, error recovery and appropriate raw-data history. • Develop Bronze, Silver and Gold datasets with clear data contracts, business keys, dimensional models and documented lineage. • Write and tune SQL and PySpark transformations; monitor pipeline health, investigate failures and improve performance and capacity use. • Apply access controls, secure credentials and data quality checks in collaboration with platform, security and business owners. • Use Git, pull requests, code reviews and CI/CD pipelines to deploy Fabric and Azure data assets through development, test and production environments. • Create technical documentation, operational runbooks and automated tests; support SIT, UAT, production releases and incident resolution. • Collaborate with Power BI developers on curated datasets and semantic-model requirements.

Preferred candidate profile

Basic Qualifications: • Bachelors degree in computer science, engineering, information systems or a related discipline, or equivalent practical experience. • 45 years of hands-on experience in data engineering, including substantive delivery using Microsoft Fabric and Azure data services.

Knowledge/Skills/Abilities: • Strong SQL, Python and PySpark skills; practical experience with data modelling, Delta Lake and distributed processing. • Hands-on Microsoft Fabric experience with Lakehouse, OneLake, pipelines, notebooks and Warehouse; familiarity with Dataflows Gen2 and Power BI integration. • Hands-on Azure experience with ADLS Gen2, Azure Data Factory, Key Vault and identity/access patterns; experience with on-premises connectivity is beneficial. • Working knowledge of Git branching, pull requests, Azure DevOps or equivalent CI/CD, environment configuration and automated deployment. • Experience with incremental ingestion, orchestration, monitoring, data quality and performance tuning. • Clear communication, troubleshooting and documentation skills; ability to work with cross functional teams. Essential Physical Demands/Work Environment: • Comfortable working in a cross-cultural, matrix and agile delivery environment.