Azure Data Engineer

Key Responsibilities:

Data Engineering

• Design and build production-grade data pipelines using Azure Data Factory and Azure Databricks.
• Develop scalable data architectures leveraging Azure SQL, Data Lake, and modern data modelling techniques (star/snowflake schemas).
• Implement data quality validation, reconciliation processes, and resolve operational issues.

Analytics & AI Enablement

• Use Python for data processing, transformation, and analytics.
• Develop and maintain Jupyter Notebooks for exploration and experimentation.
• Utilize MLflow for experiment tracking and model lifecycle management.

BI Enablement

• Deliver curated, analytics-ready datasets for Power BI and other reporting platforms.
• Optimize data models and structures for efficient reporting, dashboarding, and analytics consumption.

Engineering Practices

• Implement CI/CD pipelines for data engineering workflows.
• Ensure secure data handling practices, including sensitive data protection.
• Monitor, troubleshoot, and resolve pipeline failures and performance issues.

Required Skills & Qualifications

Strong expertise in:

• Azure Data Factory
• Azure Databricks
• Azure SQL and Data Lake
• Solid understanding of data modelling (star and snowflake schemas).
• Experience building scalable and reliable data pipelines.
• Core Competencies
• Strong problem-solving and analytical skills.
• Ability to manage production-grade data environments.
• Focus on data quality, scalability, and performance optimization.
• Nice to Have
• Experience working in regulated data environments.
• Exposure to Azure OpenAI or AI/ML use cases.
• Knowledge of prompt engineering, evaluation frameworks, and experimentation techniques.