Senior Data Engineer

Job Summary

We are looking for an experienced Senior Data Engineer with strong expertise in Azure, Databricks, PySpark, SQL, and Azure Data Factory (ADF). The ideal candidate should have hands-on experience designing, developing, and optimizing scalable data pipelines and ETL/ELT workflows, along with integrating data from multiple sources.

Experience with Microsoft Fabric is an added advantage. The candidate should be proficient in data engineering, best practices, performance optimization, and building reliable data solutions to support business and analytical requirements.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, SQL, and Azure Data Factory (ADF).
• Build and optimize ETL/ELT workflows to process large volumes of structured and semi-structured data.
• Develop data transformation and integration solutions to consolidate data from multiple sources.
• Write complex SQL queries and optimize them for performance and efficiency.
• Develop PySpark applications for distributed data processing and large-scale data transformations.
• Work with Azure cloud services to build secure, scalable, and reliable data engineering solutions.
• Implement data validation, quality checks, error handling, and monitoring mechanisms across data pipelines.
• Troubleshoot pipeline failures and resolve data processing and integration issues.
• Optimize Databricks workloads, Spark jobs, and SQL queries to improve performance and manage costs.
• Collaborate with data architects, data analysts, data scientists, and other stakeholders to understand business requirements and deliver effective data solutions.
• Follow best practices for data security, governance, version control, and deployment.
• Explore and implement Microsoft Fabric capabilities for data integration, engineering, and analytics wherever applicable.

Required Skills and Qualifications

• 8-10 years of overall experience in data engineering.
• Strong hands-on experience with Azure Databricks for data processing and transformation.
• Proficiency in PySpark for distributed data processing and large-scale data transformations.
• Strong knowledge of SQL, including complex queries, joins, aggregations, and performance tuning.
• Hands-on experience with Microsoft Azure data services.
• Experience developing and maintaining ETL/ELT pipelines using Azure Data Factory (ADF).
• Strong understanding of data integration concepts and experience integrating data from multiple sources.
• Experience working with data lakes, data warehouses, and scalable data processing architectures.
• Knowledge of data quality, pipeline monitoring, troubleshooting, and performance optimization.
• Strong analytical, problem-solving, and communication skills.

Good to Have

• Hands-on experience with Microsoft Fabric, including Fabric Data Factory, Lakehouse, or other relevant Fabric components.
• Knowledge of Microsoft Fabric integration with Azure Databricks and other Azure data services.
• Experience with Delta Lake and lakehouse architecture.
• Familiarity with CI/CD practices and version control tools such as Git and Azure DevOps.
• Understanding of data governance, access control, and security best practices.