Azure Data Engineer

Position: Data Engineer III

Position: Data Engineer III
Type: Contract to Hire (C2H)

Location: Bangalore
Years of Experience: 58 Years
Shift Timings: 01:30 PM 10:30 PM IST

Job Description

We are looking for an experienced Data Engineer III to design, develop, and maintain scalable and reliable data pipelines and ETL/ELT processes using Microsoft Azure data services. The ideal candidate should have strong hands-on experience with Azure Databricks, Python, SQL, Spark/PySpark, and Azure data engineering services.

The candidate will work closely with data analysts, data scientists, architects, and business stakeholders to understand data requirements and deliver high-quality, scalable data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes.
• Develop and optimize data engineering solutions using Azure Databricks and Apache Spark/PySpark.
• Work with Python and SQL to perform data transformation, processing, validation, and analysis.
• Build and maintain data pipelines using Azure services such as Azure Data Factory, Azure Synapse, Azure Databricks, and Azure Functions.
• Work with Azure storage services including Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Blob Storage.
• Process and integrate structured, semi-structured, and unstructured data from various source systems and formats.
• Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
• Implement data validation and quality checks to ensure data accuracy and integrity.
• Monitor, troubleshoot, and resolve issues related to data pipelines and data processing.
• Collaborate with data analysts, data scientists, data architects, and business stakeholders to understand requirements and deliver data solutions.
• Develop and maintain technical documentation for data pipelines, processes, configurations, and best practices.
• Support CI/CD and DevOps practices for deploying and maintaining data engineering solutions.
• Work with Git and Azure DevOps for source control, pipeline management, and deployment.
• Contribute to continuous improvement of data engineering standards, processes, and best practices.

Required Skills

• 5–8 years of experience in Data Engineering or a similar role.
• Strong hands-on experience with Azure Databricks.
• Strong proficiency in Python and SQL.
• Hands-on experience with Apache Spark/PySpark and big data processing.
• Strong understanding and implementation experience of ETL/ELT processes.
• Experience with Azure Data Factory (ADF).
• Experience with Azure Synapse Analytics or similar Azure data services.
• Experience working with ADLS Gen2 / Azure Blob Storage.
• Knowledge of structured, semi-structured, and unstructured data.
• Experience developing and optimizing scalable data pipelines.
• Strong troubleshooting and debugging skills.
• Understanding of Git, Azure DevOps, CI/CD, and software development practices.
• Strong problem-solving and analytical skills.
• Good communication and collaboration skills.

Education

• Bachelor’s degree in computer science, Engineering, Information Technology, or a related field, or equivalent experience

Good to Have

• Experience with Azure Functions, Logic Apps, or other Azure integration services.
• Experience with Delta Lake, Parquet, Avro, or other modern data formats.
• Experience working with API-based data sources.
• Knowledge of Azure cloud architecture and services.
• Experience with automated testing and deployment.
• Experience working with data quality and validation frameworks.
• Experience working with stakeholders and cross-functional teams.
• Technical mentoring or leadership experience.

.