Lead I - Azure Data Engineer

Role Description Azure Data Engineer

Job Summary The Azure Data Engineer is responsible for designing, developing, and maintaining scalable data pipelines and analytics solutions on Microsoft Azure. The role focuses on building robust ETL/ELT processes, integrating data from multiple sources, optimizing data processing using Azure Databricks, and delivering reliable, high-performance data platforms for analytics and business intelligence. Key Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory.
• Build scalable data processing solutions using Azure Databricks and Apache Spark.
• Develop data transformation workflows using PySpark, Spark SQL, and SQL.
• Ingest data from databases, APIs, cloud storage, and flat files into Azure data platforms.
• Integrate Azure services including Azure Data Lake Storage, Azure Synapse Analytics, and Azure SQL Database.
• Optimize Spark jobs, SQL queries, and data processing workflows for performance and cost efficiency.
• Implement data quality checks, monitoring, logging, and error-handling mechanisms.
• Collaborate with data architects, business stakeholders, and engineering teams to design and deliver scalable data solutions.
• Implement CI/CD pipelines and deployment automation for data engineering workloads.
• Ensure data security, governance, and compliance with enterprise standards. Must Have Skills
• Strong hands-on experience with Azure Data Factory (ADF).
• Strong expertise in Azure Databricks.
• Experience with Apache Spark, PySpark, and Spark SQL.
• Strong SQL development and query optimization skills.
• Proficiency in Python for data engineering.
• Experience with Azure Cloud services.
• Hands-on experience with Azure Data Lake Storage, Azure Synapse Analytics, and Azure SQL Database.
• Knowledge of ETL/ELT concepts and modern data warehousing.
• Experience working with large-scale structured and unstructured datasets.
• Understanding of Delta Lake architecture.
• Experience with Git-based version control.
• Familiarity with REST APIs and enterprise data integration patterns.
• Strong experience in Spark performance tuning and optimization. Good to Have Skills
• Experience with Azure DevOps, GitHub Actions, or CI/CD pipelines.
• Knowledge of Azure Functions, Logic Apps, Event Hub, or Service Bus.
• Experience with Data Lakehouse architecture.
• Familiarity with monitoring and logging tools such as Azure Monitor or Log Analytics.
• Knowledge of data governance, security, and compliance best practices.
• Experience working in Agile/Scrum development environments.
• Microsoft Certified: Azure Data Engineer Associate certification is an added advantage.
• Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent experience.

Skills azure data factory,sql,databricks,python,performance tuning,