Azure Data Engineer

Position Overview

We are looking for a skilled Azure Databricks and Python Developer to design, develop, and maintain scalable data engineering solutions on Microsoft Azure. The ideal candidate will have strong experience with Azure Databricks, Python, Spark, data pipelines, and cloud-based data platforms.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks, Apache Spark, and Python.
• Build and optimize ETL/ELT workflows for structured and unstructured data.
• Develop PySpark notebooks, data transformations, and reusable data processing frameworks.
• Integrate data from various sources, including databases, APIs, files, and cloud storage.
• Work with Azure services such as:
• Azure Data Lake Storage Gen2
• Azure Data Factory
• Azure Synapse Analytics
• Azure Key Vault
• Azure Functions
• Implement data quality checks, validation rules, and error-handling mechanisms.
• Optimize Spark jobs, Databricks clusters, SQL queries, and data processing performance.
• Implement Delta Lake solutions, including schema evolution, partitioning, and optimization.
• Collaborate with data architects, analysts, software engineers, and business stakeholders.
• Participate in code reviews, unit testing, deployment, and production support.
• Follow best practices for security, governance, monitoring, and access management.
• Maintain technical documentation related to data pipelines, processes, and architecture.

Required Skills and Qualifications

• Bachelors degree in Computer Science, Information Technology, Engineering, or a related field.
• Strong hands-on experience with Azure Databricks.
• Strong programming skills in Python.
• Good experience with PySpark and Apache Spark.
• Proficiency in SQL and relational database concepts.
• Experience developing data pipelines using Azure Data Factory or similar orchestration tools.
• Experience working with Azure Data Lake Storage Gen2.
• Strong understanding of data warehousing, data modeling, and ETL/ELT concepts.
• Experience with Delta Lake and Lakehouse architecture.
• Knowledge of Git-based version control and CI/CD practices.
• Strong analytical, problem-solving, and communication skills.

Preferred Skills

• Experience with Azure DevOps and Databricks Repos.
• Knowledge of Terraform or other infrastructure-as-code tools.
• Experience with CI/CD implementation for Databricks notebooks and jobs.
• Familiarity with Azure security, networking, identity, and access management.
• Experience with streaming technologies such as Kafka, Azure Event Hubs, or Structured Streaming.
• Knowledge of Power BI and reporting data models.
• Familiarity with data governance and cataloging tools such as Unity Catalog.
• Relevant Azure or Databricks certifications.

Experience

• 3–8 years of experience in data engineering or a related role.
• Minimum 2 years of hands-on experience with Azure Databricks and Python.