Module Lead - Azure Databricks

5-8 Years

• We are seeking an experienced Azure AI Data Engineer with 5 to 8 years of hands-on experience in designing, developing, and maintaining scalable data and AI solutions on Microsoft Azure. The ideal candidate should possess strong expertise in data engineering, Azure cloud services, AI/ML integration, data pipelines, and modern analytics platforms to support business intelligence and AI-driven applications.

  Key Responsibilities

  • Design, develop, and maintain scalable data pipelines using Azure services.
  • Build and optimize ETL/ELT workflows for structured and unstructured data.
  • Implement data ingestion solutions using Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.
  • Develop and manage data lakes and data warehouse solutions on Azure.
  • Integrate AI and Machine Learning services such as Azure AI Services, Azure OpenAI, Azure Machine Learning, and Cognitive Services into enterprise applications.
  • Create and maintain data models, transformations, and processing frameworks.
  • Implement data governance, data quality, security, and compliance controls.
  • Monitor and optimize data platform performance, scalability, and cost.
  • Collaborate with Data Scientists, AI Engineers, Business Analysts, and Application Development teams.
  • Develop reusable frameworks and automation for data orchestration and deployment.
  • Support CI/CD implementation using Azure DevOps and Infrastructure as Code (IaC).
  • Troubleshoot data-related issues and provide production support.

  Azure AI & Machine Learning

  • Azure AI Services
  • Azure OpenAI Service
  • Azure Machine Learning
  • Azure Cognitive Services
  • Generative AI concepts and implementation

  Required Qualifications

  • Bachelors or Masters degree in Computer Science, Information Technology, Data Science, or related field.
  • 5 to 8 years of experience in Data Engineering and Azure Cloud technologies.
  • Strong experience with Azure Data Factory, Databricks, Synapse Analytics, and Data Lake.
  • Experience working with large-scale structured and semi-structured datasets.
  • Knowledge of AI/ML integration and Generative AI solutions on Azure.
  • Hands-on experience with Python, SQL, and Spark.
  • Strong understanding of cloud architecture, security, and performance optimization.
  • Experience with Agile development methodologies.

  Preferred Qualifications

  • Microsoft Azure Certifications such as:
    • DP-203: Azure Data Engineer Associate
    • AI-102: Azure AI Engineer Associate
    • AZ-305: Azure Solutions Architect Expert
  • Experience with Microsoft Fabric.
  • Experience in Retrieval-Augmented Generation (RAG) architectures.
  • Knowledge of Vector Databases and Semantic Search.
  • Experience integrating Azure OpenAI with enterprise applications.