AI Data Engineer

Roles and Responsibilities

• Architect and scale enterprise lakehouse solutions using Azure Databricks, ADLS Gen2, Delta Lake, and medallion architecture (Bronze, Silver, Gold).
• Design, build, and maintain production-grade batch and streaming ELT/ETL pipelines using PySpark, Spark SQL, Azure Data Factory, and Databricks Workflows.
• Develop configuration-driven, metadata-based ingestion frameworks to rapidly onboard structured, semi-structured, and unstructured data sources.
• Curate and optimize gold-layer data products, vector search indexes, and embeddings via Azure AI Search to support Retrieval-Augmented Generation (RAG) and AI agents.
• Enforce end-to-end data governance, access management, and data lineage utilizing Databricks Unity Catalog and Microsoft Purview to ensure strict compliance (Section 7216, PCAOB, NIST AI RMF).
• Implement performance tuning, Liquid Clustering, Z-Ordering, file compaction, and right-sized compute strategies to minimize platform compute costs.
• Establish CI/CD pipelines, automated data quality testing, data contracts, and proactive observability (freshness, volume, schema drift).
• Collaborate with AI developers, architects, and cross-functional teams to integrate point solutions into shared, scalable platform services.

Preferred Candidate Profile

Experience

• 4+ years of hands-on experience building production data pipelines and data platforms.
• Proven track record designing lakehouse architectures on Azure cloud infrastructure.

Technical Expertise

• Core Cloud/Data: Azure Databricks, ADLS Gen2, Azure Data Factory, Synapse.
• Languages & Querying: Strong proficiency in Python, PySpark, Spark SQL, and SQL.
• Storage Formats: Delta Lake, Parquet, Avro, JSON.
• AI Grounding: Vector Search, RAG Pipelines, Azure AI Search, Embeddings, Chunking.
• Governance & DevOps: Databricks Unity Catalog, Microsoft Purview, CI/CD (Azure DevOps/GitHub Actions), Infrastructure-as-Code (Terraform/Bicep).

Mandatory : Candidate must be willing to relocate near to Goregaon, Mumbai Location.