Senior Data Engineer

Roles and Responsibilities

• Design, construct, and manage scalable Lakehouse architecture on ADLS Gen2 and Azure Databricks following medallion pattern (Bronze, Silver, Gold).
• Develop metadata-driven ETL and ELT ingestion pipelines using Azure Data Factory, Databricks Workflows, and Structured Streaming.
• Create vector-ready data products and embeddings via Azure AI Search to enable Retrieval-Augmented Generation (RAG) and LLM applications.
• Standardize open table formats on Delta Lake while optimizing performance using partitioning, Z-Ordering, liquid clustering, and file compaction.
• Implement automated data governance, cataloging, lineage tracking, and role-based access control using Unity Catalog and Microsoft Purview.
• Ensure strict adherence to regulatory standards, data privacy, and security frameworks including PCAOB, NIST AI RMF, and PII/PHI protection.
• Optimize platform compute, storage, and pipeline execution costs to balance scale, reliability, and cost-per-workload efficiency.
• Build CI/CD data pipelines, automated testing, and observability monitors for pipeline freshness, schema drift, and data quality checks

Preferred Profile :

• Work Experience: Minimum 4+ years of hands-on experience in building production-grade data engineering platforms, pipeline architecture, and data lakes.

Technical Expertise:

• Core Technologies: Azure Databricks, Azure Data Factory, ADLS Gen2, Python, PySpark, SQL, Delta Lake.
• Governance & Security: Unity Catalog, Microsoft Purview, PII segregation.
• AI & Advanced Data: Azure AI Search, Vector Databases, Embeddings, RAG architecture.
• Bonus Skills: dbt, Delta Live Tables, Terraform, Kafka, Event Hubs, Apache Iceberg.
• Educational Qualification: Bachelors or Masters degree in Computer Science, Data Science, Engineering, or relevant technical field.
• Domain Preference: Professional services, consulting, fintech, or regulated industries handling complex audit/financial datasets