Senior Data Engineer – AI & Analytics

We are looking for a

Senior Data Engineer - AI & Analytics with strong Data Engineering expertise, analytical thinking, and AI enablement capabilities to build scalable data solutions that power analytics, dashboards, recommendations, and AI-driven use cases. The role involves designing and evolving data products within a modern

Azure + Databricks Lakehouse architecture , enabling business insights and AI solutions through curated, consumption-ready datasets. The ideal candidate will own the end-to-end data lifecycle and work closely with business, product, and engineering teams to deliver scalable and maintainable solutions.

Qualification

• BE / B.Tech / MCA / ME / M.Tech
• 7+ years of experience in Data Engineering / Analytics Engineering

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Databricks and Azure data services
• Integrate internal/external data sources and build reusable, modular data components
• Develop curated datasets and data products for analytics, dashboards, recommendations, and AI applications
• Design batch and streaming solutions; optimize Spark workloads, Delta tables, and low-latency data processing
• Drive data quality, governance, reliability, root-cause analysis, and exploratory data analysis (EDA)
• Collaborate with stakeholders to define KPIs, business metrics, and analytics-ready datasets
• Prepare and structure datasets for AI Agents / GenAI and support Azure AI Foundry integration patterns
• Design semantic and metadata-driven datasets and enable downstream AI and BI consumption
• Support Qlik / BI performance optimization and ensure consistency of business metrics
• Implement testing, CI/CD, version control, and engineering best practices using Azure DevOps
• Participate in agile delivery including planning, estimation, releases, and cross-functional collaboration

Required Skills Data Engineering & Platform

• Spark 3.x (DataFrames, SQL, Batch & Structured Streaming)
• Databricks (Workflows, SQL Warehouses, DLT, Unity Catalog, Auto Loader, Pipelines)
• Azure Data Services and Lakehouse / Medallion Architecture
• Parquet / Delta, partitioning, compaction, and performance optimization

Programming & Analytics

• Strong Python and SQL (Spark SQL, TSQL, HiveQL)
• Data quality, EDA, KPI-driven analytical modeling
• Understanding of statistical concepts and data readiness for analytics/recommendation use cases
• Experience building reusable, analytics-ready, and AI-ready datasets

AI, BI & Delivery

• Azure AI Foundry integration and AI/Agent data preparation
• Experience supporting Qlik / Power BI / Tableau workloads
• Testing frameworks (pytest, Great Expectations, Acceptance Testing)
• CI/CD with Azure DevOps and YAML pipelines
• Agile/Scrum development practices

Good to Know

• ADLS, Managed Identity, Azure AI Foundry
• Feature engineering concepts
• Airflow / ADF / Synapse Pipelines
• Scala or Java
• Data Catalogs (Purview, Unity Catalog, Apache Atlas)
• Healthcare domain experience (preferred)