Data Engineering Lead

Data Engineering Lead

Role Summary

The Data Engineering Lead will build and run the data foundations for AI services. This covers ingestion from departmental systems and national platforms, document pipelines at volume, the data platform, versioned training and evaluation datasets, retrieval (RAG) infrastructure, and synchronisation with AIKosh. The role is hands-on and sets data standards for in-house and vendor teams. It requires strong engineering on on-premise and government cloud infrastructure, not dependence on any single commercial data warehouse.

Educational Qualification

• B.Tech./B.E., M.Tech./M.S. or M.Sc. in Computer Science, Data Engineering, Information Technology or a related discipline (Must have)
• Professional certification in data platforms or Kubernetes is desirable but not mandatory; demonstrated work, as listed under Minimum Evidence at Screening, shall carry greater weight
• Candidates from non-traditional backgrounds with strong verifiable work will be considered in lieu of formal qualification

Experience

• 8+ years in data engineering or data platform work (10+ years preferred); minimum 4 years designing and running large-scale data pipelines for analytics or AI systems
• Demonstrable ownership of at least one production data platform, with documented volumes, reliability and data-quality measures
• Experience with structured, semi-structured and unstructured data, including documents and images at volume
• Experience building training and evaluation datasets with ML teams; experience with embedding and vector-index pipelines for retrieval is a strong plus
• Experience on both on-premise and cloud infrastructure
• Experience handling personal data under data-protection law is a strong plus

Key Responsibilities

Data Acquisition & Integration

• Build batch, streaming and change-data-capture ingestion from departmental databases, APIs and files, and connectors to national platforms
• Define the technical terms of data-sharing agreements with departments

Document & Unstructured Data Pipelines

• Build high-volume intake for scanned documents and images, with parsing and OCR pipelines run with the NLP & Document AI team, metadata capture and storage

Data Platform

• Run the lakehouse or warehouse, orchestration, transformation, quality checks, lineage and data catalogue on IndiaAI and on-premise infrastructure

Datasets for AI

• Maintain a registry of training data and golden evaluation datasets, with versioning, dataset documentation and access control
• Run annotation pipelines, including for Indian-language data, with the QA team

Retrieval Infrastructure & AIKosh

• Build embedding pipelines and manage the lifecycle of vector indexes, including separate indexes per department
• Synchronise with the AIKosh catalogue, and prepare datasets for publication on AIKosh

Data Governance & Privacy

• Apply classification, masking, anonymisation and retention of personal data under DPDPA 2023, and maintain access logs

Leadership & Standards

• Build pipelines directly, set data standards for vendor teams, and review their pipelines before acceptance

Minimum Evidence at Screening (one required)

• A production data platform or pipelines the candidate built, with documented volumes and reliability, verifiable through reference or artefact
• A data-quality, lineage or dataset-versioning approach the candidate put in place, with its results
• An open-source contribution to a data engineering tool, with verifiable commit history

Shortlisted candidates shall complete one practical assessment: building a pipeline over a messy multi-source dataset (CSV files, scanned-document metadata and an API), with quality checks, lineage and a versioned evaluation dataset, and explaining what changes at 100 times the volume. This shall be followed by a technical discussion of a data platform the candidate has built.