Data Engineer

Job description

Role: Data Engineer Exp: 5-8 Years

Responsibilities

• Build document ingestion pipelines from SharePoint, DMS and file stores into vector databases, including chunking and pre-processing
• Build ETL/ELT pipelines from structured enterprise sources (ERP, CRM, databases) into the AI data layer
• Design and manage a feature store — curated, versioned features for ML models
• Implement data quality validation, lineage tracking and alerting
• Build data access interfaces used by AI Engineers and Backend Engineers
• Orchestrate pipelines using Airflow or Prefect — scheduling, retry logic, SLA monitoring

Required Skills

• Python — for transformation logic, custom operators and data processing
• SQL — advanced. Window functions, CTEs, query optimisation. Can read and interpret query plans.
• Workflow orchestration — Airflow or Prefect in a production environment. Not just writing DAGs — operating them, handling failures, managing backfills.
• Cloud data storage — S3, ADLS or GCS. At least one cloud warehouse (Snowflake, BigQuery or Azure Synapse).
• Data modelling — understands medallion architecture (bronze/silver/gold) and can design a schema for a given use case.

Additional Skills

• Vector store ingestion — has built a pipeline that loads embeddings into a vector database. Understands chunking strategy and its effect on retrieval quality.
• Feature store — has contributed to or built a feature store. Understands offline vs online feature serving and training-serving skew.
• dbt — models, tests, documentation
• Data contracts — has defined or enforced a data contract between producer and consumer
• Real-time pipelines — Kafka, Azure Event Hub or Kinesis