Data Engineer

Role & responsibilities

Build and operate the data pipelines that agentic AI systems run on.

• Design, build, and maintain batch and streaming data pipelines in Python and SQL
• Build and orchestrate ETL/ELT workflows using Airflow, Dagster, or similar
• Work with distributed processing frameworks Spark or equivalent — on large datasets
• Ingest data from APIs, databases, files, and event streams (Kafka or similar) into the lakehouse
• Model data for analytics and for machine learning consumption, including feature preparation
• Build and maintain the data layer behind retrieval-augmented generation — chunking, embedding pipelines, and vector store population
• Own data quality end to end: validation, testing, monitoring, lineage, and alerting on pipeline failures
• Optimise for cost and performance — query tuning, partitioning, storage format decisions
• Implement access controls and governance appropriate to enterprise and regulated customers
• Work directly with AI engineering, platform, and product rather than through handoffs

You will be building the data foundation for Agentic AI solutions and system, where pipelines feed live agent workflows, not just dashboards, and both correctness and latency matter.

Preferred candidate profile

• 2–4 years of professional data engineering experience
• Strong Python and advanced SQL — window functions, query optimisation, complex joins
• Hands-on experience building production pipelines with an orchestration tool (Airflow, Dagster, Prefect)
• Experience with a distributed processing framework, typically Spark (PySpark)
• Working knowledge of a cloud data platform — Databricks, Snowflake, BigQuery, or Redshift
• Comfortable with Git, Docker, and at least one cloud provider (AWS, Azure, or GCP)
• Understanding of data modelling — dimensional modelling or one big table, and when each applies
• Fluent with AI coding assistants in your daily workflow, and able to review what they produce rather than ship it unread
• B.E. / B.Tech in Computer Science, Data Engineering, or a related field. Strong self-taught engineers with demonstrable production work are equally welcome

Good to have

• Streaming experience — Kafka, Flink, Spark Structured Streaming
• Lakehouse table formats — Delta Lake, Apache Iceberg, or Hudi
• dbt for transformation and testing
• Vector databases and embedding pipelines for RAG
• Kubernetes, Terraform, or CI/CD for data infrastructure
• Data observability and lineage tooling
• Exposure to feature stores or ML pipeline tooling