Lead Engineer - Data & AI

Job Requirements At Quest Global, it’s not just what we do but how and why we do it that makes us different. With over 25 years as an engineering services provider, we believe in the power of doing things differently to make the impossible possible. Our people are driven by the desire to make the world a better place—to make a positive difference that contributes to a brighter future. We bring together technologies and industries, alongside the contributions of diverse individuals who are empowered by an intentional workplace culture, to solve problems better and faster.

Overview We are looking for an AI & Data Engineer who has strong foundational expertise in data engineering and has extended that into building intelligent AI systems. This role sits at the intersection of scalable data infrastructure and production-grade AI — you will be expected to design and build robust pipelines, model data correctly from first principles, and deliver GenAI-powered applications that run reliably in enterprise environments. You will work across the full data-to-intelligence stack: ingestion, transformation, warehousing, retrieval, reasoning, and deployment. Equal weight is placed on data engineering rigor and AI engineering capability.

Key Responsibilities Data Engineering

• Design, build, and maintain end-to-end batch and real-time data pipelines using Python, PySpark, and Databricks across structured, semi-structured, and unstructured data sources.
• Develop and manage ELT/ETL workflows using dbt for transformation and Airflow for orchestration — including testing, documentation, and data quality enforcement.
• Build and maintain cloud data warehouse and lakehouse architectures (Snowflake, BigQuery, Delta Lake) following modelling best practices.
• Implement dimensional models, SCD strategies, and incremental load patterns appropriate to each use case.
• Integrate data from diverse enterprise sources: REST APIs, relational databases, Kafka event streams, and file-based systems.
• Manage data quality, lineage tracking, and metadata documentation as part of standard delivery — not as an afterthought.
• Optimise query performance, partitioning strategies, and materialization approaches in cloud warehouse environments.

AI & GenAI Engineering

• Architect and build production-grade RAG pipelines combining vector retrieval, graph traversal, and structured data lookups for multi-hop reasoning over enterprise datasets.
• Build distributed agent-based systems using LangGraph, MCP, and A2A frameworks with modular services for ingestion, retrieval, and reasoning.
• Design LLM evaluation and validation layers to enforce semantic accuracy and reduce manual review overhead in AI pipelines.
• Develop LLM-based automation workflows, recommendation systems, and document intelligence solutions.
• Build explainability layers that combine reasoning traces with LLM outputs to meet enterprise compliance and governance requirements.

Infrastructure & Integration

• Deploy data and AI systems on Kubernetes with robust CI/CD pipelines ensuring reproducibility and production reliability.
• Design FastAPI microservices to expose data and AI capabilities to upstream and downstream enterprise systems.
• Apply observability and monitoring best practices across pipelines and AI services — tracking data freshness, pipeline SLAs, and model behaviour.
• Implement caching strategies (Redis) and query optimisation techniques to meet latency requirements at production scale. We are known for our extraordinary people who make the impossible possible every day. Questians are driven by hunger, humility, and aspiration. We believe that our company culture is the key to our ability to make a true difference in every industry we reach. Our teams regularly invest time and dedicated effort into internal culture work, ensuring that all voices are heard. We wholeheartedly believe in the diversity of thought that comes with fostering a culture rooted in respect, where everyone belongs, is valued, and feels inspired to share their ideas. We know embracing our unique differences makes us better, and that solving the worlds hardest engineering problems requires diverse ideas, perspectives, and backgrounds. We shine the brightest when we tap into the many dimensions that thrive across over 21,000 difference-makers in our workplace.

Work Experience Mandatory

Data Engineering

• Strong proficiency in Python for data pipeline development, ETL/ELT automation, and data processing.
• Advanced SQL — window functions, CTEs, query optimisation, and performance tuning on large-scale datasets.
• Hands-on experience with dbt for data transformation, modelling, incremental strategies, testing, and documentation.
• Experience with Apache Airflow for orchestrating multi-step workflows — DAG design, retry logic, dependency management, and SLA monitoring.
• Proficiency with PySpark and Databricks for distributed data processing, including Delta Lake and batch/streaming workloads.
• Experience with cloud data warehouses — Snowflake, BigQuery, or Redshift — including schema design, partitioning, clustering, RBAC, and query optimisation.
• Solid understanding of data modelling paradigms: dimensional modelling (star/snowflake schema), SCDs, and medallion/lakehouse architecture.
• Familiarity with streaming ingestion using Apache Kafka or equivalent event-driven platforms.
• Experience with data quality frameworks, schema validation, and pipeline observability.

AI & GenAI Engineering

• Experience building production-grade RAG (Retrieval-Augmented Generation) pipelines with hybrid retrieval — dense vector search combined with structured or graph-based retrieval.
• Proficiency with LangChain, LangGraph, or equivalent frameworks for LLM orchestration and agentic pipeline design.
• Working knowledge of vector databases — Pinecone, Qdrant, Weaviate, or equivalent — including embedding model selection and indexing strategies.
• Experience integrating LLM APIs (OpenAI, Azure OpenAI, Anthropic) into production data and AI pipelines.
• Understanding of agentic system patterns: MCP, A2A frameworks, tool use, memory, and multi-step reasoning.

Cloud & Infrastructure

• Hands-on experience deploying data and AI systems on Azure or AWS — including AKS, Azure Data Factory, S3, Glue, or equivalent managed services.
• Experience with Docker, Kubernetes, and CI/CD pipelines for reliable, reproducible deployments.
• Familiarity with Redis for caching and FastAPI for building lightweight API service layers.

Nice to Have

• Knowledge of semantic web standards: RDF, SPARQL, OWL/TTL, JSON-LD for interoperable data modelling.
• Experience with knowledge graph platforms — Neo4j, Apache Jena — including Cypher querying and ontology design.
• Exposure to ML pipelines, object detection, NLP models, or structured data extraction from documents.
• Familiarity with data cataloguing and lineage tools: Apache Atlas, DataHub, OpenMetadata, or similar.
• Open-source contributions, technical blog posts, or documented project case studies.

Qualifications

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related technical discipline.
• Demonstrable experience in a data engineering or AI engineering role with progressively increasing scope and ownership.
• Portfolio of delivered work — production pipelines, warehouse implementations, RAG systems, or AI applications — is strongly preferred over credentials alone.
• dbt Analytics Engineering Certification is a strong plus.
• Cloud certifications — Google Professional Data Engineer, Azure DP-203, or AWS Data Analytics Specialty — are valued but not mandatory.
• Strong communication skills: able to write clear technical documentation and participate in architecture reviews and client-facing discussions.