Senior Data Engineer

Responsibilities:

• Design, build, and manage scalable data processing pipelines on the Google Cloud Platform.
• Drive engineering excellence, data quality, and continuous improvement.
• Lead end-to-end design, development, and deployment of scalable data pipelines and integration solutions.
• Manage multiple delivery workstreams.
• Translate complex business and technical requirements into robust data engineering solutions.
• Develop and optimise ETL/ELT processes using modern cloud, data warehousing, and data processing platforms.
• Collaborate with technical and business teams to translate complex requirements into robust data solutions.
• Diagnose and resolve complex production issues through root-cause analysis and system optimisation.
• Drive technology and process improvements by evaluating emerging tools and engineering best practices.
• Ensure adherence to enterprise architecture, security, compliance, and performance standards.
• Maintain high-quality technical documentation, user guides, and solution specifications.

Requirements:

• Experience in data modelling, data engineering, and data integration.
• Experience leading engineering initiatives.
• Proven experience with cloud tools such as GCP Dataflow, GCP Composer (Airflow), and GCP Cloud Storage.
• Strong expertise in cloud platforms: GCP.
• Hands-on experience with data warehousing and processing technologies.
• Experience with data streaming tools: Kafka, Cloud Data Flow, Spark Streaming.
• Strong experience in data quality and observability platforms to automate quality coverage using iceDQ, Monte Carlo, or similar tools.
• Strong proficiency with high-level programming and query languages.
• Experience collaborating with distributed teams across time zones.
• Skills: BigQuery, Lakehouse (Iceberg), Spark, Python, SQL.

AI Skills (Mandatory):

• Proficiency in leveraging AI tools for daily engineering tasks to enhance productivity, optimise effort, and ensure cost-aware AI assistance.
• Familiarity with Retrieval Augmented Generation (RAG) architectures, including semantic layers for data retrieval and grounding Large Language Model (LLM) responses.
• Conceptual understanding of LLM-based applications (e. g., chatbots, Q& A systems), encompassing prompt engineering, context management, and response generation.
• Exposure to agentic and multi-agent AI systems, including agent roles, tool utilisation, memory management, and workflow orchestration.
• Practical experience with LLM application frameworks (e. g., LangChain, Lang Graph, or Google GenAI tools) for prototyping and integrating AI-driven solutions.

Nice to Have:

• Experience with Power BI/ThoughtSpot, AI in data engineering, Stonebranch, CI/CD, Databricks and DevOps practices.
• Familiarity with enterprise databases such as DB2 SQL Server, Oracle, and PostgreSQL.
• Google Cloud Data Engineer certification is preferred.
• Experience in retail domains (customer, stores, eCommerce, marketing, supply chain, and merchandising).