Data Platform Engineer - II

Your role

We, at Khatabook, are looking to hire a Data Engineer II for our Data Engineering team.

As a Data Engineer II, you will build, maintain, and scale reliable, secure, and cost-efficient data infrastructure that enables data-driven decision-making across the organisation. You will work on production-grade data pipelines, cloud data warehouses, CDC systems, streaming platforms, and large-scale data processing.

You will play a key role in ensuring data reliability, quality, security, and performance while contributing to AI-ready data infrastructure that supports AI/ML and Generative AI use cases.

What would you do at Khatabook?

• Design, develop, and maintain scalable batch and streaming data pipelines using Python and Apache Airflow.
• Build reliable data ingestion workflows with appropriate error handling, retries, monitoring, and recovery mechanisms.
• Develop incremental data processing workflows and Change Data Capture (CDC) pipelines to ensure timely and consistent data availability.
• Build, maintain, and optimize cloud data warehouses, preferably Snowflake, with a focus on scalability, performance, and cost efficiency.
• Develop and maintain SQL-based data models and transformation workflows using tools such as DBT.
• Build event-driven data workflows using Kafka or other messaging and queuing technologies.
• Develop and optimize Apache Spark jobs for large-scale data processing.
• Build and operate data engineering workloads on AWS and work with Kubernetes and containerised environments where appropriate.
• Optimize SQL queries, data models, Spark jobs, warehouse utilisation, and pipeline execution to improve performance and reduce infrastructure costs.
• Implement data quality checks, validation frameworks, monitoring, alerting, and failure-recovery mechanisms to ensure data accuracy, freshness, and reliability.
• Implement data access controls, role-based access control (RBAC), secure data sharing, and appropriate security practices for sensitive data.
• Support data governance, metadata management, and data lineage to improve data discoverability and trust.
• Build and maintain data pipelines that prepare structured and unstructured data for AI/ML and Generative AI applications.
• Support AI-powered search and analytics use cases involving Large Language Models (LLMs), embeddings, vector databases, and Retrieval-Augmented Generation (RAG).
• Collaborate with engineering, product, and analytics teams to understand data requirements and deliver trusted, actionable data.
• Take ownership of production workloads, troubleshoot pipeline failures, investigate incidents, and continuously improve data platform reliability and operational efficiency.
• Explore AI-assisted development tools to improve Python and SQL development, debugging, and code quality while validating generated code for correctness and security.
• Participate in a shared production on-call rotation approximately once every three weeks to ensure the reliability of critical data workflows.

What are we looking for?

• You have 3 - 7 years of experience in Data Engineering or a related technical role.
• You have strong Python programming skills and advanced SQL knowledge, including complex joins, window functions, query optimisation, and performance tuning.
• You have hands-on experience building and maintaining production-grade data pipelines using Apache Airflow.
• You have experience working with modern data warehouses such as Snowflake or Amazon Redshift.
• You have experience with CDC pipelines, incremental data ingestion, or database replication.
• You have working knowledge of AWS cloud services and experience operating data engineering workloads.
• You have experience with Apache Spark or PySpark for large-scale data processing.
• You have experience with Kafka or other messaging and queuing technologies.
• You have a strong understanding of relational databases such as PostgreSQL or MySQL.
• You understand data modeling, data quality, pipeline reliability, monitoring, troubleshooting, and production support.
• You have strong analytical, problem-solving, and debugging skills.
• You understand distributed systems, data security, and performance optimisation.
• You have a strong sense of ownership and can independently troubleshoot and improve production systems.
• You can collaborate effectively with engineering, product, and analytics teams to deliver scalable data solutions.

Good to have:

• Strong knowledge of Snowflake security, RBAC, secure data sharing, and compute and storage cost optimisation.
• Familiarity with DBT for data transformation, testing, documentation, and data model management.
• Understanding of dimensional modeling and modern data warehouse architecture.
• Familiarity with Docker, Kubernetes, CI/CD, and automated testing.
• Experience with data observability, metadata management, and data lineage.
• Understanding of Generative AI concepts, including LLMs, embeddings, vector databases, and RAG.
• Exposure to AI-ready data pipelines for structured and unstructured data.
• Familiarity with AI-assisted development tools for Python and SQL.
• Exposure to AI application platforms such as AWS Bedrock or Snowflake Cortex.
• Familiarity with BI and analytics platforms such as Tableau, Redash, or similar tools.