Lead Data Engineer

Key Responsibilities

- Lead the development of scalable ETL/ELT pipelines and data models to process high-volume data.
- Design and optimize data processing applications across platforms like Databricks, Snowflake, AWS, and Azure.
- Build and manage data pipelines using Apache Airflow for orchestration and Airbyte for data integration.
- Architect solutions for Change Data Capture, batch processing, and real-time/streaming data processing.
- Develop and manage REST APIs for data ingestion and distribution to applications and users.
- Leverage AI tools to enhance the development and deployment of data pipelines.
- Oversee CI/CD pipelines and DevOps deployments using Docker, Kubernetes, and monitoring tools.
- Mentor team members in best practices for data governance, quality, and security.

Overview

- As a Lead Data Engineer at our consulting firm, you will play a pivotal role in shaping the data architecture and engineering landscape.
- This position requires an expert with multi-platform proficiency, particularly in Databricks, Snowflake, AWS, and Azure, to lead scalable data solutions.
- You will work closely with cross-functional teams to integrate MLOps/DevOps practices and guide a team in delivering production-ready solutions.
- The role demands innovation in solving complex data challenges using a diverse toolkit and requires strong leadership skills to mentor and develop team members.
- Your expertise will help build robust, high-volume data processing pipelines and drive the adoption of AI-augmented deliverables.
- The position is based in Bangalore and Gurgaon, catering to high-profile clients in the consulting sector.
- You will be responsible for ensuring data governance, quality, and security while staying ahead of industry trends in MLOps and GenAI.
- This role offers an opportunity to collaborate with data scientists and analysts to ensure data readiness for machine learning projects and contribute to the evolution of CI/CD pipelines.

Requirements

- 5+ years of experience in Data Engineering with expertise in data lakes and warehouses.
- Proficiency in Change Data Capture (CDC), batch, and real-time data processing.
- Experience with multiple cloud data platforms such as Databricks, Snowflake, AWS, or Azure.
- Advanced SQL skills for large-scale data processing and analytics.
- Strong programming skills in Python, with experience in asynchronous programming and threading.
- Experience in creating and managing REST APIs using frameworks like FastAPI or Flask.
- Hands-on experience with Apache Airflow for workflow orchestration and Airbyte for data integration.