Data Engineer -Python & Kafka

Role: Data Engineer Python & Kafka
Experience: 6-10 Years
Locations: Chennai, Bengaluru, Hyderabad, Pune, Kochi, Kolkata, NCR/Delhi

Virtual Interview: 01-OCT-26

Job Summary

We are looking for an experienced Data Engineer with strong expertise in Python and Apache Kafka to design, develop, and maintain scalable data pipelines and real-time data processing solutions. The ideal candidate should have experience working with large-scale data platforms, distributed systems, and cloud-based data engineering environments.

Key Responsibilities

• Design, develop, and optimize batch and real-time data pipelines using Python.
• Build and maintain Kafka-based streaming solutions for high-volume data processing.
• Develop scalable ETL/ELT frameworks and data ingestion pipelines.
• Integrate data from multiple sources and ensure data quality, reliability, and consistency.
• Work closely with Data Scientists, Analysts, and Application Development teams to support data requirements.
• Implement monitoring, logging, and performance tuning for data pipelines.
• Participate in architecture discussions and recommend best practices for data engineering solutions.
• Ensure compliance with data governance and security standards.

Required Skills

• Strong experience in Python programming.
• Hands-on experience with Apache Kafka, Kafka Streams, Kafka Connect, and message-based architectures.
• Experience in building ETL/ELT pipelines.
• Strong understanding of SQL and database concepts.
• Experience with distributed data processing frameworks such as Spark is preferred.
• Knowledge of REST APIs and data integration.
• Experience with cloud platforms such as GCP, AWS, or Azure.
• Familiarity with CI/CD, Git, and DevOps practices.
• Strong problem-solving and analytical skills.

Preferred Skills

• Experience with PySpark.
• Knowledge of Airflow or other workflow orchestration tools.
• Exposure to Data Lake/Data Warehouse architectures.
• Understanding of containerization technologies such as Docker and Kubernetes.