Software Engineer

Job Description: Data Engineer Python, PySpark, Apache Iceberg & Airflow

Job Title: Data Engineer
Experience: 5–11 Years
Locations: Bangalore and Chennai
Primary Skills: Python, PySpark, Apache Iceberg, Apache Airflow
Job Type: Full-Time

Job Summary

We are looking for experienced Data Engineers with strong expertise in Python, PySpark, Apache Iceberg, and Apache Airflow to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have hands-on experience building robust data pipelines, processing large volumes of data, and implementing reliable data workflows for modern data platforms.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Build and optimize large-scale data processing applications using Apache Spark.
• Implement and manage data lakehouse solutions using Apache Iceberg.
• Develop, schedule, and monitor data workflows using Apache Airflow.
• Work with data lakes, distributed processing frameworks, and cloud-based data platforms.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, error handling, logging, and pipeline monitoring.
• Collaborate with data architects, analysts, and other engineering teams to deliver reliable data solutions.
• Troubleshoot data pipeline failures and resolve performance and data integrity issues.
• Follow best practices for data security, governance, testing, and documentation.

Required Skills and Qualifications

• Experience: 5–11 years of relevant experience in data engineering.
• Python: Strong programming skills, including reusable modules, error handling, and data processing.
• PySpark: Hands-on experience with Spark DataFrames, transformations, joins, partitioning, and performance tuning.
• Apache Iceberg: Experience with Iceberg tables, schema evolution, partition evolution, and table maintenance.
• Apache Airflow: Experience developing and managing DAGs, scheduling workflows, handling dependencies, and monitoring pipeline execution.
• Strong understanding of ETL/ELT processes, data warehousing, and distributed data processing.
• Good knowledge of SQL and data modeling concepts.
• Experience working with data lake or lakehouse architectures.
• Familiarity with Git, code reviews, debugging, and software development best practices.

Good to Have

• Experience with AWS, Azure, or Google Cloud Platform.
• Knowledge of cloud data services and object storage such as Amazon S3 or Azure Data Lake Storage.
• Experience with Kafka or other data ingestion technologies.
• Familiarity with Delta Lake, Parquet, Hive, or similar data technologies.
• Experience with CI/CD, Docker, Kubernetes, and automated testing.
• Understanding of data governance, security, and observability.

Preferred Candidate Profile

• Strong analytical and problem-solving skills.
• Ability to work independently and collaborate effectively with cross-functional teams.
• Experience delivering production-grade data pipelines in enterprise environments.
• Excellent communication and documentation skills.

Job Locations

• Bangalore, Karnataka
• Chennai, Tamil Nadu

Experience Range: 5–11 Years

Keywords for Recruitment: Data Engineer, Python, PySpark, Apache Spark, Apache Iceberg, Airflow, ETL, ELT, Data Lakehouse, Data Pipelines, SQL, Cloud Data Engineering.