Data Engineer

About The Role We are looking for a

Data Engineer with strong expertise in

Hadoop, Hive, PySpark, Python, Unix, and Apache Airflow to design, develop, and maintain scalable data pipelines for enterprise data platforms. The ideal candidate will have hands-on experience working with large-scale datasets, distributed data processing, and workflow orchestration while ensuring high-quality, reliable, and efficient data solutions. Key Responsibilities

• Design, develop, and optimize scalable batch data processing pipelines using PySpark and the Hadoop ecosystem.
• Build, maintain, and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
• Develop efficient data transformation logic using Hive, PySpark, and Python.
• Create, schedule, and monitor data workflows using Apache Airflow.
• Write optimized Hive queries and improve query performance for large datasets.
• Work with Hadoop Distributed File System (HDFS) and other Hadoop ecosystem components.
• Develop automation scripts using Python and Unix shell scripting.
• Troubleshoot production issues, optimize job performance, and ensure data quality.
• Collaborate with data architects, analysts, and application teams to understand business requirements and deliver scalable data solutions.
• Follow best practices for version control, testing, deployment, and documentation. Mandatory Skills
• Strong experience with Hadoop ecosystem.
• Hands-on experience with Hive.
• Strong programming skills in Python.
• Experience developing data pipelines using PySpark.
• Strong knowledge of Unix/Linux commands and shell scripting.
• Hands-on experience with Apache Airflow for workflow orchestration.
• Experience building and maintaining ETL/ELT pipelines.
• Strong SQL skills with experience optimizing complex queries.
• Experience handling large-scale data processing and performance tuning.
• Good analytical and troubleshooting skills.

Preferred Skills

• Experience with Spark optimization and partitioning strategies.
• Knowledge of HDFS and distributed computing concepts.
• Familiarity with Git and CI/CD practices.
• Experience working in Agile/Scrum environments.
• Exposure to cloud platforms such as AWS, Azure, or GCP is an advantage.
• Knowledge of data warehousing concepts and dimensional modeling. Technical Skills Mandatory
• Hadoop
• Hive
• Python
• PySpark
• Unix/Linux
• Apache Airflow
• SQL
• ETL/ELT Development Good to Have
• Apache Spark
• HDFS
• Git
• Shell Scripting
• Data Warehousing
• AWS / Azure / GCP
• CI/CD
• Agile

Experience

• 3–8 years of experience in Data Engineering or Big Data development.
• Proven experience building production-grade ETL pipelines and working with enterprise-scale datasets.
• Experience supporting and optimizing large-scale distributed data processing environments. Skills: processing,data processing,hadoop,airflow,python,data,pipelines,apache