Data Engineer

We are looking for an experienced Data Engineer with strong hands-on experience in Big Data technologies to design, build, maintain, and optimize scalable data pipelines and processing systems.

The role involves building batch and streaming pipelines using PySpark, Scala Spark, Kafka, Airflow, Hive, and Hadoop, processing large volumes of structured, semi-structured, and streaming data, and ensuring pipeline performance, reliability, and data quality.

The candidate will own data pipelines end-to-end, from data ingestion and transformation to orchestration, monitoring, troubleshooting, and downstream data delivery.

Essential Skills & Experience:

• 4-6 years of hands-on experience as a Data Engineer building and running production data pipelines.
• Strong hands-on experience with Apache Spark, PySpark, and Scala Spark.
• Strong knowledge of Spark SQL, DataFrames, joins, UDFs, and Spark performance tuning.
• Hands-on experience with Hadoop, Hive, and HDFS.
• Strong experience with Apache Kafka, including topics, partitions, consumer groups, offsets, and message delivery.
• Hands-on experience with Apache Airflow, including DAG development, scheduling, dependencies, retries, backfills, and monitoring.
• Strong programming skills in Python, SQL, and Scala.
• Experience with incremental imports, partitioning, bucketing, and large-scale data processing.
• Hands-on experience managing Spark jobs using spark-submit and monitoring production workloads.
• Experience with AWS, particularly S3, EMR, Glue, EC2, and IAM.
• Experience with Docker for packaging and deploying data workloads.
• Understanding of data modelling, schema evolution, and data formats such as Parquet, ORC, Avro, and JSON.
• Strong analytical, problem-solving, debugging, and data-quality mindset.

Key Responsibilities:

• Design, develop, and maintain scalable Big Data solutions using Apache Spark, PySpark, Scala Spark, Hive, Hadoop, HDFS, Kafka, and Airflow.
• Build and maintain batch and streaming data pipelines using PySpark, Scala, Python, and SQL.
• Process large volumes of structured, semi-structured, and streaming data efficiently.
• Build complex data processing pipelines and integrate data through REST APIs.
• Implement incremental data processing, partitioning, and bucketing to improve Spark performance and cost efficiency.
• Submit, monitor, tune, and troubleshoot Spark jobs, including resource sizing, executors, memory, and failure recovery.
• Read, process, and write data using Hive and Spark.
• Build Kafka-based pipelines for real-time and near-real-time data processing.
• Design and manage Airflow DAGs, scheduling, dependencies, retries, backfills, SLAs, and alerts.
• Deploy and run data workloads using AWS and Docker.
• Troubleshoot production pipeline failures and drive issues to root cause and closure.
• Improve pipeline performance, reliability, scalability, and data quality.
• Work closely with Product, Analytics, and Engineering teams to deliver accurate and timely data.
• Document data pipelines, data models, processes, and technical solutions.