Data Engineer

Job Summary

We are looking for an experienced Data Engineer with strong expertise in distributed data processing, real-time streaming, and data modeling. The ideal candidate should have hands-on experience with Apache Spark, Apache Flink, Kafka, and SQL to design, develop, and maintain scalable batch and streaming data pipelines.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and Apache Flink.
• Build real-time data ingestion and streaming solutions using Apache Kafka.
• Develop batch and stream-processing applications for large-scale data processing.
• Design, implement, and maintain data models to support analytical and business reporting requirements.
• Perform data transformation, cleansing, aggregation, and validation across multiple data sources.
• Optimize Spark jobs, streaming applications, SQL queries, and data processing workflows for performance.
• Work with structured and semi-structured data in distributed computing environments.
• Implement data quality checks, error handling, monitoring, and pipeline reliability mechanisms.
• Collaborate with data architects, data analysts, data scientists, and business teams to understand data requirements.
• Follow best practices for data governance, security, scalability, and documentation.

Required Skills

• Strong hands-on experience with Apache Spark and PySpark.
• Experience with Apache Flink for real-time stream processing.
• Good knowledge of Apache Kafka, including topics, partitions, producers, consumers, and event streaming.
• Strong SQL skills and experience working with large datasets.
• Experience in data modeling, including conceptual, logical, and physical data models.
• Knowledge of ETL/ELT processes and data pipeline development.
• Understanding of distributed computing, parallel processing, and data processing optimization.
• Experience with data quality, data validation, and troubleshooting pipeline issues.

Good to Have

• Experience with cloud data platforms such as AWS, Azure, or GCP.
• Knowledge of data lakes, data warehouses, and lakehouse architectures.
• Familiarity with Delta Lake, Apache Iceberg, or Apache Hudi.
• Experience with workflow orchestration tools such as Airflow.
• Knowledge of schema management and formats such as Avro, Parquet, and JSON.
• Experience with data governance, metadata management, and data cataloguing.

Candidate Profile

The ideal candidate should have practical experience building scalable batch and real-time data processing solutions using Spark, Flink, and Kafka, along with strong data modeling and SQL capabilities.

Key Skills for Sourcing: Data Engineer, Apache Spark, PySpark, Apache Flink, Kafka, Data Modeling, SQL, ETL/ELT, Real-Time Streaming, Batch Processing.