Data Engineer

Experience

5–6 Years Role Summary We are looking for a skilled

Data Engineer with strong expertise in

Scala, Apache Spark, Kafka, and Google Cloud Platform (GCP) to design, develop, and optimize scalable data pipelines for both batch and real-time streaming workloads. The ideal candidate should have hands-on experience with ETL development, Airflow orchestration, BigQuery, Dataproc, Medallion Architecture, and data quality frameworks. The role involves building high-performance, reliable, and scalable data processing solutions while ensuring observability, governance, and continuous optimization across enterprise data platforms. Key Responsibilities

• Design, develop, deploy, and maintain scalable ETL pipelines for both batch and streaming data processing.
• Build and optimize data workflows using Scala, Apache Spark, Apache Kafka, and Google Cloud Platform (GCP).
• Develop and maintain Apache Airflow DAGs with proper dependency management, scheduling, retries, and idempotent execution.
• Build robust streaming pipelines with Kafka, ensuring efficient handling of: + Backpressure + Fault tolerance + Checkpointing + Exactly-once processing
• Develop enterprise data pipelines using: + Dataproc + BigQuery + Cloud Storage + Kafka
• Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate transformation and summarization layers.
• Design and implement comprehensive data quality validation and reconciliation processes.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement monitoring, logging, observability, and operational dashboards for production pipelines.
• Create operational documentation, runbooks, and troubleshooting guides.
• Analyze pipeline failures and perform root cause analysis to improve platform reliability.
• Collaborate with Data Architects, Business Analysts, and Data Scientists to deliver scalable data solutions.
• Participate in code reviews and ensure adherence to engineering best practices. Must-Have Skills
• 5–6 years of experience as a Data Engineer.
• Strong hands-on experience with Scala.
• Strong expertise in Apache Spark.
• Experience building Batch and Streaming ETL pipelines.
• Strong experience with Apache Kafka.
• Hands-on experience with Google Cloud Platform (GCP).
• Experience working with: + Dataproc + BigQuery + Cloud Storage
• Experience with Apache Airflow orchestration.
• Strong understanding of: + DAG dependency management + Idempotent execution + Scheduling + Retry mechanisms
• Experience implementing Medallion Architecture.
• Strong understanding of Data Quality frameworks and validation.
• Experience in Spark Performance Optimization.
• Experience with monitoring, logging, and observability.
• Ability to create production runbooks and operational documentation.
• Strong SQL skills.
• Good understanding of distributed data processing concepts. Skills: gcp,dataproc,kafka,scala