Senior Data Engineer

Roles and Responsibilities :

• Design, develop, test, deploy and maintain large-scale data pipelines using Scala and Spark on GCP.
• Collaborate with cross-functional teams to gather requirements and deliver high-quality data solutions.
• Develop scalable architectures for big data processing using Kafka, Data Bricks, and other technologies.
• Ensure data quality by implementing robust testing frameworks and monitoring dashboards.

Job Requirements :

• 10-19 years of experience in designing and developing large-scale data systems.
• Strong proficiency in Scala programming language with expertise in Spark.
• Experience working with cloud-based platforms such as GCP (Google Cloud Platform).
• Knowledge of messaging queues like Kafka.
• Spark Structured Streaming, Apache Spark, Scala, Apache Kafka, Databricks, Any cloud, preferably GCP.
  In depth coding knowledge in Spark Structured Streaming.
  Spark Internals - Spark execution concepts in depth, including query planning, DAG generation, shuffle operations, partitioning strategies, and memory management.
  Understanding of Spark optimization techniques such as broadcast joins, caching/persistence strategies, partition pruning, adaptive query execution, and skew handling.
  Databricks Platform Knowledge - cluster sizing strategies, job optimization, Delta Lake internals, workflow orchestration, and performance tuning.
  Data Ingestion and Architecture Patterns - Exposure to ingestion frameworks and modern data integration patterns.
  Spark internals and execution architecture.
  Databricks platform expertise and optimization techniques.
  Advanced batch and streaming ingestion patterns.
  Performance tuning and troubleshooting at scale.
  Data architecture and design decision-making.
  Technical leadership, mentoring, and stakeholder communication