Digital : Apache Spark

Role & responsibilities

• Experience in Big Data technologies with strong expertise in Apache Spark, PySpark/Scala Spark, Hadoop, Hive, and SQL.
• Develop and maintain scalable batch and ETL data pipelines using Spark RDDs, DataFrames, Spark SQL, and distributed processing frameworks.
• Design, optimize, and troubleshoot Spark applications with a focus on performance tuning, partitioning, caching, serialization, and resource utilization.
• Work with large-scale structured and unstructured datasets stored in HDFS, Hive, S3, and cloud-based data lakes.
• Develop complex data transformation, aggregation, validation, and reconciliation logic using Spark SQL, SQL, and Scala/Python.