BigData Engineer - Architect

Responsibilities

Ensure architecture and design activities follow established SDLC and governance standards. Participate in code reviews, design reviews, and architecture assessments. Promote reliability, maintainability, scalability, and performance considerations throughout solution development. Drive continuous improvement in architecture practices and engineering standards.

Required Knowledge & Experience

Technical Expertise

12+ years of experience in software engineering, data engineering, or distributed systems development.

Strong hands-on expertise with Apache Spark (Spark SQL, Structured Streaming, DataFrames, Dataset APIs). Experience designing and building large-scale distributed data processing systems. Strong knowledge of Spark optimization techniques including:

• Partitioning Strategies

• Shuffle Optimization

• Join Optimization

• Memory Management

• Resource Utilization

• Performance Tuning

Proficiency in Scala, Java, or Python. Experience with technologies such as:

• Hadoop

• Hive

• Iceberg

• AWS EMR

• AWS Glue

• GCP Dataproc

• BigQuery

Experience designing and operating batch and streaming data pipelines. Understanding of cloud-native architecture and distributed systems principles. Experience deploying Spark solutions on AWS and/or GCP platforms.

Scope & Positioning

Individual contributor architecture role focused on distributed data platforms and Spark-based solutions. Provides architecture leadership and technical guidance across one or more engineering teams. Responsible for solution architecture quality, technology selection, design governance, and technical direction within the domain. Partners with Engineering Managers and Technical Leads to deliver scalable and reliable platform capabilities. Serves as a key technical advisor for Spark architecture, distributed systems design, and cloud-native data platforms.

Preferred Qualifications

• Experience with AWS services such as EMR, Glue, S3, Lambda, EKS, ECS, Step Functions, and CloudWatch.

• Experience with GCP services such as Dataproc, BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.

• Experience managing terabyte-to-petabyte scale data processing environments.

• Experience with Spark Structured Streaming, real-time analytics, and event-driven data processing.

• Contributions to Spark optimization, platform engineering, or open-source data ec