Big Data Engineer

Experience and Skills

Seeking an experienced Big Data Engineer with strong expertise in Spark, Scala, and Hadoop ecosystem technologies to design, develop, and optimize large-scale data processing solutions. • The candidate will be responsible for building high-performance data pipelines, managing distributed systems, and enabling scalable analytics platforms. • Required Skills: Apache Spark, Scala, Hadoop Ecosystem (HDFS, Hive, Yarn), SQL & Data Modeling, Distributed Computing Concepts, ETL/Data Pipeline Development

Preferred Skills:

Apache NiFi, Apache Kafka, AWS Services, Airflow, CI/CD & DevOps

Roles and Responsibilities

• Design and develop scalable Big Data applications using Spark and Scala. • Build and maintain data ingestion, transformation, and processing pipelines. • Work with Hadoop ecosystem components for distributed data storage and processing. • Develop batch and real-time data processing solutions. • Integrate multiple data sources using Apache NiFi and Kafka. • Optimize Spark jobs for improved performance and resource utilization. • Collaborate with business stakeholders, architects, and data consumers. • Implement data quality, governance, and security standards. • Support production deployments and troubleshoot complex data issue