Data Engineer

• Spark
• Scala
• Python
• Bigdata

Data Engineer - Spark, Scala, Python & Big Data
Experience: 5 - 8 Years Primary Skills: Apache Spark, Scala, Python, Big Data
Secondary Skills:Shell/Python Scripting, GCP Data Stack, Airflow, CI/CD, Kafka, SQL/NoSQL

Role Overview -

We are looking for an experienced Data Engineer with strong expertise in Apache Spark, Scala, Python, and Big Data technologies to design, develop, and optimize scalable data pipelines and data processing solutions.

Key Responsibilities -

- Design, develop, and maintain scalable data pipelines using Spark, Scala, and Python.
- Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
- Develop batch and streaming data processing solutions.
- Optimize Spark jobs for performance, scalability, and resource utilization.
- Work with GCP data services, Apache Airflow, and Kafka for data integration and workflow orchestration.
- Write SQL queries and work with relational and NoSQL databases.
- Implement CI/CD pipelines, automated testing, and deployment processes.
- Troubleshoot production issues and ensure data quality, reliability, and consistency.

Required Skills -

- 5 - 8 years of experience in Data Engineering or Big Data development.
- Strong hands-on experience with Apache Spark, Scala, and Python.
- Good understanding of distributed computing and ETL/ELT pipelines.
- Experience with SQL, data transformation, and performance tuning.
- Exposure to GCP, Airflow, Kafka, shell scripting, and CI/CD.
- Strong problem-solving, debugging, and communication skills.

Preferred: Experience with cloud-based data platforms, real-time data processing, and large-scale data engineering projects.