Senior Data Engineer

Role & responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using Python and PySpark.
• Develop efficient PySpark applications for processing large volumes of structured and semi-structured data.
• Write optimized Python scripts and modules for data processing, automation, and transformation.
• Develop complex SQL queries for data extraction, transformation, validation, and analysis.
• Perform data cleansing, transformation, aggregation, and validation across large datasets.
• Optimize PySpark jobs for performance, scalability, memory utilization, and execution time.
• Work with distributed computing concepts such as Spark architecture, DataFrames, RDDs, partitions, joins, caching, and transformations/actions.
• Build and maintain robust and reusable data engineering frameworks.
• Troubleshoot data pipeline failures and resolve performance and data-quality issues.
• Implement data validation, error handling, logging, and monitoring mechanisms.
• Collaborate with Data Scientists, Analytics teams, Software Engineers, and Business stakeholders.
• Participate in code reviews and follow coding, testing, and documentation best practices.
• Contribute to the design and implementation of scalable cloud-based data solutions.

Preferred candidate profile

• 7+ years of overall experience in Data Engineering / Big Data / Python development.
• Strong hands-on experience with Python.
• Strong hands-on experience with PySpark / Apache Spark.
• Strong proficiency in SQL and relational databases.
• Good understanding of ETL/ELT concepts and data pipeline development.
• Experience working with large datasets and distributed data processing.
• Strong understanding of Spark DataFrames, transformations, actions, joins, partitioning, caching, and optimization techniques.
• Experience with data formats such as JSON, CSV, Parquet, and Avro.
• Good understanding of data structures, algorithms, and object-oriented programming in Python.
• Experience with Git/version control and software development best practices.
• Strong debugging, analytical, and problem-solving skills.