Lead Data Engineer

Greetings from SwaaS!

Role: Lead Data Engineer

Experience: 8+ Years

Location: Guindy, Chennai

Job Description:

• Design, develop, and maintain scalable end-to-end ETL/ELT data pipelines using Python and PySpark for large-volume data ingestion, transformation, processing, and integration.
• Develop and optimize data engineering solutions using Databricks and PySpark, including data transformation, performance tuning, job optimization, and handling large datasets.
• Leverage AWS cloud services to design, implement, and manage reliable and scalable data solutions, following best practices for performance, security, availability, and cost optimization.
• Lead the development and implementation of ETL workflows, ensuring data quality, accuracy, integrity, and efficient movement of data across multiple sources and target systems.
• Work closely with business stakeholders, data analysts, architects, and engineering teams to understand requirements, define technical solutions, and deliver robust data platforms and pipelines.
• Troubleshoot and optimize data pipelines, Spark jobs, Databricks workloads, and ETL processes, identifying performance bottlenecks and implementing solutions to improve reliability and processing efficiency.
• Provide technical leadership by establishing data engineering best practices, conducting code reviews, mentoring junior engineers, contributing to architectural decisions, and evaluating modern technologies to improve data solutions.