Senior Data Engineer

About Position:

We are seeking a highly skilled Senior Data Engineer with strong expertise in Python, PySpark, Advanced SQL, and AWS to design, build, and maintain enterprise-scale data platforms and processing solutions. The ideal candidate will have hands-on experience developing both batch and streaming data pipelines, optimizing large-scale data workloads, and leveraging AWS cloud services to deliver reliable, scalable, and high-performance data solutions. You will play a critical role in enabling data-driven decision-making by building robust data engineering frameworks and modern data platforms.

• Role: Senior Data Engineer Python, PySpark & AWS
• Location: Pune, Bengaluru
• Experience: Between 6 to 12 Years
• Job Type: Full Time Employment

What You'll Do:

• Design, develop, and maintain scalable batch and real-time data pipelines using Python, PySpark, and Apache Spark.
• Build and optimize data ingestion, transformation, and processing frameworks for large-scale enterprise data environments.
• Develop and support streaming solutions using Spark Streaming and Spark Structured Streaming.
• Create efficient ETL/ELT pipelines to move, process, and transform data across multiple systems and platforms.
• Implement data quality controls, validation frameworks, and monitoring mechanisms to ensure data accuracy and reliability.
• Design and optimize advanced SQL queries, stored procedures, and data transformation logic.
• Leverage AWS services such as EMR, S3, Lambda, EC2, and Athena to build cloud-native data solutions.
• Collaborate with business stakeholders, data scientists, analysts, and application teams to understand and deliver data requirements.
• Troubleshoot and resolve performance bottlenecks in data pipelines and Spark workloads.
• Improve data platform scalability, performance, reliability, and operational efficiency.
• Participate in architecture discussions and contribute to data platform modernization initiatives.
• Follow engineering best practices, coding standards, and CI/CD processes.
• Develop technical documentation and support production deployments and monitoring activities.

Expertise You'll Bring:

• 6 to 12 years of experience in Data Engineering, Big Data, or Data Platform Development.
• Strong hands-on expertise in Python programming and data engineering frameworks.
• Extensive experience with PySpark and Apache Spark application development.
• Deep understanding of Spark architecture, distributed computing concepts, and performance optimization techniques.
• Hands-on experience implementing batch and streaming data processing solutions.
• Strong knowledge of Spark DStreams and Spark Structured Streaming.
• Expertise in building scalable ETL/ELT pipelines and modern data processing frameworks.
• Advanced SQL skills with experience writing and optimizing complex queries.
• Hands-on experience with CTEs, Window Functions, Nested Queries, Joins, and Query Performance Tuning.
• Strong experience with AWS services including EMR, S3, EC2, Lambda, Athena, and related cloud-native services.
• Experience designing and supporting cloud-based data platforms and data lakes.
• Strong troubleshooting, debugging, and analytical problem-solving abilities.
• Understanding of data modeling, data governance, and data quality principles.
• Experience working in Agile development environments.
• Excellent communication and stakeholder collaboration skills.
• Bachelors degree in Computer Science, Information Technology, Engineering, or a related discipline.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment.

Let's unleash your full potential at Persistent - persistent.com/careers

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."