Walk-in || Pyspark Data engineer

Role & responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and Python.
• Build ETL/ELT frameworks for ingesting, transforming, and loading large datasets.
• Develop Spark jobs for data cleansing, transformation, aggregation, and validation.
• Optimize Spark applications for performance using partitioning, caching, broadcast joins, and tuning techniques.
• Work with structured and unstructured data from multiple sources.
• Implement batch and real-time data processing solutions.
• Develop and optimize SQL queries, stored procedures, and data models.
• Collaborate with business analysts, architects, and data scientists to deliver data solutions.
• Ensure data quality, governance, security, and compliance standards.
• Implement CI/CD and DevOps practices for data engineering workloads.
• Troubleshoot production issues and provide operational support.