Pyspark Developer

Technical Skills:

• Experience in design, development and deployment of big data applications and ETL jobs using PySpark APIs/sparkSQL.
• Experience in design, build and deployment of Python based applications.
• Experience in writing complex SQL queries/procedures using Relational databases like SQL Server or Oracle
• Experience in version control system like Git and CI/CD pipeline is a must.
• Experience in Delta lake APIs is a plus
• Experience in Docker and Kubernetes is a plus
• Knowledge of AWS services like S3, Athena, Glue, Lambda, Redshift or Cloud platform is a plus

Responsibilities:

• Designing, implementing, and maintaining jobs written in PySpark for extracting raw data, applying transformation, and writing into file/tables on Apache Spark cluster.
• Writing complex queries as per the business rules and data extraction logic
• Query tuning and Performance optimization of various components as and when required.
• Work with development leads, System Architects and other teams (as needed) to manage dependencies, risks and issues.
• Contributing in all phases of the development lifecycle
• Support testers and defect resolution
• Update status/risks/issues/impediments in daily scrum calls to the customer and track it to closure.
• Writing testable, scalable and efficient code
• Working on the necessary documentations as needed