Python+ETL+Pyspark

Key Responsibilities:

• ETL Development
• Design develop and maintain ETL pipelines using Python and PySpark
• Extract transform and load data from multiple structured and unstructured sources
• Build reusable and scalable data processing frameworks
• Ensure data quality validation and consistency
• PySpark Big Data Processing
• Develop and optimize PySpark jobs for large scale data processing
• Work with Spark DataFrames and RDDs
• Implement transformations aggregations and joins in Spark
• Optimize jobs for performance and scalability
• Python Development
• Develop backend logic and data processing scripts using Python
• Write modular reusable and efficient code
• Integrate APIs and automate workflows
• Data Management Integration
• Work with data lakes and warehouses S3 HDFS Redshift Hive
• Handle file formats like Parquet ORC JSON CSV
• Perform data cleansing enrichment and transformation
• Collaboration Support
• Work with data engineers analysts and business stakeholders
• Debug and troubleshoot ETL data pipeline issues
• Participate in Agile Scrum ceremonies
• Maintain documentation and coding standards

Technical Requirements:

• Primary skills Domain Finacle Core Functional Finacle Core WMS Grand Master Technology ETL Data Quality ETL Others Technology Machine Learning Python

Additional Responsibilities:

• Core Skills
• 2 5 years of experience in Python and ETL development
• Hands on experience with PySpark mandatory
• Strong understanding of data processing and pipelines
• Solid knowledge of SQL and database concepts
• Technical Skills
• Experience with Apache Spark ecosystem
• Good knowledge of data structures and algorithms basic to intermediate
• Familiarity with Big Data technologies Hadoop Hive
• Experience with version control Git
• Understanding of REST APIs and integrations

Preferred Skills:

Technology->AI-Data science->PYTHON,Technology->Big Data - Data Processing->PySpark,Technology->ETL & Data Quality->ETL - Others