PySpark Big Data Developer

We are seeking a highly skilled and experienced BigData/PySpark Engineer to join our dynamic Big Data Analytics team. This role is pivotal in designing, developing, and optimizing robust, scalable data pipelines for large-scale data processing and analytics. Key Responsibilities: Design & Development: Create and optimize scalable ETL (Extraction, Transformation, Loading) pipelines using PySpark for massive datasets. Coding & Engineering: Write clean, efficient, well-documented code primarily in Python (PySpark) often leveraging frameworks/tools. Collaboration: Work with cross-functional teams (senior developers, data engineers, analysts, business partners) to understand data requirements and ensure seamless solution integration. Troubleshooting & Optimization: Debug and resolve data processing issues and performance bottlenecks in Spark applications and other big data technologies. Full SDLC Involvement: Participate in the entire software development lifecycle, from requirements analysis and design to testing, deployment, and operations. Data Integrity: Ensure high data quality and integrity throughout the data lifecycle.

This candidate possesses  4-8 years of experience  in developing and managing Enterprise Applications, demonstrating a robust foundation in Big Data technologies and a strong grasp of software development principles. Key Experience & Expertise: Enterprise Application Development:  4-8 years in developing and managing enterprise-grade applications. Object-Oriented Programming (OOP):  Solid foundation in OOP concepts. Big Data Development:  Expertise in  PySpark, HDFS, Hive, Sqoop, and Hadoop  for Big Data environments. Database Technologies:  Good exposure to  SQL Server  and  ORACLE  databases. Experience with query writing for data validation/manipulation Scripting & Automation:  Proficient in  Shell Scripting  and experience with job scheduling tools like  Autosys . BI Reporting Tools:  Some exposure to BI tools, specifically  Tableau . Tools & Practices:  Proficient with Git; experience with JIRA, Confluence. Familiarity with DevOps and CI/CD pipelines.

Job Family Group: Technology

Job Family: Applications Development

Time Type: Full time

Most Relevant Skills Please see the requirements listed above.

Other Relevant Skills For complementary skills, please see above and/or contact the recruiter.

Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.   If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi .

View Citi’s EEO Policy Statement and the Know Your Rights poster.