Senior Specialist - Data Engineering

Role: PySpark Application Support Lead Experience: 8-12 Years Location :Anywhere in India Focus Production Support Troubleshooting Optimization Team Leadership Role Summary Leadership Skills and proven experience in leading technical teams handling escalations and conducting knowledgesharing sessions The Pyspark Application support is responsible for the monitoring and troubleshooting Pyspark Jobs on the Cloudera environment This role ensures high availability performance and security of ETL services while supporting developers analysts and business users across environments Prod and Nonprod Technical Skills Primary PySpark Application support Handson experience with Pyspark jobs development ETL support Monitor PySpark batch jobs streaming pipelines and scheduled workflows Airflow ADF Oozie etcChecking for failures and performance degradation Troubleshoot job failures performance degradation schema mismatches data quality issues Resolve errors related to Spark SQL DataFrame transformations cluster configuration and storage layers Handle failures caused by upstreamdownstream systems Kafka Delta Lake Hive S3ADLS etc Coordinating with development team for fixing the failures Coordinating with QA team on data validation failures Supporting in production release and change management process Incident Problem Management Root cause analysis for recurring issues Performance tuning at a basic level cache queues agents Familiarity with ITIL processes for AMS support Tools Ticketing systems ServiceNowJira Monitoring tools Version control basics GitSVN for migration scripts Soft Skills Ability to work with client stakeholders Good communication with analytical and problemsolving skills Ability to Effectively interact with stakeholders colleagues and other members of the team

Demonstrated ability to be organized and selfdirected

Strong interpersonal skills with the ability to work cross functionally

Responsibilities

Monitoring Maintenance

Proactively monitor Informatica environments for uptime and performance

Execute scheduled health checks and housekeeping tasks

Maintain logs and escalate anomalies promptly

Incident Management

Respond to incidents within SLA timelines

Perform initial triage resolve L2 issues and escalate L3 as needed

Document RCA and preventive measures

User Support

Assist users with access issues report failures and portal navigation

Manage user provisioning and security roles

Change Release Support

Support deployments and migrations between environments

Validate postdeployment checks and rollback plans

Compliance Documentation

Ensure adherence to security and audit requirements Maintain SOPs runbooks and knowledge base for AMS operations Production Support Incident Management Act as the primary point of contact for highpriority incidents debugging and resolving complex PySpark application issues in production Technical Leadership Lead the efforts to design build and configure improvements to existing data pipelines and applications Team Management Mentorship Guide mentor and manage a team of support specialistsdevelopers to achieve operational excellence and meet project deadlines Performance Optimization Analyze existing PySpark applications and data pipelines for performance bottlenecks and implement optimization techniques eg Spark UI analysis tuning executors Stakeholder Interaction Collaborate with crossfunctional teams developers Data Engineers DevOps to ensure seamless application functionality Documentation Best Practices Create maintain and update technical documentation standard operating procedures SOPs and ensure code compliance with best practices Automation Implement monitoring improvements and automate recurring manual tasks