Support Data Engineer - Spark

Role & responsibilities

Job Summary

As a Software Engineer for DataPelago data processing customer success, you will develop workload assets and automation tools for product validation and product support which will help ensure customers deploy and operate DataPelago data processing products successfully in production. You will apply AI effectively in combination with the knowledge from both product validation and product support to proactively avoid bugs and issues and to drastically reduce time to diagnose, mitigate, and resolve bugs and issues.

Job Requirements

What You'll Do:

• Develop synthetic workload queries and datasets for testing DPA-S, which reflect the essential environment and workload characteristics represented in the field.
• Develop analysis and diagnostic tools that analyze various logs from DPA-S, Apache Spark, and their deployment environment to aid in pinpointing the root cause of functional and performance anomalies and potential mitigations and resolutions.
• Field and address or escalate customer support issues as part of the product support rotation.
• Conduct end-to-end product validation, collaborating with others to define, review, and execute test plans.
• Analyze artifacts from product validation testing and issues raised by customers to identify potential configuration tuning opportunities, product usage recommendations, and product bugs.
• Leverage automation and AI techniques effectively in the entire process of product validation and product support.
• Create bug reports, RCA documents, configuration guides, and best practice documents.

What You'll Bring:

• 5+ years of experience related to development, validation and support of enterprise data processing software products.
• 2+ years of experience providing direct support to external customers.
• 2+ years of experience supporting Apache Spark pipelines for data engineering, analytics. Proficient in running Spark jobs from various clients, Spark resource provisioning, Spark parameter configuration and tuning, using Spark History Server and Spark UI, analyzing Spark event logs and other Spark logs.
• Solid knowledge of SQL and Python.
• Proficient in testing data products and test automation.
• Good understanding of the deployment, and operations of data processing products and pipelines in the cloud in SaaS, VM-based, container-based, and serverless scenarios. Experience with GCP, AWS, and their Apache Spark related offerings preferred.
• Good customer engagement and communication skills.