Senior Data Engineer - Azure Databricks & Spark

About Position:

We are seeking an experienced Senior Data Engineer with strong expertise in Azure Databricks, Python, Spark, and cloud-based data engineering solutions. The ideal candidate will be responsible for designing, building, and maintaining scalable data pipelines and distributed data processing frameworks. You will play a key role in supporting enterprise analytics initiatives by developing secure, reliable, and high-performance data solutions on Azure cloud platforms.

• Role: Senior Data Engineer - Azure Databricks & Spark
• Location: All Persistent Locations
• Experience: 6 to 9 Years
• Job Type: Full-Time Employment

What You'll Do:

• Design, develop, and maintain distributed data pipelines using Python and Spark.
• Build and manage large-scale data processing workloads on Azure Databricks.
• Develop scalable ETL/ELT frameworks for processing structured and unstructured data.
• Manage data storage and cloud deployments using Azure Storage, Azure Container Registry (ACR), and Azure Kubernetes Service (AKS).
• Orchestrate workflows and automate pipeline execution using Apache Airflow and internal scheduling frameworks.
• Design and implement high-performance Spark applications using PySpark.
• Maintain source code repositories using GitHub and document technical solutions and processes in Confluence.
• Implement secure data engineering practices including encryption, access controls, credential management, and sensitive data handling.
• Support vulnerability assessment and remediation activities using tools such as CodeQL, Cycode, and related security scanning solutions.
• Optimize data pipelines and Spark workloads for performance, scalability, and reliability.
• Collaborate with architects, developers, analysts, and business stakeholders to deliver enterprise data solutions.
• Participate in Agile ceremonies including sprint planning, daily stand-ups, code reviews, and release activities.
• Monitor production environments, troubleshoot issues, and perform root-cause analysis.

Expertise You'll Bring:

• 6-9 years of experience in Data Engineering and Big Data Technologies.
• Strong hands-on expertise in Azure Databricks, Python, Spark, and PySpark.
• Experience developing enterprise-scale distributed data processing applications.
• Strong proficiency in Python for data engineering, automation, orchestration, and pipeline development.
• Expertise in Spark for large-scale batch and distributed data processing.
• Experience implementing secure data access controls and handling sensitive data in enterprise environments.
• Hands-on experience with Azure Kubernetes Service (AKS), Azure Container Registry (ACR), and Azure Storage services.
• Experience orchestrating workflows using Apache Airflow or similar workflow management platforms.
• Knowledge of data security standards, encryption mechanisms, and secure credential management practices.
• Experience using GitHub for source code management and collaborative development.
• Familiarity with vulnerability scanning and remediation tools such as CodeQL and Cycode.
• Strong understanding of distributed computing, cloud-native architectures, and modern data engineering practices.
• Excellent analytical, debugging, troubleshooting, and problem-solving capabilities.
• Ability to work effectively in Agile environments and collaborate with cross-functional teams.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent - persistent.com/careers

Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind.