Lead Data Engineer

Lead Data Engineer Position Overview
Have you ever wanted to be part of something BIG
Now is the time to make an immediate impact at a leading global technology company, Mastercard.
This role is part of the AIDPE Data Engineering platform team, responsible for building and evolving large scale backend data systems, real time and batch pipelines, and AI enabled services that power analytics, decisioning, and automation across the organization.
This is a backend engineering role, focused on Scala/Python/Java, distributed data platforms (Cloudera/Spark), and cloud based architectures, with a strong emphasis on AI agent creation and intelligent automation. Visualization tools (e.g., Qlik) are consumers of the platform, not the core focus of this role.
You will work with massive transactional datasets, modern big data and cloud platforms, and AI driven workflows to transform how Mastercard processes, enriches, and operationalizes data at global scale.

PRIMARY RESPONSIBILITIES
Backend Data Platform Engineering

Design, develop, and maintain backend services and data pipelines using Scala, Python and Java
Build and optimize batch and streaming workloads on Cloudera Data Platform (CDP) using Spark
Work with Cloudera Manager to support platform configuration, monitoring, performance tuning, and operational stability
Design and implement high quality datamarts and curated datasets with strong emphasis on data integrity, performance, and reliability

AI Agents Intelligent Automation

Design, build, and integrate AI powered agents that operate to support:

Anomaly detection and operational intelligence
workflow automation

Apply generative AI driven, or rule based agents to reduce manual effort and improve scalability across backend systems

Cloud Modern Data Architecture

Build and support data and compute workloads in AWS environments
Leverage Databricks for large scale data processing, advanced analytics
Contribute to cloud native and hybrid architectures integrating on prem and cloud platforms

Integration Downstream Enablement

Enable downstream consumers (analytics, visualization, reporting tools such as Qlik) through well designed, reliable backend data interfaces
Partner with analytics, fraud, and business teams to ensure backend systems meet evolving needs without compromising platform stability

Documentation Collaboration

Create clear technical documentation, including architecture diagrams, data flows, and design specifications
Participate in Agile/Scrum ceremonies and cross functional design reviews
Mentor and upskill team members in backend engineering, big data, and AI agent concepts

KNOWLEDGE AND SKILL REQUIREMENTS
Required

BS/BA degree in Computer Science, Engineering, Information Systems, or related field
Strong handson experience with Scala,Python/Java in backend or data intensive systems
Experience working with Cloudera Data Platform (CDP) and Spark
Familiarity with Cloudera Manager for cluster administration, monitoring, or troubleshooting
Strong understanding of data modeling concepts, distributed systems, and large scale data processing
Excellent problem solving skills and ability to work independently in complex environments

GOOD TO HAVE / STRONGLY PREFERRED

Hands on experience building, integrating, or supporting AI driven agents or intelligent automation solutions
Experience with Databricks for data engineering or ML workloads
Experience working in AWS (e.g., S3, EC2, EMR, Glue, Lambda, IAM, or equivalent services)
Knowledge of streaming and big-data technologies:
Kafka
Hadoop ecosystem
Hive/Impala
Exposure to model monitoring, or AI platform enablement
Experience with ETL tools such as Informatica
Experience working in Agile / Scrum teams within large enterprises