Principal Data Engineer

Role Summary Join the Data Commercialization Platform (DCP) engineering team to design and build cloud-based architectures for Business Intelligence and analytics, leveraging Databricks and Snowflake on AWS, with multi-cloud extension patterns into Azure. You will lead platform-grade solutions that are governed, scalable, and interoperable, using open data standards and enterprise security controls.

What You’ll Do (Key Responsibilities) • Design and develop cloud BI/analytics architecture on AWS Databricks and Snowflake, enabling governed analytics, ML, and data engineering workloads at scale. • Drive multi-cloud architecture patterns by extending DCP capabilities into Azure, including Azure Databricks (ADB) and ADLS Gen2, while maintaining an engine-agnostic posture. • Lead cross-platform interoperability across Databricks, Snowflake, and AWS data services (e.g., catalog federation and governed access patterns), ensuring seamless end-user experiences and consistent security controls • Participate in architecture reviews, iteration planning, and feature sizing with technical partners across Mastercard, shaping platform roadmaps and execution plans. • Define and deliver service-level procedures and platform solutions (automation, provisioning, repeatability), improving reliability, speed-to-onboard, and operational excellence. • Build roadmaps and identify improvement opportunities for primary platform services (cost optimization, governance, automation, resiliency, multi-region strategy. • Introduce new technologies and architectures aligned with enterprise standards, emphasizing open formats (Iceberg/Delta) and avoiding vendor lock-in. • Own requirements management and project planning/control for platform initiatives, including delivery coordination and operational readiness. • Enable secure access and automation patterns (service principals, role-based controls, separation of duties) across Databricks, Snowflake, and AWS. • Support BI enablement on Databricks, including approved capabilities like Databricks AI/BI, driving governed self-service analytics adoption.

Core Platform & Technology Scope (Must-Have Exposure) • Databricks (AWS + Azure Databricks) — analytics/ML, lakehouse patterns, governed catalogs and platform enablement. • Snowflake — distribution/consumption platform patterns and multi-platform coexistence with Databricks. • AWS — platform services supporting analytics and governance (e.g., data lake + orchestration + compute patterns). • Azure — DCP multi-cloud extension patterns (ADB + ADLS Gen2) for regulated/client-driven needs.

Required Qualifications • Proven experience designing cloud analytics / BI architectures for large-scale data platforms. • Hands-on expertise across Databricks + Snowflake + AWS, with demonstrated ability to integrate governed services across platforms. • Working knowledge of Azure architectures and patterns for extending analytics platforms (including Azure Databricks and ADLS Gen2). • Strong architecture and engineering leadership: ability to drive standards, automation, and repeatable delivery • Experience operating in enterprise governance/security constraints, implementing access control and automation identities (service principals).

Preferred Qualifications (Nice to Have) • Familiarity with open table formats and interoperability patterns (e.g., Iceberg/Delta) in multi-engine environments. • Experience building or operating platform automation (workspace provisioning, access automation, CI/CD enablement)