Principal Data Engineer

Role & responsibilities

Preferred candidate profile

JOB DESCRIPTION

Principal Data Architect / Data Engineering AWS / Azure DEX Platform

Experience

8–10 years

Role type

Highly technical individual contributor / senior engineering role

Platform

Enterprise DEX data platform on AWS, or equivalent Azure cloud data-platform experience

Reporting line

Reports to the DEX Platform Lead

Role purpose

Design, build and continuously improve the DEX cloud data platform through deep hands-on engineering and technical architecture. The current platform is on AWS; strong AWS experience is preferred, while equivalent Azure cloud data-platform experience will also be considered. This role turns platform architecture and DEX 2.0 medallion goals into robust data pipelines, reusable platform patterns and production-ready implementations. The Principal Data Architect / Data Engineer will be a technical authority and hands-on contributor—working closely with the DEX Platform Lead, platform engineers and central AI / Cloud teams to deliver secure, scalable, reliable data capabilities and enable AI initiatives. This is an individual-contributor engineering role, not a people-management position.

Platform context

DEX ingests data from various on-premises and cloud-based source systems into cloud landing and processing layers. The current platform uses AWS services including Amazon S3, AWS EMR with Apache Spark, AWS Glue Data Catalog, Amazon Redshift, Amazon AppFlow and Amazon CloudWatch, alongside Apache Airflow, data quality and governance capabilities, masked views and data sharing. Candidates with equivalent Azure data-platform experience (for example, Azure Data Lake Storage, Azure Databricks / Spark, Microsoft Purview, Azure Data Factory and Synapse Analytics or equivalent services) will also be considered, with the ability to apply transferable cloud data-engineering concepts. The platform supports analytics, data marts, operational data products, regulatory reporting, BI and data science. The role contributes hands-on engineering and technical design across this lifecycle and the evolution toward DEX 2.0.

Key responsibilities

Hands-on data platform engineering

• Design, develop, test, tune and operate production-grade data pipelines and reusable components on AWS or Azure, with strong hands-on ownership from source ingestion through curated data and serving; apply equivalent cloud-native services and patterns to the DEX environment.
• Build distributed batch data processing using Apache Spark on AWS EMR or Azure Databricks / equivalent Spark services; optimize job logic, partitioning, file sizing, parallelism, shuffle, memory, compute utilization and runtime cost.
• Design Amazon S3 data layouts and lifecycle practices, including partition strategy, storage formats, encryption, retention, access controls and performance considerations.
• Implement and support ingestion integrations from on-premises and cloud-based systems using applicable push/pull, file transfer, JDBC, AWS AppFlow and comparable enterprise patterns.
• Develop and optimize Amazon Redshift data models and workloads, including SQL, schema design, data loading, performance tuning, workload management, secure access and data sharing.
• Use AWS Glue Data Catalog and related metadata capabilities to support discoverability, schema management, lineage and governed data access.
• Write clean, modular, maintainable code; conduct code reviews; build unit, integration, data-quality and regression tests; and troubleshoot production issues.

DEX 2.0 medallion and lakehouse engineering

• Contribute to the technical design and implementation of DEX 2.0 using Bronze, Silver and Gold (or equivalent governed) layers on AWS; define and implement layer-specific data contracts, transformations, quality checks and promotion criteria.
• Help map current RAW, TRANSFORM, temporary, business-layer and data-mart workloads to the target medallion design; implement migration, reconciliation, dual-run and cutover patterns while maintaining existing consumers.
• Implement reliable schema and partition evolution, incremental processing, idempotency, replay/recovery, late-arriving data handling and data versioning where appropriate.
• Evaluate and implement open table formats such as Apache Iceberg where appropriate, including catalog integration, snapshots/time travel, schema and partition evolution, compaction and interoperability across engines.
• Build reusable patterns for data quality, metadata, lineage, security, access and observability across medallion layers; document technical standards and reference implementations.

Orchestration, DevOps and production reliability

• Build and maintain Apache Airflow DAGs and workflow integrations, applying reusable patterns for dependencies, retries, backfills, scheduling, alerting and operational ownership.
• Instrument pipelines and platform components with logs, metrics and alerts using Amazon CloudWatch and complementary platform tools; monitor data freshness, quality, throughput, failures and cost.
• Contribute to CI/CD, infrastructure as code, environment configuration and release automation using approved tools; apply secure coding, secrets management, testing and rollback practices.
• Participate in production support, incident response and root-cause analysis; create runbooks, resolve recurring failure modes and improve service reliability, recovery and operational readiness.
• Apply AWS security best practices including least privilege, role-based access, encryption, network controls, secure credentials, auditability and sensitive-data protection.

Architecture collaboration and AI enablement

• Work with the DEX Platform Lead to translate target architecture and non-functional requirements into detailed technical designs, implementation plans, estimates and engineering decisions.
• Collaborate with platform, data, application, security, operations and governance engineers; present technical options and trade-offs clearly and contribute to architecture and code reviews.
• Partner with the central AI / Cloud teams to enable AI initiatives through fit-for-purpose data architecture, trusted and discoverable datasets, secure access, reliable data pipelines and production integration.
• Assess data readiness for AI use cases, including quality, timeliness, lineage, privacy and access controls; contribute to operationalization and monitoring of data products that support AI workloads.
• Proactively identify platform bottlenecks, technical debt and modernization opportunities; validate recommendations with prototypes, benchmarks or production evidence.

Required qualifications and experience

• 8–10 years of relevant experience in data engineering, cloud data platforms or data architecture, with substantial hands-on delivery and production ownership.
• Strong practical data-platform experience on AWS or Azure. AWS experience is preferred and should include services such as Amazon S3, AWS EMR, Apache Spark, Amazon Redshift, AWS Glue Data Catalog and Amazon CloudWatch; candidates with Azure equivalents such as ADLS, Databricks / Spark, Synapse Analytics, Azure Data Factory and Microsoft Purview will also be considered.
• Advanced proficiency in SQL and a programming language used for data engineering, such as Python, Scala or Java; strong debugging, testing and code-review practices.
• Demonstrated experience building and operating scalable ETL / ELT pipelines, data warehouse or lakehouse solutions, and distributed data processing workloads.
• Solid understanding of data modelling, file formats, partitioning, catalogues, schema evolution, incremental processing, performance tuning and cost optimization.
• Experience with workflow orchestration such as Apache Airflow and CI/CD or infrastructure-as-code practices for data platforms.
• Hands-on understanding of cloud security, privacy, data governance, data quality and production support in an enterprise environment.
• Ability to work as a senior individual contributor: take technical ownership, mentor through engineering practice, collaborate across teams and deliver with limited supervision.
• Clear written and verbal communication, including the ability to document designs and explain technical decisions.

Preferred / value-adding experience

• Apache Iceberg or another open table format, including practical lakehouse implementation and interoperability considerations.
• Experience designing or implementing medallion (Bronze / Silver / Gold) or equivalent layered data architectures and migrating legacy pipelines.
• Experience with AWS AppFlow, JDBC, managed file transfer or comparable source-ingestion tools and patterns.
• Experience with Redshift data sharing, BI-serving workloads, enterprise data products or regulatory / audit-sensitive data.
• Exposure to enterprise AI / ML data enablement, including preparation and governed delivery of trusted data for AI use cases.
• AWS or Azure certification in data engineering, architecture or a related discipline.

Success measures

Success will be measured by the quality, reliability and performance of implemented data pipelines; effective contribution to DEX 2.0 medallion delivery; improved data quality, freshness, observability and recovery; secure and reusable AWS data patterns; measurable performance and cost improvements; and timely technical enablement of platform and AI initiatives.

Role attributes

Highly technical • Hands-on builder • Strong engineering discipline • Analytical problem-solver • Architecture-aware • Collaborative • Production-minded • Security-conscious • Continuous learner