Senior/Lead Azure Data Engineer

KEY RESPONSIBILITIES

• Design & implement governed ingestion pipelines consuming on-site & off-site events into canonical

event schema

• Build and maintain Kafka or Kinesis-based streaming pipelines with schema validation, data quality

checks, and alerting on source failures

• Integrate batch ingestion from databases and alongside the streaming path, managing orchestration

via Airflow or equivalent

• Enforce privacy flags at ingestion time quarantining or anonymising events without valid consent

before they reach downstream layers

• Maintain a raw event store with partitions by date and source, serving as the audit source for the

pipeline

• Implement deterministic identity across fragmented systems
• Build and maintain the cookie-to-login stitching mechanism
• Design identity logic spanning on-site sessions, CRM records etc
• Monitor identity resolution quality match rate, false positive rate, unresolved session ratio — and

iterate on matching logic to improve coverage over time

• Compute windowed aggregations over the event store
• Build and maintain governed, versioned feature definitions in a feature store ensuring normalisation,

encoding, and embedding lookup logic is consistent between training and serving pipelines

• Collaborate with data scientists to translate model feature requirements into production-grade

pipeline implementations with no training-serving skew

• Implement automated data contract validation between pipeline layers, schema compatibility checks,

completeness assertions, and anomaly detection

• Define and enforce SLAs on pipeline freshness, completeness, and accuracy with monitoring

dashboards and escalation paths for breaches

• Ensure PII is classified, tagged, and handled according to GDPR and CCPA requirements at every stage

of the pipeline like ingestion, storage, feature compute, and serving

• Contribute and maintain data lineage document, full traceability from raw event to feature to model

prediction

• Work closely with the Data Architect to implement data contracts and schema agreements between

pipeline layers, and flag design risks early

• Partner with ML Engineers to ensure feature pipelines meet model training and online inference

requirements

• Participate in code reviews, contribute to engineering standards, and mentor junior engineers where

applicable

REQUIRED QUALIFICATIONS

• 6-10 years of hands-on data engineering experience, with at least 2 years in a senior or lead capacity on

production systems

• Demonstrable experience building and operating event-driven, streaming data pipelines at scale in a

production environment

• Prior experience on a personalisation, recommendation, or user behavioural analytics platform is

strongly preferred

• Experience working within a regulated data environment like GDPR, CCPA, or equivalent.
• Batch and structured streaming, including windowed aggregations, stateful processing, and

performance tuning

• Event streaming platforms including producer/consumer design, partition management, and exactlyonce semantics
• Data Lakehouse engineering like Delta Lake, Apache Iceberg, or equivalent, including ACID

transactions, schema evolution, and time travel

• Pipeline orchestration using Apache Airflow, AWS Glue, Azure Data Factory, or Databricks Workflows

including DAG design, dependency management, and failure handling

• SQL and Python at production standard quality with clean, tested, version-controlled code
• Cloud data platform exposure on at least one of: AWS (S3, Glue, Kinesis, Redshift), Azure (ADLS, Data

Factory, Synapse), or Databricks

• Feature store design and operation: Databricks Feature Store, Feast, Tecton, or equivalent
• Data quality frameworks: Great Expectations, dbt tests, or equivalent for automated pipeline

validation

• Deterministic and probabilistic matching, entity deduplication, graph-based stitching
• CI/CD for data pipelines with automated testing, deployment, and monitoring using GitHub Actions,

Azure DevOps, or equivalent

PREFERRED QUALIFICATIONS

• Experience with graph data modelling and graph processing frameworks — Neo4j, Amazon Neptune,

GraphX, or GraphFrames

• Familiarity with vector embedding pipelines - batch encoding of text at scale, embedding storage, and

integration with ANN search infrastructure

• Working knowledge of NLP pipeline engineering — tokenisation, embedding generation, and chunking

for unstructured text at scale

• Experience with real-time feature serving and integration like Redis, DynamoDB, or equivalent cache

stores

• Familiarity with MLflow or equivalent model registry — understanding of how feature pipelines

connect to model training and deployment workflows

• IaC experience like Terraform, Bicep, or AWS CDK for reproducible data infrastructure deployment
• Experience with data mesh or federated data architecture patterns in multi-team environments
• Strong written and verbal communication — able to explain complex pipeline design decisions to nonengineering stakeholders clearly
• Able to make design decisions where requirements are incomplete and flag risks proactively
• Collaborative working style with cross-functional teams spanning data engineering, ML, platform, and

product

• Understanding that interfaces between pipeline layers are as important as the implementations within

them

• Takes responsibility for pipeline reliability, DQ, and SLA adherence end-to-end, not just the code

written

WHAT WE OFFER

• Awesome Culture: Creative Synergies has a flat organization and an agile culture of positivity,

entrepreneurial spirit, customer centricity, celebrating technical excellence, teamwork, and

meritocracy

• Opportunity to work with Customers who are technology Leaders (including Global Fortune 500

Customers) & work on Real-World Problems that matter and are often mission-critical

• Leadership role with significant influence over AI strategy and team direction.
• Access to state-of-the-art GPU infrastructure and cutting-edge AI tools.
• Competitive compensation package with performance-based incentives.
• Flexible working arrangements with hybrid options.
• Continuous learning budget for conferences, courses, and certifications