Azure Data Engineer

Job description

Must have skills Databricks ADF SQL Python

Nice to have skills GenAI Power BI Exposure to AI related Projects

Years of experience 7

Key Responsibilities

Design and implement reusable cloud data connectors and extraction patterns for Microsoft Fabric Snowflake Databricks Azure Synapse BigQuery and comparable platforms

Build data workflows for direct platform connections client data warehouse copies mirrored or replicated ERP databases and structured or unstructured document sources

Develop the ERPindependent data warehouse and databasemirroring blueprint with standard configuration validation governance and operational controls

Evaluate native platform connectors and integrate them with the EY Cloud Connector Suite through standard APIs events and configuration contracts

Implement extraction controls including schema capture rowcount reconciliation source and destination validation restartability exception handling and execution traceability

Develop scalable patterns for incremental extraction change data capture partitioning parallel processing pagination and highvolume data movement

Capture and preserve extraction parameters schemas selected objects run history technical metadata and evidence required for auditability

Create automated integration dataquality performance volume resilience and security tests

Produce implementation guides configuration references technical design documents operational runbooks and connectordevelopment documentation

Required Technical Skills

Strong handson experience with at least two modern data platforms such as Microsoft Fabric Azure Data Factory Databricks Snowflake Azure Synapse Analytics

Proficiency in SQL and one or more development languages such as Python C Java Scala or TypeScript

Experience developing data pipelines reusable connectors integration APIs or enterprise ingestion frameworks

Good knowledge of relational databases data warehouses lakehouses object storage and enterprise data formats

Experience with REST APIs JDBCODBC connectivity filebased interfaces and cloudnative connector frameworks

Experience with incremental loads change data capture partitioning batching pagination retries resumability and performance optimisation

Ability to capture schemas metadata lineage run information dataquality outcomes and sourcetotarget reconciliation evidence

Knowledge of secure network and identity patterns including private endpoints firewall controls managed identities service principals and encrypted transport

Experience with CICD source control automated testing and Infrastructure as Code

Strong troubleshooting capability across data authentication connectivity connector and platformperformance issues

Preferred GoodtoHave Skills

Experience with Microsoft Graph and SharePoint APIs for secure document or invoice retrieval

Knowledge of database replication mirroring read replicas change data capture or logbased extraction

Familiarity with Delta Lake Apache Spark Parquet JSON CSV and XML

Experience with ERPderived financial data such as general ledger subledger master and transactional datasets

Understanding of auditability evidence generation sourcetotarget reconciliation and data lineage

Experience assessing vendornative connectors through structured proofs of concept

Microsoft Fabric Azure Data Engineer Databricks or Snowflake certification

Professional Attributes

Strong analytical and problemsolving skills with attention to data completeness performance and traceability

Ability to separate reusable product capabilities from clientspecific configuration and implementation needs

Comfortable evaluating vendor capabilities and documenting constraints decisions and technical tradeoffs objectively

Effective collaboration with cloud engineers ERP specialists security teams product owners and auditdata stakeholders

Ownership mindset for developing testing documenting and supporting productiongrade connector capabilities

Skills

Mandatory Skills : Azure BLOB, Azure Data Factory, Azure SQL, Azure Synapse Analytics, Databricks
Good to Have Skills : AZURE DATA LAKERole & responsibilities

Preferred candidate profile