Senior Data Engineer

Responsibilities

• Strong Skill in Big Query, GCP Cloud Data Fusion (for ETL/ELT) and PowerBI.
• Need to have strong skill in Data Pipelines
• Able to work with Power BI and Power BI Reporting
• Design and implement the data engineering framework and data pipelines using Databricks

and Azure Data Factory.

• Document the high-level design components of the Databricks data pipeline framework.
• Evaluate and document the current dependencies on the existing DEI toolset and agree a

migration plan.

• Lead on the Design and implementation an MVP Databricks framework.
• Document and agree an aligned set of standards to support the implementation of a

candidate pipeline under the new framework.

• Support integrating a test automation approach to the Databricks framework in conjunction

with the test engineering function to support CI/CD and automated testing.

• Support the development teams capability building by establishing an L&D and knowledge

transition approach.

• Support the implementation of data pipelines against the new framework in line with the

agreed migration plan.

• Ensure data quality management including profiling, cleansing and deduplication to support

build of data products for clients

Skill Set

Experience working in Azure Cloud using Azure SQL, Azure Databricks, Azure Data Lake, Delta

Lake, and Azure DevOps.

• Proficient in Python, Pyspark and SQL coding skills.
• Profiling data and data modelling experience on large data transformation projects creating

data products and data pipelines.

• Creating data management frameworks and data pipelines which are metadata and business

rules driven using Databricks.

• Experience of reviewing datasets for data products in terms of data quality management and

populating data schemas set by Data Modellers

• Experience with data profiling, data quality management and data cleansing tools.