Data Engineer

Your key responsibilities

• Design, develop, and maintain robust data pipelines to process and analyze large datasets.
• Utilize SQL for data extraction, transformation, and loading (ETL) processes.
• Work with Data Bricks and Data Lake technologies to manage and optimize data storage and processing.
• Collaborate with security teams to understand data requirements and integrate data from various security tools.
• Create and maintain documentation for data pipelines, processes, and workflows.
• Monitor and troubleshoot data pipeline performance and data quality issues.
• Develop data models and visualizations to present security insights to stakeholders.
• Stay updated with industry trends and best practices in data engineering and security analytics.

Skills and attributes for success

• Conceptual knowledge of security applications and technologies, as well as operating system platforms including Cloud technologies
• Proven experience as a Data Engineer or in a similar role.
• Strong proficiency in SQL and experience with data manipulation and querying.
• Hands-on experience with Data Bricks and Data Lake technologies.
• Familiarity with security tools and frameworks for data extraction.
• Knowledge of data modelling, ETL processes, and data warehousing concepts.
• Excellent problem-solving skills and attention to detail.
• Proficiency in Azure Cloud services, with a strong understanding of Azure security tools such as Azure Security Center, Azure Monitor, and Azure Sentinel
• Build and maintain strong relationships across all levels of a matrixed, geographically and culturally dispersed organization
• Strong communication and collaboration skills.
• Possess knowledge of the Information Technology Infrastructure Library (ITIL) to drive compliance in Incident, Problem, and Change processes and drive as necessary

To qualify for the role you must have

• Degree in Computer Science or related discipline, or equivalent work experience
• 7-10 years working within Information Technology in a global firm
• Knowledge of Azure Data Lake, Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Notebooks, PySpark, Spark SQL
• Working knowledge of data analysis programming or scripting languages, such as Python (with libraries like pandas, NumPy) or PowerShell
• Experience with Azure SQL and database management systems for data extraction, transformation, and loading (ETL) processes.
• Certifications in data visualization or analytics platforms.
• Experience with cloud platforms (e.g., AWS, Azure, Google Cloud).
• Knowledge of programming languages such as Python or Scala.
• Familiarity with data visualization tools (e.g. Power BI, Tableau).