Senior Data Engineer

Day to day:

Insight Global is looking for an experienced Snowflake Data Engineer to design, build, and optimise modern data pipelines and cloud-based analytical environments. The ideal candidate will have strong expertise in SQL, Python, Snowflake features (Warehouses, Streams, Tasks, Time Travel), and cloud platforms such as Azure. This role will support enterprise data initiatives, ensure high performance, and enable advanced analytics including AI/ML integrations.

Key Responsibilities

Snowflake Development & Engineering

• Develop high‑performance SQL including advanced ANSI SQL, stored procedures, and user-defined functions.
• Build and manage Snowflake virtual warehouses, ensuring correct sizing, scaling, and multi‑cluster configurations.
• Optimise Snowflake workloads using clustering keys, caching strategies, warehouse configuration, and query tuning.
• Implement Streams and Tasks for real‑time ingestion, change data capture (CDC), and automated scheduling.
• Leverage Time Travel and Fail‑Safe features for data recovery, auditing, and historical analysis.

Data Pipelines & Integration

• Build and maintain ETL/ELT pipelines, including Snowflake ingestion using Snowpipe.
• Use Python for scripting, workflow automation, and in‑database processing through Snowpark.
• Integrate Snowflake with cloud storage, networking, and compute resources across Azure and AWS.

Data Architecture & Warehousing

• Apply strong data warehousing principles and dimensional modelling techniques.
• Ensure data models support analytics, reporting, and scalable business intelligence use cases.

Security, Governance & Compliance

• Implement data governance practices including RBAC, data access controls, encryption, and secure data sharing.
• Configure secure data exchange using Snowflake Secure Data Sharing.

AI/ML & Emerging Technologies

• Integrate Snowflake with AI/ML solutions, including Snowflake Cortex and large language model (LLM) workflows.
• Use generative AI tools to automate data engineering and accelerate development tasks.

DevOps/DataOps

• Build and maintain CI/CD pipelines supporting data engineering workflows.
• Automate deployment, testing, and environment management using modern DevOps/DataOps practices.

Must Haves:

8 + years of experience in data engineering

Prove experience creating pipelines using Azure Data Factory

Strong hands-on experience in ETL/ELT orchestration and production operations

Hands on Snowflake experience

Proficiency with SQL: Advanced ANSI SQL, stored procedures, functions

Proficiency with Python: For scripting, automation, and using Snowpark for in-database processing

Experience building data pipelines using tools like Snowpipe

Data modelling experience

Expertise with Azure, AWS (storage, networking)

Strong DevOps/DataOps experience (CI/CD, automation)

Experience with OLTP databases such as PostgresDB or MS SQL Server or Azure SQL

Pluses:

Performance Tuning: Caching, clustering keys, virtual warehouses

Data Governance: Role-Based Access Control (RBAC), data sharing, encryption

Virtual Warehouses: Sizing, scaling, multi-cluster warehouses

Streams & Tasks: Change data capture and scheduling

Time Travel & Fail-Safe: Data recovery and historical access

Data Sharing: Secure data exchange

AI/ML Integration: Using Snowflake Cortex, integrating LLMs

Generative AI Tools: Leveraging AI for data tasks