Data Engineer

Responsibilities:

• Design, develop, and maintain robust data pipelines using Airflow, DBT, and Python within a BigQuery environment.
• Implement and maintain data integrations with various sources and targets using tools like Airbyte, Elementary, and Metaflow, with a focus on optimizing data storage and retrieval in BigQuery.
• Write advanced SQL queries and scripts for data manipulation and retrieval in BigQuery.
• Develop and optimize data models and schemas in Preset and other visualization tools, tailored for performance in BigQuery.
• Utilize Pyspark for big data processing tasks, ensuring compatibility with BigQuery's infrastructure.
• Work with APIs, Docker, and GCP services to enhance and automate data workflows, particularly focusing on integration with BigQuery.
• Apply best practices in continuous integration and continuous deployment (CI/CD) environments using testing and pipeline validation techniques.
• Employ asynchronous architectures, queues, caching strategies, and distributed computing to enhance system performance.
• Write custom code in Python and utilize Jinja within DBT for dynamic pipeline generation, specifically designed for efficiency in BigQuery.
• Adopt design patterns and engineering best practices to ensure code scalability and maintainability.

Key outcomes:

• Develop a Unified Data Foundation: Spearheaded the creation of a single source of truth that centralizes our data assets, ensuring consistency and reliability across the board.
• Drive Impact Across Teams: Your work will directly influence the success of our business analysts, data scientists, and machine learning engineers by providing them with the crucial data they need to excel.
• Optimize Business Logic: Design and manage data tables that support and enhance business logic functions, delivering tailored data solutions that drive strategic decision-making.