dbt Architect

An experienced DBT Architect to design, develop, and optimize data transformation pipelines within a modern cloud-based data platform built on Databricks Lakehouse hosted on AWS The ideal candidate will have strong expertise in DBT (Data Build Tool), SQL development, data modeling, and cloud-native data engineering practices.

The role involves transforming raw and curated data into business-ready datasets, implementing best practices for data quality, testing, documentation, and CI/CD automation while collaborating closely with data architects, data engineers, analysts, and business stakeholders.

The core responsibilities for the job include the following:

Data Transformation and Modeling:

• Design, develop, and maintain scalable data transformation pipelines using DBT in Databricks.
• Develop and manage DBT models, including staging, intermediate, and mart layers.
• Implement dimensional data models (Star Schema, Snowflake Schema, Data Vault) based on business requirements.
• Translate business requirements into reusable, maintainable, and performant data models.

Databricks Development:

• Build and optimize transformation logic leveraging Databricks SQL and Delta Lake capabilities.
• Develop efficient data processing workflows within Databricks Lakehouse architecture.
• Optimize data transformations for performance, scalability, and cost efficiency.
• Implement partitioning, clustering, and Delta Lake optimization techniques.

Data Quality and Governance:

• Implement data quality checks, validations, and testing using DBT tests.
• Develop custom DBT macros and reusable components.
• Create data lineage and maintain metadata documentation.
• Ensure adherence to data governance and security standards.

CI/CD and DevOps:

• Implement and manage CI/CD pipelines for DBT deployments.
• Automate testing, deployment, and monitoring processes.
• Integrate DBT with Git-based source control systems.
• Support code reviews and enforce development standards.

Collaboration and Stakeholder Engagement:

• Collaborate with data architects and data engineers to design end-to-end data solutions.
• Work with business users and analytics teams to understand reporting and analytical requirements.
• Mentor junior developers and promote best practices across the team.
• Participate in agile ceremonies, including sprint planning, retrospectives, and code reviews.

Requirements:

• Must Have: Data Build Tool (DBT) experience of 4 years; overall 9+ years into data engineering and analytics; 3+ years of Databricks development experience; AWS cloud-based data platforms; AWS cloud-based data platforms.

Core Technologies:

• Strong expertise in DBT Core and/or DBT Cloud.
• Advanced SQL development experience.
• Hands-on experience with Databricks Lakehouse Platform.
• Experience with Delta Lake, Unity Catalog, and Databricks Workflows.
• Strong understanding of data warehousing and dimensional modeling concepts.

Data Engineering:

• Knowledge of ETL/ELT design patterns.
• Experience working with large-scale datasets.
• Understanding of batch and near real-time processing concepts.
• Performance tuning and optimization of SQL transformations.

DevOps and Automation:

• GitHub, GitLab, or Azure DevOps.
• CI/CD pipeline implementation.
• Automated testing frameworks.
• Infrastructure-as-Code exposure preferred.