Automation Test Engineer- Data

Overview of Role
As a Senior QA Engineer Data, you will lead the QA effort for complex data engineering pipelines on Google Cloud Platform. You’ll design and implement multi-layered testing strategies—integration, end-to-end, and data quality tests—across tools like dbt, Dataflow, Dataproc, BigQuery, AlloyDB, Cloud SQL, Cloud composer and Cloud Run, etc. Your work ensures the accuracy, reliability, and performance of data systems at scale.

Responsibilities
Testing Strategy & Test Design
• Define and maintain testing methodologies for the full GCP data engineering stack: Dataflow (TestPipeline), Dataproc (spark-testing-base or pytest), Cloud Run container tests, and SQL-based data validation in BigQuery/dbt.
• Develop and execute data quality frameworks using dbt tests (schema, singular, freshness) and external tools like Great Expectations, Soda Core, and Dataplex.
Pipeline & Database Testing
• Implement integration, and regression tests for ETL/ELT pipelines, including container- level and HTTP-triggered tests for Cloud Run.
• Use emulators or dedicated test instances to test Spanner, Cloud SQL, and AlloyDB. Validate stored procedures and database functions with sample data.
End-to-End (E2E) Pipeline Validation
• Orchestrate comprehensive E2E tests via Cloud Composer/Apache Airflow or scripting. Simulate real-world data flows and validate intermediate and final outputs.
CI/CD & Automation
• Embed QA in CI/CD pipelines (e.g., GitLab CI, Jenkins, GitHub Actions), automating test execution at all levels including data quality validations and dbt runs.
• Use IaC tools (Terraform, Deployment Manager) to provision reproducible test environments.
Collaboration & Stakeholder Engagement
• Partner with data engineers and stakeholders to review design for testability.
• Mentor junior QA team members, champion QA best practices, and lead efforts to improve data quality KPIs and test effectiveness.
Monitoring & Observability
• Utilize Cloud Logging, Monitoring, and observability tools (e.g., Elementary Data) to track pipeline health, test results, and identify anomalies.

Required Qualifications
• 5+ years in QA or software testing, including focused on data pipelines/data warehouses
• Proficient in complex SQL and writing data validation queries.
• Strong experience with GCP data tools: dbt, BigQuery, Dataflow, Dataproc, Cloud Run, Spanner, AlloyDB, Cloud SQL.
• Hands-on with data-quality frameworks: Great Expectations, Soda Core, dbt-utils, dbt-expectations.
• Familiarity with database emulators and test environment provisioning.
• CI/CD automation experience (Jenkins, GitLab CI, GitHub Actions).
• Skilled in scripting languages (Python, pytest).
• Excellent communication, mentoring ability, critical thinking, and attention to detail.
• API testing with PI Testing Postman REST APIs

Preferred Qualifications
• GCP Professional Data Engineer certification
• Experience enhancing dbt tests with dbt-utils and dbt-expectations.
• Knowledge of BigQuery data-quality services like Dataplex.
• Proficient with observability tooling (Cloud Logging, Elementary Data).
• Familiarity with data governance, lineage, and compliance validation.
• Exposure enterprise-based data migration project
• Exposure to Datawarehouse modernization