Senior Data Engineer QA

Job Description - Senior QA Data Engineer

Key Responsibilities:

• Testing Strategy: Plan, design, and execute comprehensive testing strategies for new features, enhancements, and system changes.
• Hands-on Testing: Create, maintain, and execute manual and automated test cases ensuring thorough coverage.
• Automation Leadership: Champion automation best practices by designing scalable, maintainable solutions supporting continuous delivery.
• Functional & Non-functional Testing: Validate system quality, performance, and reliability across the SDLC.
• Defect Management: Identify, analyse, and troubleshoot defects with root cause analysis, collaborating with cross-functional teams for resolution.
• Reporting: Produce clear, detailed defect reports to support efficient remediation.
• Stakeholder Communication: Share progress, risks, and quality insights with technical and non-technical stakeholders.

Knowledge & Experience Required:

Data Quality & Validation (Azure Data Platforms)

• ADF & Databricks Validation: Ensure accuracy, reliability, and consistency across complex workflows.
• ETL/ELT Testing: Validate ingestion, transformations, schema, and reconciliation between source and target systems.
• Data Profiling: Identify anomalies, duplicates, and integrity issues.
• Lineage & Monitoring: Validate lineage, auditing, and monitoring using observability tools.

Test Automation for Data Platforms

• Automation Frameworks: Design frameworks using Python (PyTest, PySpark, Pandas) or C#.
• Reusable Scripts: Build scripts for pipelines, transformations, and integrations.
• SQL Integration Tests: Develop integration tests for queries and workflows.

SQL & Database Testing

• Advanced SQL: Validate business rules, transformations, and workflows.
• Source-to-Target Validation: Ensure accuracy post-transformations.
• Database Structures: Validate indexes, constraints, stored procedures, and performance.

Business Intelligence & Reporting Testing

• BI Tools: Validate dashboards, reports, datasets, and models.
• Power BI: Work with DAX queries and tools like DAX Studio.
• Power Platform: Test Power Apps and Dataverse solutions.

CI/CD & Data Pipeline Testing

• Pipeline Integration: Embed validation tests into Azure DevOps/GitHub Actions.
• Deployment Validation: Support deployment validation for ETL pipelines and BI models.

Data Warehouse & Traditional ETL Tools

• Warehouse Testing: Validate architectures, data modelling, and enterprise platforms.
• ETL Tools: Experience with SAP BO Data Services, Informatica, SSIS.

Quality Engineering & Collaboration

• Enterprise QA: Strong focus on automation and quality engineering practices.
• Agile Practices: Active contribution to sprint ceremonies and Three Amigos sessions.
• Mentorship: Mentor QA engineers, conduct code reviews, and promote best practices.
• Collaboration: Partner with BAs and Developers to refine requirements and acceptance criteria.

Essential Skills:

• Cloud & Data Platforms: Azure Data Factory, Databricks, Synapse, SQL.
• Databases & Storage: SQL Server, Delta Lake.
• Programming & Scripting: Python (PySpark, Pandas), SQL, C#, REST APIs, DAX.
• Automation Tools: PyTest, Great Expectations, DataDiffPy, Postman.
• CI/CD & DevOps: Azure DevOps, GitHub Actions.
• Monitoring: Azure Monitor, Log Analytics, Databricks Monitoring.
• BI & Power Platform: Power BI, Power Apps, Dataverse.
• Database Testing: Advanced SQL for validation and performance analysis.
• Collaboration & Soft Skills: Strong teamwork, communication, and Agile collaboration.