Data Engineer

In order of importance, list the primary responsibilities critical to the performance of the position. It is recommended not to list actual tasks but focus on essential responsibilities that highlight accountability and level of judgment required.

Pipeline Development & Data Engineering

• Lead end-to-end technical delivery of data engineering solutions aligned with business initiatives and data platform architecture.
• Design, develop, test, and support ingestion and transformation (ETL/ELT) pipelines using Azure Databricks. Configure applications for best performance.
• Develop data processing jobs using PySpark, SQL, and Python with an emphasis on reliability, maintainability, and performance.
• Apply Lakehouse patterns to deliver curated data layers, Implement and manage governed data assets using Unity Catalog (organization, access controls, and best practices).

Orchestration & Operational SDLC

• Orchestrate end-to-end workflows using Azure Data Factory (ADF), including scheduling, dependency management, monitoring, and operational support.
• Establish, improve, and implement SDLC processes including development, testing, and production deployments.
• Apply Agile and DevOps practices. Ensure platform requirements (functional and non-functional) are met per design and architecture.

Collaboration & Stakeholder Engagement

• Collaborate with the Data/Business Analyst, Testing Engineer, and stakeholders to clarify requirements and ensure delivered outputs meet expectations.
• Collaborate with cross-functional stakeholders including ZTD Infrastructure, IT, business, and Information Security teams to evolve technology solutions.
• Engage and collaborate with technology partners and industry forums to stay abreast of technology trends, applying them as per business needs.

Engineering Standards & Quality

• Adhere to engineering standards through pull requests, code reviews, documentation, and reusable design patterns.
• Troubleshoot development and pre-production issues and drive continuous improvements to stability and performance.

Documentation & Handoff

• Document features, modules, requirements, and the risks of each project.
• Provide implementation documentation and support a structured handoff of production defects/bugs to the PADL DevOps Team.

Technical Skills:

• Azure Databricks (workspace management, cluster configuration, notebooks, jobs)
• PySpark and Spark SQL for large-scale data processing
• Python for scripting, automation, and data engineering tasks
• Unity Catalog for data governance, access controls, and asset organization
• Azure Data Factory (ADF) for pipeline orchestration, scheduling, and monitoring
• Lakehouse architecture patterns (Bronze/Silver/Gold data layers)
• Delta Lake format and optimization techniques
• CI/CD pipelines using GitHub Actions
• Git-based version control and collaborative development workflows
• Strong SQL skills for data transformation and validation
• Knowledge of data quality and testing frameworks
• Knowledge of DevOps and Agile methodologies