LEAD DATA ENGINEER - PySpark

TNB Tenaga Malaysia to migrate its existing Cloudera data ecosystem to the modern Databricks Data Intelligence Platform (Lakehouse Architecture). This migration is not a mere technical uplift, but a critical enabler aligned directly with TNBs Reimagining TNB 2025 (RT25) strategy and its overarching Energy Transition (ET) Aspiration Plan, specifically supporting the Grid of the Future and Sustainability Pathway 2050 pillars. TNBs existing Cloudera environment (v7.1.7) to the Databricks Lakehouse Platform using a Lift-and-Shift, then Modernize approach.

This approach ensures minimal operational disruption during migration while establishing a scalable foundation for advanced analytics, AI, and domain-led modernization.

• Lead Data Engineer
• Role Summary

Leads the design, development, migration, and deployment of data pipelines, ETL processes, and data engineering best practices for the Databricks platform.

• Key Responsibilities
• Lead migration of HiveQL, Impala, Spark SQL, Bash, and Informatica workloads.
• Design and build batch and streaming pipelines using Databricks.
• Develop PySpark-based transformation frameworks.
• Implement Delta Live Tables and Databricks Workflows.
• Establish CI/CD standards and coding best practices.
• Conduct performance tuning and optimization.
• Support production deployments and hypercare activities.
• Mentor and guide development teams.
• Required Skills
• Databricks Platform
• Apache Spark / PySpark
• Delta Live Tables (DLT)
• Databricks Workflows
• Structured Streaming
• SQL, Python, Scala
• Informatica Migration
• Git, CI/CD Pipelines
• Data Quality and Testing Frameworks
• Agile Delivery Methodologies

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.