Data Engineer

NCS Pte Ltd is seeking a Senior Data Engineer with strong hands-on technical depth and independent problem-solving ability to support a large-scale data platform migration engagement with Singtel. The role involves reviewing vendor-produced source-to-target mappings, validating data schema designs, and building robust ETL pipelines and data mart layers on Databricks (Azure) to power downstream Power BI dashboards and scheduled reporting.

The successful candidate will work directly within the NCS delivery team alongside Singtel stakeholders and the incumbent vendor, taking full ownership of the data engineering workstream - from schema validation through pipeline development, data quality assurance, and go-live support.

Schema Review & Data Validation

Review and validate design spec for data marts to be developed in Databricks.

Assess the optimized data schema proposed by the vendor to confirm completeness, correctness of data types, and alignment with business requirements.

Cross-reference the backend queries of Cognos reports against the proposed schema to ensure all required tables and fields are included.

Identify gaps, redundancies, and inconsistencies; document findings and work collaboratively with the vendor and Singtel teams to resolve them.

ETL Pipeline Development

Design and build scalable, optimized ETL pipelines in Databricks using PySpark and SQL to ingest, transform, and load data.

Develop aggregated views and data mart layers to serve as the analytical foundation for Power BI dashboards and Databricks scheduled reports.

Implement robust error handling, logging, data validation checks, and reconciliation frameworks across all pipeline stages.

Data Architecture & Goverce

Apply medallion architecture (Bronze / Silver / Gold) principles to organize data layers within Databricks Unity Catalog.

Maintain and contribute to the data catalogue, data dictionary, business glossary, and technical data lineage documentation.

Ensure data modelling standards, naming conventions, and data type conversion guidelines are adhered to across all deliverables.

AI-Accelerated Delivery

Leverage AI-powered developer tools/features (e.g., Databricks Assistant, Genie, Claude, Copilot) to accelerate ETL code generation, schema analysis, and validation tasks.

Champion automation-first practices within the team; identify repetitive tasks suitable for AI augmentation and implement accordingly.

Stakeholder & Vendor Engagement

Actively participate in client workshops, technical reviews, and sprint ceremonies with Singtel stakeholders and the platform vendor.

Proactively communicate risks, propose mitigations, and escalate blockers with a solutions-focused mindset.

Prepare and maintain technical documentation, status reports, and delivery artifacts to the standard expected in a large enterprise engagement.

Core Technical Proficiency

Strong hands-on experience with SQL, Python, and PySpark for data transformation and pipeline development.

Hands-on production experience with Databricks: ETL pipeline build, Delta Lake, Unity Catalog, Databricks Workflows, and Databricks SQL.

Hands-on experience with Oracle (on-premises) - including reading Oracle schemas, understanding execution plans, and migrating Oracle SQL to Spark SQL.

Data Engineering Fundamentals

Strong knowledge of ETL/ELT development patterns, incremental loading, SCD handling, and change data capture.

Experience with data validation frameworks, reconciliation strategies, and exception/error handling at scale.

Solid understanding of data modelling (relational, dimensional, and flat/wide schemas) and data architecture patterns (medallion, star schema, data vault).

Practical experience creating and maintaining source-to-target mapping documents, data dictionaries, and business glossaries.

Knowledge of data type conversion challenges between Oracle and Spark/Databricks environments.

AI & Automation Tools

Ability to critically evaluate and refine AI-generated code for production-quality data pipelines.

Experience Level

Minimum 5 years of relevant working experience in data engineering roles.

Prior experience in a data platform migration engagement (on-premises to cloud) is strongly preferred.

Databricks Certified Data Engineer Associate or Professional (preferred)

Microsoft Certified: Azure Data Engineer Associate (DP-203) (preferred)

Microsoft Certified: Azure Fundamentals (AZ-900) (advantageous)

Oracle Database SQL Certified Associate (advantageous)

Any recognized AI/ML certification (e.g., AWS Certified Machine Learning, Google Professional Data Engineer) (advantageous)

Familiarity with Cognos report structures and backend SQL, enabling effective analysis of report dependencies.

Experience with Power BI data models (DirectQuery, import, composite) to support collaboration with the BI team.

Knowledge of Apache Spark performance tuning, query optimization, and cluster configuration on Databricks.

Exposure to data quality frameworks (e.g., Great Expectations, Databricks Delta Live Tables constraints).

Understanding of DevOps/DataOps practices: Git branching, CI/CD pipelines for data, and automated testing of data pipelines.

Experience working in Agile / Scrum delivery teams in a client-facing consulting environment.

Familiarity with Singtels data ecosystem or telco industry data models is an advantage.

Strong technical and logical reasoning skills - ability to independently understand complex work scopes, identify the right approach, and articulate trade-offs clearly.

Ownership and accountability mindset - takes responsibility for deliverable quality and timelines without requiring close supervision.

Proactive risk identification - surfaces risks early, proposes practical mitigations, and drives resolution.

Positive, can-do attitude - maintains constructive energy under pressure and in ambiguous situations.

Client stakeholder management - communicates professionally with Singtel business and technical stakeholders at various levels.

Vendor collaboration - works constructively with the incumbent platform vendor, balancing assertiveness with diplomacy.

Strong team player - contributes to team knowledge sharing, pair reviews, and collective problem-solving.

Clear written and verbal communication - able to produce high-quality technical documentation and present findings concisely.