Data Engineer

Remote role but 4 weeks training in Hyderabad or Bangalore.

Job Description

Insight Global is seeking a Data Engineer to join a leading financial services client on a high-profile data migration programme. In this role, you will support the end-to-end migration of large-scale data assets from legacy on-premise Data Lake environments to a modern AWS-hosted Lakehouse platform. Day-to-day, you will be responsible for migrating and optimising data pipelines, extraction logic, and scheduling processes, converting existing SQL and Spark-based consumption patterns for Snowflake and Apache Iceberg, and performing rigorous data reconciliation and validation to ensure accuracy throughout the migration. You will work closely with data owners, business stakeholders, and internal platform teams to understand data requirements, support the hand-off and sign-off process, troubleshoot data issues, and ensure migrated datasets meet both technical and business expectations. This role requires strong hands-on experience with data engineering technologies, a proactive approach to problem-solving, and the ability to deliver high-quality solutions within a fast-paced enterprise environment.

Required Skills & Experience

• Bachelor's or Master's in Computer Science, Applied Mathematics, Engineering, or a related quantitative field.
• Minimum of 3-5 years of professional "hands-on-keyboard" coding experience in a collaborative, team-based environment. Ability to troubleshoot SQL and basic scripting experience.
• Professional proficiency in Python or Java.
• Deep familiarity with the full Software Development Life Cycle (SDLC) and CI/CD best practices & K8s deployment experience.

Candidates must demonstrate a sophisticated understanding of the following modelling concepts to ensure data correctness during reconciliation:

• Temporal Data Modelling: Managing state changes over time (e.g. SCD Type 2).
• Schema Management: Expertise in Schema Evolution (for Iceberg, Apache) and enforcement strategies.
• Performance Optimisation: Advanced knowledge of data partitioning and clustering.
• Architectural Theory: Balancing Normalisation vs. Denormalisation and the strategic use of Natural vs. Surrogate Keys.
• Surrogate Keys.

While candidates are not expected to be experts in every tool, the collective team must cover the following technologies:

Area Technologies

Extraction & Logic Kafka, ANSI SQL, ITF, Apache Spark

Data Formats JSON, Avro, Parquet

Platforms Hadoop (HDFS/Hive), Snowflake, Apache Iceberg, Sybase IQ