Senior Data Engineer

Job Title: Senior Data Engineer

Position type: Full Time

Work Location: Bangalore

Working style: Hybrid

Shift Time: 12:30 PM to 9:30 PM

People Manager role: No

Role Overview

Role Overview: Enterprise Data Analytics Engineering team is leading the development of cloud-based products and analytics capability for Aon. With a diverse portfolio of projects, and a sizeable production estate supporting an already successful business, we are in a critical growth phase and need the best minds in industry to help advance our journey. The DA Data Engineering team is responsible for building the data foundation that powers analytics, operational decision-making, and AI-driven products across the organization. As a Senior Data Engineer, you will play a critical role in designing, delivering, and operating scalable data platforms and data products that enable trusted, governed, and actionable data at enterprise scale. This role goes beyond traditional ETL development. You will help build and shape modern data architecture, implement engineering best practices, and build reusable data capabilities that support reporting, advanced analytics, machine learning, and emerging AI use cases. Working closely with business stakeholders, architects, product owners, and data scientists, you will transform complex data ecosystems into AI-ready reliable and consumable data products.

Responsibilities

• Design, develop, deploy, and operate scalable cloud-based data, analytics, automation and tooling solutions using modern data platforms and Cloud services.
• Support integration of AI-enabled capabilities by providing trusted data pipelines, feature-ready datasets, APIs, and operational controls required by downstream AI, analytics, and application services
• Apply engineering best practices as you design and deliver high-quality, scalable solutions.
• Own your solution end-to-end through design, implementation, and operations.
• Own production reliability of data systems, including monitoring, observability, incident response, and SLA adherence
• Collaborate cross-functionally across business stakeholders, product owners, and other engineers to deliver integrated, customer-focused and resilient data solutions.
• Contribute to the evolution of Aon's data platforms through technical feedback and innovation.
• Contribute to engineering standards, documentation, and technical mentorship within the team.

Qualifications

• Required education and certifications are critical for the role: bachelor's or master's degree in computer science, Engineering, or a related field.
• Required years of experience: 5+ years of relevant experience

Skills / Competencies

• Pipeline Performance
• Spark Optimization
• Proven experience designing and optimizing Spark pipelines against multi-terabyte datasets in production
• Deep understanding of Spark memory management, caching strategies and cluster resource allocation
• Ability to read and act on Spark UI diagnostics, query plans, and job metrics to resolve performance bottlenecks
• Lakehouse Delta Architecture
• Strong working knowledge of Delta Lake internals - MERGE optimization, Z-ordering/liquid clustering, schema evolution, Change Data Feed
• Ability to design and evolve layered, reliable data architectures (e.g., medallion or equivalent multi-stage patterns) that support reprocessing and scale without compromising data quality
• Pipeline Orchestration Design
• Hands-on experience with at least one major orchestration tool (Databricks Workflows, Airflow, or dbt)
• Data Governance Platform Enablement
• Experience with Unity Catalog (or equivalent) for access control, lineage, and data discovery across teams
• Understanding of data security practices - PII tagging, masking, row/column-level access control
• Foundational / General Competencies
• Strong proficiency in Python/PySpark and/or Scala, and SQL
• Solid grasp of distributed systems concepts (partitioning, parallelism, fault tolerance)
• Experience with DataOps and Engineering practices - Git, CI/CD Pipelines, Environment management
• Ability to communicate technical tradeoffs clearly to both engineering and non-engineering stakeholders
• AI-Enabled Engineering Productivity
• Hands-on fluency with AI-assisted development tools (e.g., Claude Code, GitHub Copilot, Databricks Assistant) for code generation, refactoring, debugging, and documentation
• Track record of infusing AI tooling across the development lifecycle - design/spec drafting, coding, code review, testing, and documentation - not just at the code-generation step.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.