Senior Data Engineer (Databricks on AWS)

We are seeking a talented individual to join Oliver Wyman Tech, a Marsh business. This role will be based in Pune/Gurugram. This is a hybrid role that has a requirement of working at least three days a week in the office. We are seeking an experienced Data Engineer to join our Enterprise Data Platform team and play a key role in building and evolving our cloud data platform on the Databricks Lakehouse Platform hosted on AWS. As a Data Engineer, you will design, develop and optimise scalable data pipelines, reusable data products and curated datasets that support enterprise reporting, advanced analytics, machine learning and emerging AI use cases. You will help ensure that enterprise data is secure, trusted, governed and readily available for business consumption. Working closely with data architects, product managers and data scientists you will develop modern data engineering solutions using Databricks, Unity Catalog and Delta Lake while following enterprise architecture, security and engineering standards. You will also contribute to the adoption of advanced platform capabilities including Delta Sharing, AI/BI, data governance and AI-ready data engineering practices. This is an excellent opportunity for an experienced engineer who enjoys solving complex data challenges, building scalable cloud solutions and contributing to a modern enterprise data platform.

Senior Principal Engineer - Data Engineering (Databricks + AWS) We will count on you to:

• Design, develop and maintain scalable data pipelines and reusable data products using the Databricks Lakehouse Platform on AWS.
• Build automated ingestion pipelines from enterprise applications, databases, APIs, cloud storage, files and streaming data sources using modern engineering practices.
• Design and implement robust ETL/ELT processes using PySpark, SQL and Delta Lake following Medallion Architecture principles (Bronze, Silver and Gold layers).
• Deliver high-quality, trusted and AI-ready datasets that support enterprise reporting, self-service analytics, machine learning and Generative AI use cases.
• Implement and maintain enterprise data governance using Unity Catalog, including metadata management, data lineage, security, access controls and compliance with organisational policies.
• Develop scalable semantic datasets and data models to support Power BI and other enterprise Business Intelligence platforms.
• Support implementation of advanced Databricks capabilities including Delta Sharing, AI/BI, Lakehouse Monitoring and other emerging platform capabilities.
• Monitor and optimise data pipeline performance, scalability, reliability and cloud cost efficiency.
• Implement data quality controls, automated testing and monitoring to ensure data accuracy, completeness and reliability.
• Apply modern DataOps and DevOps practices including CI/CD, Git-based development, code reviews and deployment automation.
• Collaborate with business stakeholders, product owners, architects and data scientists to understand requirements and deliver scalable technical solutions.
• Contribute to engineering standards, reusable frameworks, documentation and knowledge sharing across the Enterprise Data Platform team.
• Stay current with emerging cloud technologies, Databricks capabilities and industry best practices, recommending improvements where appropriate.

What you need to have:

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering or a related discipline.
• Minimum 5+ years of enterprise data engineering experience delivering production-grade data platforms and solutions.
• Hands-on experience developing enterprise data solutions using Databricks on AWS.
• Strong experience developing scalable data pipelines using PySpark, SQL and Python.
• Experience designing and implementing Lakehouse architectures using Delta Lake.
• Experience implementing enterprise data governance using Unity Catalog or equivalent enterprise data catalog solutions.
• Experience integrating data from multiple enterprise systems including databases, APIs, SaaS platforms, cloud storage and streaming technologies.
• Experience developing enterprise datasets for Power BI or equivalent Business Intelligence platforms.
• Strong understanding of data modelling, ETL/ELT design patterns and modern cloud data architectures.
• Experience working with cloud-native AWS services such as S3, IAM, Glue, Lambda, EventBridge or equivalent services.
• Experience implementing CI/CD pipelines, Git-based source control and automated deployment processes.
• Experience optimising Spark workloads and tuning large-scale distributed data processing jobs.
• Experience implementing data quality, monitoring and observability practices within enterprise data platforms.
• AWS Cloud Certification or equivalent

What makes you stand out?

• Delta Sharing
• Databricks AI/BI
• Databricks Workflows
• Auto Loader
• Structured Streaming
• Lakehouse Monitoring
• MLflow
• Infrastructure as Code (Terraform or equivalent)
• Data mesh or data product concepts
• Exposure to Machine Learning or Generative AI data preparation
• Databricks Certified Data Engineer

Why join our team:

• We help you be your best through professional development opportunities, interesting work and supportive leaders.
• We foster a vibrant and inclusive culture where you can work with talented colleagues to create new solutions and have impact for colleagues, clients and communities.
• Our scale enables us to provide a range of career opportunities, as well as benefits and rewards to enhance your well-being. Oliver Wyman is a business of Marsh (NYSE: MRSH), a global leader in risk, reinsurance and capital, people and investments, and management consulting, advising clients in 130 countries. With annual revenue of over $24 billion and more than 90,000 colleagues, Marsh helps build the confidence to thrive through the power of perspective. For more information, visit oliverwyman.com, or follow us on LinkedIn and X. Marsh is committed to embracing a diverse, inclusive and flexible work environment. We aim to attract and retain the best people and embrace diversity of age, background, caste, disability, ethnic origin, family duties, gender orientation or expression, gender reassignment, marital status, nationality, parental status, personal or social status, political affiliation, race, religion and beliefs, sex/gender, sexual orientation or expression, skin color, or any other characteristic protected by applicable law. Marsh is committed to hybrid work, which includes the flexibility of working remotely and the collaboration, connections and professional development benefits of working together in the office. All Marsh colleagues are expected to be in their local office or working onsite with clients at least three days per week. Office-based teams will identify at least one “anchor day” per week on which their full team will be together in person.