Lead Data Engineer [T500-27420]

About us: We are a highly successful 190-year-old, Fortune 500 commercial property insurance company of 6,000+ employees with a unique focus on science and risk engineering. Businesses worldwide trust our expertise to protect their assets, relying on our comprehensive risk assessments and robust, engineering-based insurance solutions to safeguard against fire, natural disasters, and other perils. Serving over a quarter of the Fortune 500 and major corporations globally, we deliver data-driven strategies that enhance resilience, ensure business continuity, and empower organizations to thrive. FM India is a strategic location for driving our global operational efficiency. Our presence in India allows us to leverage the country’s talented workforce and advance our capabilities to serve our clients better. We have diverse corporate functions that emphasize research, advanced technologies like AI and analytics, risk engineering, research, finance, marketing, HR, etc. working together to provide innovative solutions and nurture lasting relationships – from co-workers to clients.

Role Title: Lead Data Engineer Position Summary: The Lead Data Engineer is responsible for data analysis, data modelling, data collection, data integration, data storage and preparation of data for consumption. This includes creating and managing data infrastructure, data pipeline design, implementation and data verification. Along with the team, responsible for ensuring the highest standards of data quality, security and compliance. Additionally, this role will utilize AI to implement methods to improve data reliability and quality, combine raw information from different sources to create consistent data sets. The incumbent will need to be proficient in DataOps and be able to provide technical expertise to other Data Engineers. Incumbents will design and build data solutions and integrations which may involve diverse data platforms, software, hardware, technologies and tools. Uses available approved technologies and recommends solution options. Incumbents may design and build data solutions and integrations which may involve diverse development platforms (including 3rd party systems), software, hardware, technologies and tools. The Lead Data Engineer is the third level position in the Data Engineer job family. Those holding this position are typically assigned to be the technical lead for medium to large projects and initiatives, and provide direction, work effort estimates and delivery coordination to junior data engineers. This role balances independent contribution with mentoring and technical leadership of other data engineers and analytics practitioners.

Job Responsibilities: Lead the data acquisition and integration strategy:

• Define and implement the strategy for bringing structured and unstructured data sources into Data Analytics. Perform as the technical lead with the Product teams (Underwriting and Risk; Client Service, Sales and Marketing; Claims; Account and Location Engineering) as well as emerging data sources (purchased data sets; external data; etc.)
• Partner with product owners, developers, solution architects, business analysts, data engineers, data analysts, data scientists and others to understand data and reporting needs
• Assist in or lead the development and implementation of data solutions using data modeling techniques and technologies such as Fabric, Synapse Analytics, SQL Server, Azure services, and others as required
• Validate solutions are accurate through detailed and disciplined testing methodologies and taking opportunities to use AI in this process.
• Responsible for ensuring the developed solutions are accurate, good quality, and delivered according to schedule
• Ensure tables and views are designed for data integrity, efficiency and performance, and are easy to comprehend

Move and Store Data:

• Design and implement the data flow, infrastructure pipelines, ETL/ELT, structured and unstructured data movement and storage solutions
• Design data models and data flows into and out of Data Analytics databases
• Understand and design data relationships between business and data subject areas
• Follow standards for naming conventions, code documentation and code review
• Support data exploration and transformation needs
• Lead the strategy and implementation of data cleansing and mentor other team members on data quality and data cleansing tasks as needed
• Define and implement approach to data profiling to identify data anomalies and resolve issues
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Lead the build-out of the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and Azure technologies

Production Support and Operations:

• Lead the definition and creation of prototypes and proof of concepts for the selected solutions, and implement complex data engineering projects
• Project planning and facilitation
• Create the development project implementation plans for strategic data analytics projects
• Prioritize work within assigned projects and assign work to development team
• Create estimates, timelines and resourcing needs for development team
• Lead the team in the development and implementation of the project
• Support production applications and reduce maintenance
• Identify repeat support issues and implement solutions to reduce those issues
• Manage and address operational data issues by determining and fixing root cause
• Mentor developers, data analysts and data scientists who need to interact with data
• Liaison with dba(s) and Infrastructure Teams on the infrastructure and configuration needed to support Data Analytics solutions.
• Monitor system performance and execute opportunities for optimization
• Monitor storage capacity and reliability
• Remediate production issues quickly, with appropriate validation and deployment steps
• Supply ad-hoc data extracts and analysis to respond to tactical business needs
• Provide clear and concise communication to users, teammates and leadership

Support planning and execution of team priorities:

• Identify work tasks and capture them in team backlog
• Participate in backlog and technical debt planning and prioritization
• Resolve colliding priorities and escalate as needed
• Network with product teams and infrastructure teams to keep abreast of databases, business processes and technology changes which may impact data delivery.

Skill and Experience:

• 6-7 years of experience required to perform essential job functions.
• Ability to read and create data models Relational (3rd Normal Form) and non-relational (Inmon/Kimball) database theory
• Strong SQL skills
• Build scalable, fault-tolerant data pipelines
• Design and support ETL/ELT processes
• Design, build, maintain data lakes and data warehouses
• Programming languages SQL, Python, Spark, Kafka, Azure, JSON
• Implement data governance and data security measures

Must Have Skills:

• Data warehousing
• SQL development (Microsoft SQL Server / Azure SQL – stored procedures, complex queries, views, functions, DB design & architecture): 5 years
• Experience with medallion architecture (Data/Delta Lake, Lakehouse)
• Experience working with Parquet format
• Working knowledge of PySpark / PySQL
• ETL experience (SSIS / ADF or other Microsoft platforms)
• Azure cloud experience: 4–5 years
• Coding skills: C# / Python
• Experience with Azure DevOps and CI/CD pipelines

Good to Have Skills:

• Databricks
• Microsoft Fabric
• Kafka
• Kubernetes