Data Engineer

Job Overview:

We are seeking a skilled Data Engineer to join our team. The successful candidate will be responsible for development and optimization of data pipelines, implementing robust data checks, and ensuring the accuracy and integrity of data flows. This role is critical in supporting data-driven decision-making processes, especially in the context of our insurance-focused business operations.

Key Responsibilities:

• Lead the design, development, and optimization of scalable, high-performance data pipelines, ensuring reliable data ingestion and transformation across distributed systems, while proactively identifying and resolving performance bottlenecks and system inefficiencies.

• Define and enforce enterprise-grade data quality frameworks, implementing advanced validation rules, anomaly detection, and reconciliation processes to ensure data accuracy, completeness, and integrity across large-scale datasets.

• Own end-to-end monitoring and reliability of data platforms, establishing observability practices (alerts, SLAs, dashboards) and driving root-cause analysis to minimize downtime and ensure seamless continuity of critical data workflows.

• Govern and continuously improve data pipelines through periodic audits and reviews, ensuring adherence to internal standards, data governance policies, and regulatory compliance, while driving standardization and best practices across teams.

• Collaborate with cross-functional stakeholders (architects, analysts, product owners) to align data solutions with business objectives, and actively contribute to backlog prioritization, sprint planning, and technical problem-solving in Agile environments.

• Mentor junior engineers and drive technical excellence, promoting reusable frameworks, coding standards, and best practices to improve overall team productivity and solution quality.

Candidate Profile:

• 6-10 years experience on Data Engineering role working with Databricks & Azure Cloud technologies.

• Bachelor's degree in computer science, Information Technology, or related field. • Strong proficiency in PySpark, Python, SQL.

• Strong experience in data modeling, ETL/ELT pipeline development, and automation • Hands-on experience with performance tuning of data pipelines and workflows

• Proficient in working on Azure cloud components Azure Data Factory, Azure Data Bricks, Azure Data Lake etc.

• Experience with data modeling, ETL processes, Delta Lake and data warehousing.

• Experience on Delta Live Tables, Autoloader & Unity Catalog.

• Preferred - Knowledge of the insurance industry and its data requirements.

• Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.

• Excellent communication and problem-solving skills to work effectively with diverse teams

• Excellent problem-solving skills and ability to work under tight deadlines.