Big Data Hive

Role Overview

We are looking for a Big Data Engineer with strong expertise in Databricks, Azure, Python, and Hive to design, build, and optimize scalable data platforms. The ideal candidate will have hands-on experience in ETL/ELT development, Big Data technologies, and modern data lake architectures.

Key Responsibilities

• Design, develop, and maintain robust ETL/ELT pipelines for batch and real-time data processing.
• Build and manage large-scale data ingestion frameworks for structured and unstructured data.
• Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and scalability.
• Implement data quality, governance, security, and compliance standards across data platforms.
• Develop and support data warehouses, data marts, and data lakes.
• Collaborate with Data Governance, Data Quality, Data Architecture, Data Science, DevOps, BI, and business teams to deliver data solutions.
• Contribute to data platform modernization and cloud-based analytics initiatives.

Mandatory Skills

• Databricks
• Azure Data Platform
• Python
• Hive
• SQL

Required Technical Skills

• Strong experience in ETL/ELT pipeline development.
• Hands-on experience with Apache Spark, Hadoop, and Hive.
• Experience with NoSQL and distributed databases such as HBase and Cosmos DB.
• Knowledge of Data Warehouses, Data Marts, and Data Lakes.
• Experience implementing Medallion Architecture.
• Understanding of Data Quality validation frameworks and best practices.
• Strong problem-solving and performance tuning skills.

Preferred Skills

• Scala development experience.
• Exposure to real-time data processing frameworks.
• Experience with Azure cloud-native data services.
• Knowledge of CI/CD and DevOps practices in data engineering.