Senior/Lead Bigdata Engineer

Key Responsibilities

• Design, build, and maintain scalable data pipelines using Python & PySpark
• Develop and optimize ETL/ELT workflows for structured and semi-structured data
• Work with Azure data services to ingest, process, and store large datasets
• Write efficient SQL queries for data analysis, transformation, and reporting
• Collaborate with data analysts, data scientists, and business teams to deliver reliable datasets
• Implement data quality checks, monitoring, and performance tuning
• Handle batch and (if applicable) streaming data workloads
• Follow best practices for data security, governance, and documentation

Required Skills

Programming & Data Processing

• Strong hands-on experience with Python
• Solid expertise in PySpark / Apache Spark
• Experience with data transformations, joins, aggregations, window functions

Databases & SQL

• Advanced SQL(CTEs, indexing, query optimization)
• Experience with relational and analytical databases

Cloud & Azure

• Hands-on experience with Microsoft Azure, including:
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS)
• Azure Synapse Analytics
• Azure Databricks
• Understanding of cloud-based data architecture

Data Engineering Concepts

• ETL vs ELT
• Data modeling (Star / Snowflake schemas)
• Partitioning, caching, performance tuning
• Handling large-scale datasets

Good to Have

• Experience with streaming tools (Kafka, Event Hubs)
• Knowledge of CI/CD pipelines for data workflows
• Exposure to Airflow / Azure Data Factory scheduling
• Understanding of DevOps / Git / version control
• Basic knowledge of data governance & compliance

Education

• Bachelors degree in Computer Science, Engineering, or related field
  (Equivalent practical experience is welcome)

What We Offer

• Opportunity to work on large-scale, real-world data platforms
• Exposure to modern cloud & big-data technologies
• Collaborative and growth-focused environment