Senior Data Engineer

Role & responsibilities

• Design, build, and optimize ETL/ELT pipelines using PySpark on Azure Databricks.
• Develop real-time and near-real-time ingestion pipelines using Spark Structured Streaming and Azure Event Hubs
• Develop and tune complex SQL models in BigQuery for analytics and reporting
• Build ingestion pipelines from APIs and third-party sources into Azure Data Lake / BigQuery
• Implement incremental loads, CDC, and MERGE/upsert patterns for large datasets.
• Experience on data lake/warehouse solutions across GCP and Azure
• Build dashboards and reporting layers in Looker Studio
• Lead code reviews, mentor engineers, and drive Git-based CI/CD practices for data pipelines
• Partner with product/business teams to translate requirements into scalable data models
• Optimize cost & performance of already running queries
• Carefully plan and implement data load requirement.
• Regular monitoring of the data pipelines and systems and maintains the uptime of the overall data availability.
• Co-ordinate with stake holders and provide relevant data and business insights.

Required Skills:

• 5+ years in data engineering
• Strong SQL (joins, window functions, query optimization)
• Well versed in Python and PySpark (Azure Databricks)
• Spark Structured Streaming for real-time data processing
• Azure Event Hubs for event ingestion and streaming architecture
• BigQuery , AWS DynamoDB, Azure Storage Tables, Azure SQL, Azure Event Hub.
• Azure Data Lake (ADLS Gen2)
• Multi-cloud experience: GCP + Azure
• Looker Studio for BI/reporting
• Git version control
• REST API integration (auth, pagination, error handling)

Good to have:

• Orchestration tools (Airflow, Databricks Workflows)
• Data quality/governance frameworks
• Cloud cost optimization experience