Data Engineer

Responsibilities

• Independently understand business and technical requirements and translate them into scalable data engineering solutions.
• Take end-to-end ownership of assigned projects, modules and data pipelines with minimal supervision.
• Design, develop, test and deploy robust end-to-end data pipelines on the Azure Data Platform.
• Build scalable and reusable data ingestion and transformation frameworks using Azure Data Factory and Azure Databricks.
• Develop data transformation and processing logic using Python, PySpark and SQL.
• Work with structured, semi-structured and unstructured data from databases, APIs, files and other enterprise data sources.
• Design and implement batch and, where required, near-real-time data processing solutions.
• Develop optimized Databricks workloads using Apache Spark and Delta Lake.
• Perform data analysis, profiling and validation to identify data quality, completeness and consistency issues.
• Implement appropriate data quality checks, error handling, logging and monitoring within data pipelines.
• Troubleshoot data pipeline failures, performance issues and data discrepancies independently.
• Optimize SQL queries, Spark jobs and data pipelines for performance and scalability.
• Create reusable components and frameworks to improve development efficiency across projects.
• Participate in solution design and technical discussions and provide recommendations on implementation approaches.
• Implement CI/CD processes for data engineering components using Azure DevOps.
• Deploy and manage ADF, Databricks and related Azure data components across development, test and production environments.
• Prepare appropriate technical design, data mapping and operational documentation.
• Collaborate with solution architects, analysts, BI developers, source-system teams and other engineering teams.
• Support production deployments and troubleshoot post-production issues when required.

Requirements

• Strong hands-on experience with:
  • Azure Data Platform
    • Azure Data Factory (ADF) pipelines, datasets, linked services, triggers, parameterization and reusable frameworks
    • Azure Databricks
    • Azure Data Lake Storage Gen2 (ADLS Gen2)
    • Azure Synapse Analytics / Azure SQL
    • Azure Key Vault
    • Azure Monitor / Log Analytics or equivalent monitoring capabilities
    • Azure Event Hubs exposure/experience is preferred
  • Databricks & Data Processing
    • Strong hands-on experience with Databricks and Apache Spark
    • Strong PySpark development skills
    • Experience with Delta Lake / Delta tables
    • Understanding of Medallion or similar layered data architectures
    • Experience implementing incremental and full-load processing patterns
    • Ability to troubleshoot and optimize Spark workloads
    • Experience with Databricks Workflows/Jobs
    • Knowledge of Unity Catalog is preferred
  • SQL & Python
    • Strong SQL development skills
    • Ability to write and optimize complex SQL queries
    • Experience with joins, CTEs, window functions and analytical queries
    • Strong working knowledge of Python
    • Ability to develop reusable Python/PySpark modules and utilities
  • DevOps / CI-CD
    • Hands-on experience with Azure DevOps
    • Git-based source control
    • Branching and code-management practices
    • CI/CD implementation for Azure Data Factory and Databricks
    • Experience deploying solutions across multiple environments

Qualifications & Experience

• 46 years of overall experience in Data Engineering / Data Platform development.
• Preferably 4+ years of hands-on experience with Microsoft Azure Data Platform technologies.
• Strong hands-on experience with Azure Data Factory, Databricks, SQL and Python/PySpark.
• Bachelors degree in Computer Science, Information Technology, Engineering or a related discipline.
• Experience delivering at least one or more end-to-end Azure data engineering projects.
• Good understanding of data warehousing, data lake and modern data platform concepts.
• Strong analytical and problem-solving skills.
• Ability to work independently and manage multiple technical activities.
• Good written and verbal communication skills.

Preferred Skills

• Microsoft Azure / Databricks certifications.
• Experience with Unity Catalog and Databricks governance.
• Experience with REST API-based data ingestion.
• Knowledge of dimensional modelling and data warehouse concepts.
• Experience working with Power BI or downstream analytics platforms.
• Exposure to streaming/event-driven data processing.
• Experience working in enterprise-scale Azure environments.

Job Details

• Job Type: Full time
• Job Opening Name: Data Engineer
• State: Telangana
• Country: India
• Location: Hyderabad, Noida/ Chennai

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.