BENGALURU · FULLTIME
DATA ENGINEER - Databricks
Happiest Minds Technologies
Bengaluru · onsite · Posted 43d ago
Sourced from
Undisclosed3–8 yrsfulltimeBengaluru
About a minute. You'll make an account on the way through.
agent · working
- Read this posting0.8s
- Found the form on Naukri · 6 questions to answer1.8s
- Decided the answers for all of them2.9s
- Tailor your résumé to this role
- Waiting for you
Section · 01
About this role
Senior Databricks Data Engineer
Role Overview
We are looking for an experienced Senior Databricks Data Engineer to design, develop, and maintain scalable ETL/ELT data pipelines using Databricks, PySpark, Spark SQL, and Delta Lake. The candidate should have strong hands-on experience in data transformation, pipeline orchestration, performance optimization, data quality, and production support within an AWS-based data engineering environment.
The role also requires practical experience using Cursor, GitHub Copilot, Windsurf, or similar agentic AI-enabled development tools to accelerate development, testing, debugging, documentation, and code optimization.
Key Responsibilities
- Design, develop, test, and maintain scalable ETL/ELT data pipelines using Databricks and PySpark.
- Develop and optimize data transformation logic using PySpark, Spark SQL, and SQL.
- Build and maintain Delta Lake tables following Bronze, Silver, and Gold layers of the Medallion Architecture.
- Implement Delta Lake capabilities, including:
- MERGE and upsert operations
- Schema enforcement and schema evolution
- Time Travel and version management
- OPTIMIZE and VACUUM
- Incremental and change-based data processing
- Develop and manage Databricks Workflows and Jobs for pipeline orchestration, scheduling, dependency management, retries, and alerting.
- Design efficient incremental-load and restart/recovery mechanisms to ensure reliable pipeline execution.
- Troubleshoot pipeline failures, performance bottlenecks, data-quality issues, and production incidents.
- Optimize Spark workloads through effective use of partitioning, joins, caching, broadcast strategies, shuffle management, file sizing, and query execution plans.
- Develop reusable data engineering utilities, libraries, and frameworks using PySpark and Python.
- Write complex SQL queries for data transformation, reconciliation, validation, analysis, and reporting.
- Integrate Databricks pipelines with external applications and systems using REST APIs.
- Develop and support inbound and outbound file integrations using SFTP.
- Process structured and semi-structured data in formats such as CSV, JSON, Parquet, and Delta.
- Work with AWS services used in the data engineering ecosystem, particularly Amazon S3.
- Implement exception handling, logging, auditing, monitoring, alerting, and operational recovery mechanisms.
- Implement data-quality validations and reconciliation controls across different pipeline stages.
- Participate in code reviews and ensure adherence to coding, security, performance, and data engineering best practices.
- Collaborate with architects, business analysts, source-system teams, QA teams, DevOps teams, and other engineering stakeholders.
- Support production deployments, release validation, operational monitoring, and troubleshooting of production data pipelines.
- Create and maintain technical documentation, pipeline specifications, operational procedures, and support runbooks.
- Use Cursor or similar agentic AI-enabled IDEs to improve development productivity, including:
- Generating and refactoring PySpark, Python, and SQL code
- Creating unit tests and data-validation scripts
- Troubleshooting errors and performance issues
- Generating technical documentation and code explanations
- Reviewing and validating AI-generated code for correctness, security, maintainability, and performance
Must-Have Skills
- Strong hands-on experience in Databricks-based data engineering.
- Advanced proficiency in PySpark, Spark SQL, and SQL.
- Strong understanding of Apache Spark architecture and distributed data processing.
- Hands-on experience building ETL/ELT pipelines at enterprise scale.
- Strong experience with Delta Lake and Medallion Architecture.
- Practical experience with Delta Lake features such as MERGE, schema evolution, Time Travel, OPTIMIZE, and VACUUM.
- Hands-on experience developing and managing Databricks Workflows and Jobs.
- Strong understanding of Spark performance optimization, including:
- Data partitioning
- Join strategies
- Caching and persistence
- Shuffle optimization
- Data skew handling
- Query execution plans
- Small-file management
- Strong SQL skills, including complex joins, window functions, aggregations, reconciliation, and data-validation queries.
- Working knowledge of Python for developing reusable utilities, frameworks, and automation scripts.
- Experience working with CSV, JSON, Parquet, and Delta file formats.
- Hands-on experience integrating data pipelines with REST APIs and SFTP systems.
- Experience working with Amazon S3 in a data engineering environment.
- Experience implementing logging, exception handling, auditing, monitoring, alerting, and restart/recovery mechanisms.
- Experience troubleshooting and supporting production data pipelines.
- Hands-on knowledge of Cursor, GitHub Copilot, Windsurf, or another agentic AI-enabled development environment.
- Ability to write effective prompts, review AI-generated code, identify hallucinated or incorrect logic, and apply organizational security and data-privacy standards while using AI tools.
- Strong problem-solving, debugging, communication, and stakeholder-collaboration skills.
Preferred Skills
- Knowledge of Databricks Unity Catalog, access controls, lineage, and data governance.
- Experience with Databricks Asset Bundles, Databricks CLI, or CI/CD-based deployment approaches.
- Basic understanding of AWS IAM, Secrets Manager, CloudWatch, and Lambda.
- Experience with cloud security, secrets management, and credential-handling best practices.
- Knowledge of automated testing frameworks for PySpark and data pipelines.
- Familiarity with Git-based development, branching strategies, pull requests, and CI/CD pipelines.
- Experience working in Agile/Scrum delivery environments.
- Databricks or AWS certification would be an advantage.
Education and Experience
- Bachelors degree in Computer Science, Information Technology, Engineering, or a related discipline.
- Relevant professional experience in data engineering, including substantial hands-on experience with Databricks, PySpark, SQL, and Delta Lake.
- Experience delivering and supporting enterprise-scale data platforms in production environments.
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Section · 02
Skills
AutomationAgileApache SparkData EngineerAuditingCI/CD Pipelinesaws iamapisApacheautomated testingAws Servicescachingamazon web servicesamazon s3copilotcursorCSVAWS CloudWatchcloudAI ToolsCloud SecurityArchitectureArtificial Intelligence
Section · Company
About Happiest Minds Technologies
Happiest Minds Technologies
IT Services & Consulting
6.5k+
employees
2011
15 years old
Bangalore / Bengaluru, Karnataka
India
About
A technology firm specializing in digital transformation services utilizing advanced technologies to enhance enterprise efficiency and customer experience.
Industries
IT Services & Consulting
Employee ratings
1,441 reviews
3.4/ 5
Culture
3.4
Career growth
2.8
Work-life
3.6
Employees rate it well for
Work Life Balance210Work Culture165Senior Management138Salary104Learning80
Find them on