Specialist, Data Engineer (Databricks)

99843

Specialist, Data Engineer (Databricks)

5-8 Years

20 LPA Fixed + Benefits

Hybrid

Raheja Mindspace, Hyderabad

099843

Specialist, Data Engineer - India

Specialist Data Engineer (Python and Databricks)

• Experience: 5-8 Years
• Location: Hyderabad (Hybrid)

Job Description

As a Data Engineer you ll be responsible for acquiring, curating, and publishing data for analytical or operational uses. Data should be in a ready-to-use form that creates a single version of the truth across all data consumers, including business users, data scientists, and Technology. Ready-to-use data can be for both real time and batch data processes and may include unstructured data. Successful data engineers have the skills typically required for the full lifecycle software engineering development from translating requirements into design, development, testing, deployment, and production maintenance tasks. You ll have the opportunity to work with various technologies from big data, relational and SQL databases, unstructured data technology, and programming languages.

Key Responsibilities:

• • Provides basic to moderate technical consultation on data product projects by analyzing end to end data product requirements and existing business processes to lead in the design, development and implementation of data products.
  • Produces data building blocks, data models, and data flows for varying client demands such as dimensional data, standard and ad hoc reporting, data feeds, dashboard reporting, and data science research & exploration.
  • Applies secure software and systems engineering practices throughout the delivery lifecycle to ensure our data and technology solutions are protected from threats and vulnerabilities.
  • Translates business data stories into a technical story breakdown structure and work estimate so value and fit for a schedule or sprint is determined.
  • Creates simple to moderate business user access methods to structured and unstructured data by such techniques such as mapping data to a common data model, NLP, transforming data as necessary to satisfy business rules, AI, statistical computations and validation of data content.
  • Assists the enterprise DevSecOps team and other internal organizations on CI/CD best practices experience using JIRA, Jenkins, Confluence etc.
  • Implements production processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
  • Develops and maintains scalable data pipelines for both streaming and batch requirements and builds out new API integrations to support continuing increases in data volume and complexity.
  • Writes and performs data unit/integration tests for data quality with input from a business requirements/story, creates and executes testing data and scripts to validate that quality and completeness criteria are satisfied. Can create automated testing programs and data that are re-usable for future code changes.
  • Practices code management and integration with engineering Git principle and practice repositories.

Education:
Undergraduate studies in computer science, management information systems, business, statistics, math, a related field or comparable experience and education strongly preferred. Graduate studies in business, statistics, math, computer science or a related field are a plus.

Experience:

Five years of relevant experience with data quality rules, data management organization/standards, practices, and software development. Experience in data warehousing, statistical analysis, data models, and queries. One to three years experience with Cloud technology and infrastructure including security and access management.

Additional JD

We are looking for a hands-on Data Engineer with strong expertise in Python, AWS, SQL, and Databricks to develop and support scalable data solutions, data pipelines, and cloud-based data platforms.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL processes.
• Develop data integration, data cleansing, and data quality solutions.
• Build and support data warehousing solutions.
• Create and maintain data quality rules, mappings, and validations.
• Write and optimize complex SQL queries.
• Develop Python-based data engineering solutions.
• Support AWS cloud-based data platforms.
• Troubleshoot production data issues and drive resolutions.
• Work closely with cross-functional teams in Agile environments.
• Follow SDLC, CI/CD, and GitHub best practices.

Must Have Skills

• Python (2+ Years)
• AWS Cloud (3+ Years)
• Databricks (2+ Years)
• Strong SQL (Complex Query Writing & Optimization)
• Data Warehousing
• ETL / Data Integration
• Data Quality & Data Cleansing
• Oracle and/or MS SQL Troubleshooting
• Production Support Experience
• Agile / Scrum Methodology
• SDLC Experience

Good to Have Skills

• CI/CD Pipelines
• GitHub
• Pharos
• Perl
• AI Tools

Ideal Candidate

• 5-8 years of Data Engineering experience.
• Strong hands-on coding experience in Python and SQL.
• Experience building scalable enterprise data solutions.
• Strong troubleshooting and problem-solving abilities.
• Experience working in cloud-native data environments.

Technical Skills:

• • Amazon Web Services
  • Data Engineering
  • Data Management
  • Data Pipelines
  • Data Quality
  • Extract Transform Load (ETL)
  • Programming Skills
  • Business Acumen

Soft Skills & Attributes:

• • Data application and practice knowledge
  • Solid communication skills
  • Ability to influence and set priorities
  • Mentoring and coaching abilities to guide associates
  • Identifying and removing blockers to progress
  • Ability to work in agile and global environments
  • Insurance/financial services industry knowledge a plus
  • Other criteria, including leadership skills, competencies and experiences may take precedence