Senior Data Engineer

Job Description:

We are looking for Senior Data Engineer with 8+Yrs of experience in Hyderabad, Bangalore, Navi Mumbai & Gurugram locations. Candidates Should have experience in Pyspark, Python, SQL & Databricks. Should be proficient in Python Programming.

Location : Navi Mumbai , Hyderabad , Gurugram , Bangalore

Work Mode : All 5 days work from office

The impact you will have:

• Guide Big Data Transformations: Implementation of comprehensive big data projects, including the development and deployment of innovative big data and AI applications.
• Ensure Best Practices: Guarantee that Databricks best practices are applied throughout all projects to maintain high-quality service and successful implementation.
• Support Project Management: Assist the Professional Services leader and project managers with estimating efforts and managing risks within customer proposals and statements of work.
• Architect Complex Solutions: Design, develop, deploy, and document complex customer engagements, either independently or as part of a technical team, serving as the technical lead and authority.
• Enable Knowledge Transfer: Facilitate the transfer of knowledge and provide training to team members, customers, and partners, including the creation of reusable project documentation.
• Contribute to Consulting Excellence: Share expertise with the consulting team and offer best practices for client engagement, enhancing the effectiveness and efficiency of other teams.

Minimum qualifications:

• Educational Background: Bachelors degree in Computer Science, Information Technology, or a related field (or equivalent experience).
• Experience: 8+ years of experience as a Data Engineer, with proficiency in at least two major cloud platforms (AWS, Azure, GCP).
• Proven experience in designing, developing, and implementing comprehensive data engineering solutions using Databricks, specifically for large-scale data processing and integration projects
• Develop scalable streaming and batch solutions using cloud-native components.
• Perform data transformation tasks, including cleansing, aggregation, enrichment,and normalization, utilizing Databricks and related technologies.
• Experience in applying DataOps principles and implementing CI/CD and DevOps practices within data environments to optimize development and deployment workflows.

Technical Skills:

• Expert-level proficiency in Spark Scala, Python, and PySpark.
• In-depth knowledge of data architecture, including Spark Streaming, Spark Core, Spark SQL, and data modeling.
• Hands-on experience with various data management technologies and tools,such as Kafka, StreamSets, and MapReduce.
• Proficient in using advanced analytics and machine learning frameworks, including Apache Spark MLlib, TensorFlow, and PyTorch, to drive data insights and solutions.

Databricks Specific Skills:

• Extensive experience in data migration from on-premises to cloud environments and in implementing data solutions on Databricks across cloud platforms (AWS,Azure, GCP).
• Skilled in designing and executing end-to-end data engineering solutions using Databricks, focusing on large-scale data processing and integration.
• Proven hands-on experience with Databricks administration and operations, including notebooks, clusters, jobs, and data pipelines.
• Experience integrating Databricks with other data tools and platforms to enhance overall data management and analytics capabilities.

Certifications:

• Certification in Databricks Engineering (Professional)
• Microsoft Certified: Azure Data Engineer Associate
• GCP Certified: Professional Google Cloud Certified.
• AWS Certified Solutions Architect Professional