AWS Databricks

In person Drive - Kochi - Jun 27th

Job Summary Architect role for an experienced professional with strong expertise in GenAI fundamentals Python Databricks platform PySpark and Amazon S3 responsible for designing scalable hybrid analytics and AI solutions that support business intelligence outcomes. The role focuses on robust data architectures secure pipelines and reliable workflows that enable advanced insights in a collaborative hybrid work model.

Responsibilities

• Design robust end to end data and analytics architectures that leverage Databricks SQL Databricks Delta Lake and PySpark to deliver scalable reporting and advanced analytics solutions for business teams.
• Develop secure and efficient data ingestion patterns from Amazon S3 and other enterprise sources that ensure reliable data availability optimized storage usage and predictable performance across environments.
• Define best practices for organizing and managing Delta Lake tables including partitioning and optimization strategies to support high performance query workloads and downstream machine intelligence use cases.
• Implement Databricks Workflows to orchestrate complex batch and near real time data processing pipelines that provide timely high quality information to analytics and reporting stakeholders.
• Apply GenAI fundamentals to design solution patterns where foundation models and generative capabilities can augment analytics reporting narratives and self service insights in a safe and governed manner.
• Collaborate with BI and AI product teams using AI BI Genie capabilities to design semantic layers reusable data models and guided analytics experiences that simplify data consumption for business users.
• Create Python and PySpark framework components that standardize logging error handling configuration management and testing so that engineering teams can deliver data pipelines faster and with fewer defects.
• Review solution designs notebooks SQL logic and workflow configurations from engineering teams to ensure consistency with architecture standards security policies and regulatory requirements.
• Partner with information security and platform operations teams to align data architectures with identity management encryption data masking and monitoring controls that protect sensitive information.
• Work with product owners and business stakeholders to translate complex analytical needs into clear solution designs data contracts and service level expectations that can be implemented on the Databricks platform.
• Guide optimization of Databricks clusters job configurations and SQL queries to control cost maximize performance and ensure reliable operation within the hybrid work environment and day shift coverage.
• Document architecture patterns reference implementations and decision records in a structured and accessible way so that engineering teams across the organization can reuse proven designs.
• Mentor data engineers and analytics developers on Databricks PySpark GenAI basics and cloud data design principles to build a strong internal community of practice focused on quality and innovation.

Qualifications

• Require extensive hands on experience in designing and implementing data solutions using Databricks SQL Databricks Delta Lake Databricks Workflows and PySpark for large scale analytics needs.
• Require strong proficiency in Python programming for data processing automation scripts and integration tasks including familiarity with modular coding practices and testing techniques.
• Require practical knowledge of GenAI basics with the ability to identify where generative techniques can enhance analytics workflows data exploration and business decision support.
• Require proven experience working with Amazon S3 as a primary cloud storage layer including secure data organization lifecycle strategies and integration with data processing platforms.
• Require prior background in building or supporting AI and BI solutions using tools such as AI BI Genie or similar platforms that enable advanced visualizations and insight generation.
• Require minimum twelve years of overall experience in data engineering analytics or architecture roles with at least several years focused on cloud based data platforms and modern data stacks.
• Prefer exposure to hybrid work models and global enterprise environments where coordination with distributed teams and stakeholders is essential for project success.
• Prefer experience defining data governance practices quality checks and documentation approaches that increase trust in analytics and support compliance objectives for the organization.