GURUGRAM · FULLTIME
AT&T Databricks - Lead
Prodapt Solutions
Gurugram · onsite · Posted 4d ago
Sourced from
Undisclosed8–13 yrsfulltimeGurugram
Your match
?
Sign in to see your match score, skill gaps & tailored resume.
Section · 01
About this role
Overview
Responsibilities
1. Technical Leadership Solution Design
2. Palantir Foundry Databricks Migration
3. Databricks Engineering
4. Data Architecture Governance
5. Team Leadership Mentoring
6. Customer Stakeholder Management
7. Delivery Program Execution
8. DevOps Production Engineering
Requirements
Databricks - Mandatory
Data Engineering
Programming
Cloud
DevOps
We are looking for a Senior Lead - Databricks Data Modernization to lead the design and implementation of a data modernization program involving both new data feature development in Databricks and the migration of existing capabilities from Palantir Foundry to Databricks.
The role requires a strong combination of Databricks engineering, data architecture, technical leadership and program execution skills. The candidate will be responsible for translating business and technical requirements into scalable Databricks solutions, guiding the engineering team, driving the Palantir-to-Databricks migration, and ensuring high-quality delivery.
The Senior Lead will work closely with customer stakeholders, enterprise/data architects, product owners, engineering teams and AI/ML teams and will be expected to take ownership of technical decisions and end-to-end delivery.
Responsibilities
1. Technical Leadership Solution Design
- Lead the design and development of scalable Databricks-based data solutions.
- Translate business requirements into technical architecture, data models and implementation plans.
- Define appropriate patterns for data ingestion, transformation, processing, storage and serving.
- Establish engineering standards, coding standards and reusable frameworks for the Databricks development team.
- Make technology and design decisions balancing performance, scalability, maintainability, security and cost.
- Review technical designs and code produced by the engineering team.
- Identify opportunities to modernize and simplify existing solutions.
2. Palantir Foundry Databricks Migration
- Lead the technical assessment and migration of existing Palantir Foundry capabilities to Databricks.
- Analyze existing Foundry pipelines, datasets, transformations, business rules and dependencies.
- Develop migration strategies and target-state architectures for different Foundry capabilities.
- Define appropriate Databricks-native replacements rather than simply performing a one-to-one migration.
- Identify migration complexity, dependencies, risks and technical gaps.
- Develop migration plans covering:
- Data ingestion
- Data transformation
- Data quality
- Data comparison/reconciliation
- Data models
- APIs/integrations
- Downstream analytics and applications
- Lead source-to-target mapping and functional/data validation.
- Establish approaches for achieving data and functional parity between Palantir and Databricks.
- Identify opportunities to improve performance, scalability and maintainability during migration.
3. Databricks Engineering
- Provide hands-on technical leadership using:
- Databricks
- Apache Spark
- PySpark
- Spark SQL
- Python
- Delta Lake
- Unity Catalog
- Lakeflow / Spark Declarative Pipelines
- Databricks Workflows
- Design and implement batch and near-real-time data pipelines.
- Implement incremental processing, CDC and SCD patterns.
- Design and optimize Medallion Architecture implementations.
- Implement data quality, validation, reconciliation and error-handling frameworks.
- Optimize Spark workloads and Delta tables for performance and cost.
- Ensure production-grade engineering practices across the team.
4. Data Architecture Governance
- Work with enterprise architects to define the target Databricks data architecture.
- Define appropriate Bronze, Silver and Gold data layers.
- Establish data modelling and data integration standards.
- Ensure appropriate use of Unity Catalog for governance, access control and lineage.
- Define standards for data quality, metadata, lineage and observability.
- Ensure solutions comply with enterprise security and governance requirements.
5. Team Leadership Mentoring
- Lead and mentor a team of Databricks/data engineers.
- Break down requirements into technical tasks and assign work across the team.
- Provide technical guidance and unblock engineers on complex issues.
- Conduct design and code reviews.
- Establish engineering best practices and reusable components.
- Support capability development and knowledge sharing within the team.
- Act as the primary technical escalation point for the engineering team.
6. Customer Stakeholder Management
- Work directly with customer technical and business stakeholders.
- Lead technical discussions, requirement clarification and solution workshops.
- Present architecture, migration approaches and technical recommendations to senior stakeholders.
- Translate complex technical issues into clear business implications.
- Manage technical dependencies and communicate risks proactively.
- Work with customer teams to prioritize migration and new feature development.
- Build strong working relationships with customer architecture, data and application teams.
7. Delivery Program Execution
- Own technical delivery for assigned workstreams.
- Develop technical delivery plans, estimates and milestones.
- Identify technical risks, dependencies and mitigation strategies.
- Track progress against technical deliverables.
- Ensure solutions meet agreed quality, performance and security standards.
- Coordinate with project/program managers to ensure successful delivery.
- Support release planning, production deployment and transition to operations.
8. DevOps Production Engineering
- Establish CI/CD practices for Databricks solutions.
- Implement Git-based development and deployment processes.
- Use Databricks Asset Bundles and appropriate deployment automation.
- Define automated testing strategies for data pipelines.
- Establish monitoring, logging and alerting standards.
- Lead root-cause analysis of production issues.
- Drive performance and cost optimization of Databricks workloads.
Requirements
Databricks - Mandatory
- Strong hands-on experience with Databricks
- Apache Spark / PySpark
- Spark SQL
- Delta Lake
- Unity Catalog
- Databricks Workflows / Jobs
- Lakeflow / Spark Declarative Pipelines
- Databricks Asset Bundles
- Databricks performance optimization
- Medallion Architecture
Data Engineering
- Large-scale data processing
- ETL/ELT architecture
- Data modelling
- Batch and incremental processing
- CDC
- SCD Type 1 / Type 2
- Data quality and reconciliation
- Data integration
- Structured and unstructured data processing
Programming
- Strong Python
- Strong SQL
- Good understanding of software engineering principles
- Experience developing reusable and production-grade frameworks
Cloud
Strong experience in Azure Cloud
Experience with cloud storage, networking, security and cloud data services is preferred.
DevOps
- Git
- CI/CD
- Automated testing
- Databricks deployment automation
- Infrastructure-as-code concepts
- Production monitoring and operational support
Hands-on experience with Palantir Foundry will be strongly preferred because the project involves migration of existing Foundry capabilities.
Sourced from naukri
Let the agent run this one for you.
Tailored resume, auto-apply, and referral lookup — in under 2 minutes.
Section · 02
Skills
CdcAutomationApache SparkData ingestioncoding standardsapisApacheautomated testingdata architectureCloud StorageBusiness RequirementsAnalyticsCloud DataData EngineeringcloudArchitectureMicrosoft AzureArtificial Intelligence
Section · Company
About Prodapt
Prodapt Solutions
IT Services & Consulting
3.0k+
employees
1999
27 years old
Chennai, Tamil Nadu
India
About
Prodapt Solutions specializes in connected technology services, supporting telecom operators and digital service providers in optimizing their networking capabilities.
Industries
IT Services & ConsultingTelecom
Employee ratings
1,231 reviews
3.5/ 5
Culture
3.4
Career growth
2.7
Work-life
3.5
Employees rate it well for
Salary134Work Life Balance134Senior Management121Learning95Work Culture88
Find them on