HYDERABAD · FULLTIME
Distinguished Engineer, Data Platforms
S&P Global Market Intelligence
Hyderabad · onsite · Posted 28d ago
Sourced from
Undisclosed10–15 yrsfulltimeHyderabad
About a minute. You'll make an account on the way through.
agent · working
- Read this posting0.6s
- Found the form on Naukri · 6 questions to answer1.0s
- Decided the answers for all of them3.1s
- Tailor your résumé to this role
- Waiting for you
Section · 01
About this role
About the Role:
Grade Level (for internal use): 13
Key Responsibilities - Data Pipeline Transition and Platform Delivery
- Partner with the core Databricks team to plan and execute the transition of existing data pipelines to the target enterprise data platform.
- Drive implementation of repeatable engineering patterns for ingestion, transformation, testing, deployment, and monitoring across onboarded datasets.
- Guide the design and operation of cloud-native data pipelines leveraging relevant AWS services such as:
- Amazon S3 for durable storage
- AWS Glue for integration and catalog-driven processing
- AWS Lambda for event-driven processing
- Amazon Kinesis for streaming use cases
- AWS Lake Formation for governed data lake controls
- Promote the use of AWS IAM, encryption, and environment-level controls to enforce secure access to platform resources and data products in line with enterprise governance expectations.
- Influence architectural decisions related to Databricks, Delta Lake, Apache Iceberg, Unity Catalog, metadata-driven processing, and governed data lake design.
- AI-Assisted Engineering and Context Engineering
- Champion the adoption of AI-powered development tools, including Claude, GitHub CoPilot, LLMs, and other AI-assisted engineering solutions, to increase engineering velocity, code quality, and operational effectiveness.
- Apply LLM-based tools to support common data engineering activities such as:
- Code generation
- Code refactoring
- SQL optimization
- PySpark optimization, etc.
- Develop and implement context engineering practices that provide AI tools with the necessary technical background, architectural constraints, data schemas, coding standards, platform patterns, security requirements, and governance expectations.
- Create reusable context packs, prompt libraries, and AI-enabled engineering playbooks for common data platform tasks, including Databricks pipeline migration, ingestion framework development, test generation, code review support, and operational troubleshooting.
- Use LLMs to accelerate understanding and modernization of legacy pipelines, including dependency analysis, code explanation, transformation logic interpretation, metadata extraction, and refactoring recommendations.
- Data Onboarding and Asset-Agnostic Enablement
- Provide technical direction for offshore teams supporting data onboarding to the enterprise platform in an asset-agnostic manner.
- Define and operationalize onboarding patterns that can support a broad range of data assets, domains, and source systems without requiring bespoke platform redesign for each use case.
- Work with partner teams to simplify and standardize how data is ingested, transformed, governed, and published to the platform.
- Establish reusable ingestion and processing patterns across batch and streaming use cases using technologies such as AWS Glue, AWS Lambda, Amazon Kinesis, or event-driven integrations where appropriate.
- Data Mastering Platform Integration
- Support integration of platform pipelines and datasets with the enterprise data mastering platform.
- Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows.
- Help establish data quality controls, reconciliation processes, metadata alignment, and stewardship workflows required to support trusted mastered data in the platform.
- Contribute to issue resolution and continuous improvement related to mastering-related ingestion and distribution workflows.
- Support data mastering capabilities aligned with platforms such as NeoXam DataHub, including: Data acquisition, Cleansing, Enrichment, Mastering, Reconciliation, Golden copy generation Downstream distribution of trusted data products.
- Semantic Modeling and Data Discoverability
- Support semantic modeling practices that help create consistent business definitions, reusable data concepts, and governed consumption patterns across the enterprise data platform.
- Partner with business, architecture, data governance, and platform teams to align technical data structures with business-friendly semantic definitions.
- Contribute to defining common entities, attributes, relationships, hierarchies, metrics, and business terms across key data domains such as instruments, issuers, accounts, portfolios, risk, market data, reference data, and investment data.
- Help connect semantic modeling concepts with metadata management, business glossaries, data catalogs, lineage, and governed data products.
- Technical Influence and Engineering Excellence
- Serve as a senior technical expert for offshore data platform engineering, providing guidance on architecture, design patterns, implementation quality, and production readiness.
- Influence architectural decisions across pipeline migration, lakehouse design, cloud-native data engineering, metadata-driven processing, AI-assisted engineering, and governed data platform operations.
- Provide technical guidance to engineers and partner teams without direct people management responsibility.
- Contribute to design reviews, code reviews, implementation planning, and technical problem-solving for complex data platform initiatives.
- Support delivery execution by translating broader architectural direction into practical implementation patterns, engineering tasks, reusable technical assets, and AI-enabled productivity accelerators.
- Drive rigor across sprint delivery, production readiness, support models, and incident management for business-critical data platforms.
- Collaboration, Governance, and Platform Standards
- Partner effectively with global platform, architecture, governance, security, data mastering, and Databricks-aligned teams to ensure offshore delivery aligns with enterprise standards.
- Contribute to technical discussions related to pipeline architecture, onboarding frameworks, cloud platform controls, data governance, AI-assisted engineering practices, semantic modeling, and mastering integrations.
- Support governance requirements through appropriate controls around lineage, schema consistency, data quality, retention, access control, auditability, metadata, and secure data distribution.
- Help establish cloud engineering standards for infrastructure as code, release automation, and environment promotion using tools and services such as AWS CodePipeline, AWS CodeBuild, and infrastructure automation frameworks where appropriate.
- Act as a technical bridge between offshore execution teams and global platform stakeholders to ensure alignment, reduce ambiguity, and accelerate delivery.
- Help define standards for responsible AI use in engineering workflows, including review practices, sensitive data handling, secure prompt construction, validation requirements, and production deployment controls.
- 10+ years of experience in data engineering, data platforms, cloud data architecture, software engineering, or
Sourced from naukri
Section · 02
Skills
Aws GlueAws LambdaAutomationclaudecloud engineeringaws iamApacheautomated testingAws ServicesAutomotiveamazon web servicesamazon s3AnalyticsCloud DataCapital MarketscloudAI ToolsArchitectureAWS CertifiedArtificial Intelligence