Data Engineer

Role Summary

Design, build, and optimize Azure data pipelines and lakehouse solutions. Deliver secure, reliable datasets with strong governance, automation, and documentation. Collaborate across teams and contribute to standards in an Agile setting.

Must-Have (Day 1)

• Experience: 46 years in data engineering
• Core Platform: Databricks with Python, Spark, Pandas (notebooks and modular code)
• Orchestration: Azure Data Factory (pipelines, integration runtimes); ingest from diverse sources
• Lakehouse: Delta Lake fundamentals; Medallion architecture (bronze/silver/gold) in production
• Storage/SQL/Performance: Azure Data Lake Storage (ADLS); strong SQL; performance-aware design
• Data Patterns: ETL/ELT; data modeling (e.g., dimensional/star schema)
• DevOps Security: CI/CD for data projects (Azure DevOps or GitHub Enterprise); familiarity with Azure Entra ID for SSO/RBAC; secure workspace/data access
• Quality Observability: Data validation/testing, code reviews, and basic monitoring/alerting for jobs/pipelines
• Ways of Working: Agile/Scrum (Jira/Confluence); clear pipeline and data contract documentation
• Collaboration: Effective stakeholder engagement; support/mentor junior team members; clear communication
• Generative AI (Day 1):
  • Prompt design for data tasks (ingestion, transformations, documentation) with clear objectives and constraints
  • Use of Copilot/ChatGPT to scaffold notebooks/jobs, generate tests, and optimize SQL/Sparkvalidates outputs before merging

Nice-to-Have (Train within 6090 days)

• Unity Catalog migration (Hive to Unity) and permissions/governance
• Databricks DevOps (cluster configuration, secret management, workspace automation)
• Azure Functions (C# or Python) for orchestration/integration
• Synapse dedicated SQL pools or dbt; Delta Live Tables
• Financial services domain exposure

Shared Expectations

• Work independently with minimal supervision while contributing to team outcomes
• Commitment to secure practices and production-grade reliability
• Continuous improvement mindset and willingness to learn new tools/technologies
• Willingness to work within regulated environment controls and policies
• Use Generative AI responsibly to improve velocity and quality (simple, structured prompts; guardrails; validate AI-assisted outputs before adoption)

Experience

6 - 10 Years

Job Details

• Job Type: Full time
• Location: Telengana, India
• Role: Data Engineer

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.