Azure or AWS Data Engineer
Relevance Lab · Bengaluru · onsite · Posted 42d ago
Undisclosed5–10 yrsfulltimeSourced from
About a minute. You'll make an account on the way through.
About this role
We are looking for a highly skilled Lead Data Engineer with expertise in Azure or AWS and Databricks to join our team. The ideal candidate will lead the design, development, and implementation of data engineering solutions, ensuring scalability, security, and efficiency in our data infrastructure. This role requires strong technical skills, and experience in managing large-scale data processing pipelines.
Key Responsibilities - Lead the design and development of scalable and reliable data pipelines using Azure Data Services or AWS Data Services and Databricks.
- Architect, implement, and optimize ETL/ELT processes to process large volumes of structured and unstructured data.
- Develop and maintain data models, data lakes, and data warehouses to support analytics and business intelligence needs.
- Collaborate with data scientists, analysts, and business stakeholders to ensure data availability and integrity.
- Implement and enforce data governance, security, and compliance best practices.
- Optimize and monitor performance of data processing frameworks (Spark, Databricks, etc.).
- Automate and orchestrate data workflows using tools such as Apache Airflow, Azure Data Factory, AWS Step Functions, or Glue.
- Guide and mentor junior data engineers in best practices and modern data engineering techniques.
- 5+ years of experience in data engineering
- Strong expertise in Azure Data Services (Azure Data Lake, Azure Synapse, Azure Data Factory) or AWS Data Services (S3, Redshift, Glue, Lambda, Step Functions, EMR).
- Proficiency in Databricks and experience with Apache Spark for large-scale data processing.
- Strong programming skills in Python.
- Experience working with SQL and NoSQL databases such as PostgreSQL, MySQL, DynamoDB, or Cosmos DB.
- Solid understanding of data governance, security, and compliance (GDPR, HIPAA, etc.) is a plus.
- Experience with real-time streaming technologies such as Kafka, Kinesis, or Event Hubs is a plus.
- Excellent problem-solving skills and the ability to work in a fast-paced, agile environment.
- Experience with machine learning pipelines and MLOps.
- Familiarity with data visualization and BI tools like Power BI, Tableau, or Looker.
- Strong communication and leadership skills to drive best practices across the team.
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Skills
AgileApache SparkData EngineerAzure Data ServicesApachebi toolsamazon web servicesAWS data engineerCosmos DBapache airflowCOSMOSAnalyticsAzure SynapseAzure Data FactoryAzure Data LakecomplianceMicrosoft AzureBusiness Intelligence
About Relevance Lab
Relevance Lab
IT Services & Consulting
286
employees
2011
15 years old
Singapore
Singapore
₹13.6L PA avg
Avg at Relevance Lab
About
Relevance Lab is a specialized IT services company with re-usable technology assets in the cloud, DevOps, automation & analytics domain.
Industries
IT Services & ConsultingAnalytics & KPO
Employee ratings
10 reviews
3.6/ 5
Culture
2.6
Career growth
2.8
Work-life
3.2
Employees rate it well for
Work-Life Balance37Overall Company36
Find them on