Specialist - Business Intelligence

Job Summary

Specialist

Function: Information Technology

Work Location: Hyderabad

Job Details

• Job Title Specialist
• JD Written by Vinod R Iyer
• Reporting To Vinod R Iyer
• JD Approved by Human Resources
• Brand Multiple

Nature of the Job

• AWS Data Engineer role is highly technical, focusing on building, maintaining, and optimizing data pipelines and infrastructure within the AWS cloud environment.
• It involves leveraging AWS technologies to ensure smooth data processing and infrastructure management, requiring problem-solving and analytical skills, as well as collaboration across teams.

Job Purpose

Data Engineers work as part of the Data Platform Team, collaborating closely with data scientists, software engineers, and analysts across the company. They assist in building and maintaining the tooling and infrastructure that support decision-making processes and enable product enhancements by ensuring a reliable and unified view of business data. Under the guidance of senior team members, they help prioritize tasks to ensure key features are developed in the proper sequence. They also contribute to the balance between supporting the immediate needs of strategic teams and Data Analysts while participating in the ongoing development of the company's data infrastructure.

Technology Stack

• Our tech stack consists of a core ETL process written in Python and SQL, which is being managed by AWS Services like Redshift, Athena, Step Functions, CloudFormation (Infrastructure as Code - IaC) having provided data from multiple sources like Salesforce, application logs, and other operational data sources. We use version control as Git and CI/CD pipelines with AWS CodeCommit, CodePipeline and CodeBuild.

Principal Responsibilities

• This Data Engineer supports our Senior Data Engineers and will be responsible for utilizing expertise and knowledge of AWS services to transform raw data and ingest it into data warehouses and data lakes. You will work closely with the development team and would be responsible for providing quality code.
• Reading data from multiple sources, transform and ingest data into Redshift using Python and SQL
• Working with different business units to understand requirements and collaborating with them to deliver data pipelines
• Design, develop, and execute data analysis plans using AWS services like Amazon Redshift and Amazon Athena
• Maintaining and creating new data pipelines using AWS Step Functions
• Build and maintain infrastructure as code (IaC) in the cloud that is resilient and can scale when needed.
• Above all, we are looking for people who are curious, think critically, are eager to learn and keen to use their experience to help and support others. You will need to be able to communicate and explain things clearly and work well in a collaborative environment as well as autonomously.

KPIs and Metrics

• Data Pipeline Development and Maintenance
  • Design, build, and maintain efficient data pipelines using AWS services such as AWS Glue, Lambda, and Step Functions.
  • Ensure scalability, reliability, and efficiency of pipelines to support large datasets.
• Data Pipeline Reliability
  • % of successful data pipeline executions vs. failures.
  • Uptime percentage for critical data pipelines.
• Data Processing Time
  • Average time taken to process data from ingestion to loading into the final storage.
  • Latency for real-time data pipelines.
• Automation Coverage
  • % of data engineering tasks that are automated (e.g., pipeline monitoring, scaling).
  • Reduction in manual intervention for routine tasks.
• ETL/ELT Processes
  • Implement Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT) processes using AWS Glue, Athena, or Redshift.
• Data Availability
  • Time taken to make new data available for consumption (batch processing time, real-time data latency).
  • Frequency of data availability to business users.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.