HYDERABAD · FULLTIME

Big Data Engineer

Purview India Consulting And Services Llp

Hyderabad · onsite · Posted 58d ago

Sourced from Foundit

Not disclosed6–11 yrsfulltimeHyderabad

About a minute. You'll make an account on the way through.

agent · working

Read this posting0.2s
Found the form on Foundit · 11 questions to answer1.7s
Decided the answers for all of them2.9s
Tailor your résumé to this role
Waiting for you

Section · 01

About this role

We are seeking a skilled Big Data Engineer to join our team in India. The ideal candidate will have extensive experience in designing and implementing robust big data solutions that support our data-driven decision-making processes.

Job description

Design, build, and maintain scalable ETL/ELT pipelines using AWS Glue, Lambda, Step Functions, and Apache Spark.
Develop and optimize data lake and data warehouse architectures using AWS S3, Redshift, and Athena.
Implement data ingestion, transformation, and storage solutions using AWS Glue, Kinesis, and Kafka.
Work with structured and unstructured data to develop efficient data pipelines.
Optimize query performance in SQL-based and NoSQL databases (Redshift, DynamoDB, Aurora, PostgreSQL, etc.).
Ensure data security, compliance, and governance using IAM roles, KMS encryption, and AWS Lake Formation.
Automate data workflows using Terraform, CloudFormation, or CDK.
Monitor and troubleshoot data pipelines, ensuring high availability and performance.
Collaborate with Data Scientists, Analysts, and Engineers to provide robust data solutions.

Required Skills:

Strong experience with AWS Data Services (Glue, Redshift, S3, Lambda, Kinesis, Athena).
Proficiency in Python, SQL, and Scala
Experience with ETL/ELT pipeline design and data warehousing concepts.
Strong knowledge of relational and NoSQL databases (PostgreSQL, MySQL, DynamoDB).
Familiarity with DevOps practices and CI/CD tools (Jenkins, GitHub Actions).
Experience with Kafka or Kinesis for real-time data streaming.
Knowledge of Terraform, CloudFormation, or CDK for infrastructure automation.
Hands-on experience with data security and governance.
Strong analytical and problem-solving skills

Sourced from foundit

Section · 02

Skills

Apache SparkScalaApache Kafkaetl toolsamazon web servicesApache HadoopPythonSQLData Modeling

Job description

Design, build, and maintain scalable ETL/ELT pipelines using AWS Glue, Lambda, Step Functions, and Apache Spark.

Develop and optimize data lake and data warehouse architectures using AWS S3, Redshift, and Athena.

Implement data ingestion, transformation, and storage solutions using AWS Glue, Kinesis, and Kafka.

Work with structured and unstructured data to develop efficient data pipelines.

Optimize query performance in SQL-based and NoSQL databases (Redshift, DynamoDB, Aurora, PostgreSQL, etc.).

Ensure data security, compliance, and governance using IAM roles, KMS encryption, and AWS Lake Formation.

Automate data workflows using Terraform, CloudFormation, or CDK.

Monitor and troubleshoot data pipelines, ensuring high availability and performance.

Collaborate with Data Scientists, Analysts, and Engineers to provide robust data solutions.

Required Skills:

Strong experience with AWS Data Services (Glue, Redshift, S3, Lambda, Kinesis, Athena).

Proficiency in Python, SQL, and Scala

Experience with ETL/ELT pipeline design and data warehousing concepts.

Strong knowledge of relational and NoSQL databases (PostgreSQL, MySQL, DynamoDB).

Familiarity with DevOps practices and CI/CD tools (Jenkins, GitHub Actions).

Experience with Kafka or Kinesis for real-time data streaming.

Knowledge of Terraform, CloudFormation, or CDK for infrastructure automation.

Hands-on experience with data security and governance.

Strong analytical and problem-solving skills