Data Engineer

Data Engineer | U S Staffing

Position: Data Engineer

Location: Bengaluru, Karnataka

Experience: 3–6 Years

Employment Type: Full-Time

Work Mode: On-site / Hybrid

Shift: Day Shift

Industry: IT / Software Development

Job Summary :

We are looking for an experienced Data Engineer with strong hands-on expertise in building and maintaining scalable data pipelines and data processing solutions.

The ideal candidate should have solid experience with Python, SQL, ETL/ELT, Apache Spark, Databricks, cloud platforms (AWS/Azure), and modern data warehousing technologies. The candidate will work closely with data analysts, data scientists, software engineers, and business teams to deliver reliable and high-quality data solutions.

Key Responsibilities :

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Collect, transform, integrate, and process large volumes of structured and unstructured data.
• Develop data-processing solutions using Python, SQL, and Apache Spark/PySpark.
• Build and optimize data pipelines using Databricks.
• Design and maintain data models, data warehouses, and data lake solutions.
• Develop complex SQL queries, stored procedures, and data transformations.
• Integrate data from databases, APIs, cloud storage, applications, and third-party sources.
• Perform data cleansing, validation, transformation, and quality checks.
• Monitor data pipelines and troubleshoot failures and performance issues.
• Optimize data-processing workloads for performance, scalability, and cost.
• Implement data quality, governance, security, and access-control standards.
• Work with cloud platforms such as AWS or Microsoft Azure.
• Develop and maintain workflow orchestration using Apache Airflow / Azure Data Factory / AWS Glue.
• Implement automated deployment and CI/CD processes for data pipelines.
• Collaborate with Data Analysts, Data Scientists, BI Developers, and application teams.
• Create and maintain technical documentation for pipelines, data models, and architecture.
• Participate in Agile/Scrum meetings, code reviews, and technical discussions.

Required Technical Skills :

Programming & Database

• Python
• SQL
• PySpark / Apache Spark
• Advanced SQL queries and performance optimization

Data Engineering

• ETL / ELT
• Data Pipelines
• Data Integration
• Data Modeling
• Data Warehousing
• Data Lakes / Lakehouse architecture
• Batch and streaming data processing

Cloud

• AWS or Azure
• AWS: S3, Glue, Lambda, Redshift, EMR
• Azure: Data Factory, ADLS, Synapse Analytics, Databricks

Big Data & Modern Data Platforms

• Apache Spark
• Databricks
• Delta Lake
• Kafka knowledge is preferred
• Snowflake experience is an advantage

Databases

• PostgreSQL / MySQL / SQL Server / Oracle
• Snowflake / Redshift / Azure Synapse
• NoSQL knowledge is an advantage

Tools

• Git
• Airflow
• Docker
• CI/CD
• Jenkins / Azure DevOps / GitHub Actions

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
• 3–6 years of professional Data Engineering experience.
• Strong hands-on experience with Python and SQL.
• Experience developing production-level ETL/ELT pipelines.
• Strong knowledge of Spark/PySpark and Databricks.
• Hands-on experience with AWS or Azure cloud services.
• Good understanding of data warehousing and data modeling concepts.
• Experience handling large datasets and optimizing data-processing workloads.
• Strong analytical and troubleshooting skills.
• Good written and verbal communication skills.
• Ability to work effectively with cross-functional teams.

Preferred Skills

• Snowflake
• Kafka
• Delta Lake
• Apache Airflow
• dbt
• Kubernetes
• Terraform
• Power BI / Tableau
• Real-time/streaming data pipelines
• Data governance and security
• Cloud certification in AWS or Azure

Key Skills

Data Engineering | Python | SQL | PySpark | Apache Spark | Databricks | ETL | ELT | Data Pipelines | AWS | Azure | Snowflake | Airflow | Kafka | Data Warehousing | Data Lake | Delta Lake | Data Modeling | CI/CD | Git