Scala with AWS

Experience: 6-12 Years

Design, develop, and optimize scalable data pipelines using scala and Pyspark in AWS environment.

• Build and maintain batch and real-time data ingestion and processing frameworks.

• Develop enterprise-grade data warehousing solutions using Scala and Pyspark.

• Analyze existing Scala and Spark applications and identify migration requirements.

• Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.

• Optimize PySpark jobs for performance, scalability, and resource utilization.

• Support cloud modernization initiatives on AWS/Databricks/Snowflake platforms.

• Implement ETL/ELT processes for structured and unstructured data.

• I ntegrate data from multiple sources including databases, APIs, files, and streaming platforms.
• • Ensure data quality, governance, security, and compliance across data platforms.
• • Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).
• • Monitor data pipelines and troubleshoot production issues
• . • Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.e