Pyspark with AWS

Design, develop, and optimize scalable data pipelines using AWS services.

• Build and maintain batch and real-time data ingestion and processing frameworks.

• Develop enterprise-grade data warehousing solutions using Py-Spark. • Convert Scala-based ETL, batch, and streaming pipelines into PySpark frameworks.

• Optimize PySpark jobs for performance, scalability, and resource utilization.

• Support cloud modernization initiatives on AWS/Databricks platforms.

Integrate data from multiple sources including databases, APIs, files, and streaming platforms.

• Ensure data quality, governance, security, and compliance across data platforms.

• Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC).

• Monitor data pipelines and troubleshoot production issues

. • Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions.

• Implement data models, metadata management, and data lineage best practices.