MS Fabric + Pyspark Tuning - S

Key Responsibilities

• Design, develop, and optimize data solutions using Microsoft Fabric and PySpark.
• Analyze and tune Spark workloads for performance, scalability, and cost efficiency.
• Develop and maintain Lakehouse architectures for enterprise analytics platforms.
• Optimize data ingestion, transformation, and processing pipelines.
• Perform Spark code reviews and recommend performance improvements.
• Implement partitioning, caching, indexing, and file optimization techniques.
• Monitor and troubleshoot Fabric workloads, Spark jobs, and resource utilization.
• Collaborate with data architects, data scientists, and business stakeholders.
• Design robust ETL/ELT frameworks using Fabric Data Factory and Notebooks.
• Ensure data quality, governance, security, and compliance standards.
• Implement best practices for distributed computing and big data processing.
• Create technical documentation, optimization reports, and performance metrics dashboards.

Required Skills

Microsoft Fabric

• Microsoft Fabric
• Fabric Data Engineering
• Fabric Lakehouse
• Fabric Data Factory
• Fabric Notebooks
• Fabric Real-Time Analytics
• Fabric OneLake
• Fabric Warehouse

PySpark & Spark Optimization

• PySpark Development
• Apache Spark Architecture
• Spark SQL
• Spark DataFrames
• Spark Structured Streaming
• DAG Optimization
• Catalyst Optimizer
• Adaptive Query Execution (AQE)
• Broadcast Joins
• Partitioning & Bucketing
• Shuffle Optimization
• Caching & Persistence Techniques

Data Engineering

• ETL/ELT Development
• Data Modeling
• Data Lake Architecture
• Medallion Architecture
• Delta Lake
• Data Quality Frameworks

Programming

• Python
• PySpark
• SQL
• Scala (Preferred)

Databases

• SQL Server
• Azure SQL
• PostgreSQL
• Snowflake (Preferred)

Cloud & Analytics

• Azure Data Services
• Azure Data Lake Storage (ADLS)
• Power BI
• Azure Monitor
• Log Analytics

DevOps & Automation

• Git
• Azure DevOps
• CI/CD Pipelines
• Infrastructure as Code (Preferred)