Solution Architect

We are looking for a highly experienced Solution Architect – Databricks to work closely with enterprise customers in designing, developing, optimizing, and supporting scalable data engineering and analytics solutions on the Databricks platform.

The ideal candidate should have strong hands-on expertise in Databricks, Apache Spark, PySpark, distributed computing, cloud platforms, performance optimization, and solution architecture. This is a highly technical and client-facing role requiring the ability to independently lead architecture discussions, troubleshoot complex Databricks/Spark issues, and provide implementation guidance.

Key Responsibilities

• Design and implement scalable Databricks Lakehouse solutions.
• Define end-to-end data engineering and platform architecture.
• Build and optimize data pipelines using Databricks, Spark, PySpark, SQL, and Delta Lake.
• Design batch and streaming data-processing solutions.
• Provide technical guidance on Databricks architecture and best practices.
• Troubleshoot and optimize complex Spark and Databricks performance issues.
• Work with Delta Lake, Unity Catalog, Workflows, Auto Loader, Databricks SQL, Lakeflow/DLT, and Serverless.
• Support enterprise Databricks implementation and modernization initiatives.
• Design and support CI/CD processes for Databricks deployments.
• Work with Git, Terraform, Azure DevOps, GitHub, GitLab, or Jenkins.
• Provide guidance on security, governance, access control, and Unity Catalog.
• Conduct architecture reviews, code reviews, troubleshooting, and technical mentoring.
• Work closely with customer architects, engineering teams, and business stakeholders.

Required Skills

• 10+ years of overall technology/consulting experience.
• 7+ years of experience in Data Engineering, Big Data, Data Platforms, or Analytics.
• Strong hands-on experience with Databricks.
• Experience delivering 6–8+ end-to-end Databricks projects.
• Strong expertise in Apache Spark and PySpark.
• Deep understanding of Spark internals including Driver/Executors, DAG, Jobs, Stages, Tasks, Partitioning, Shuffle, Catalyst Optimizer, AQE, and Memory Management.
• Strong experience in Spark performance tuning, query optimization, data skew, partitioning, and join optimization.
• Strong knowledge of Delta Lake, Unity Catalog, Databricks Workflows, Auto Loader, and Databricks SQL.
• Strong ETL/ELT, data pipelines, data modeling, batch and streaming experience.
• Deep expertise in at least one cloud platform: AWS, Azure, or GCP.
• Working knowledge of at least one additional cloud platform.
• Knowledge of Git, CI/CD, Terraform, and Databricks Asset Bundles.
• Working knowledge of MLflow/MLOps is preferred.
• Strong customer-facing consulting and communication skills.

Preferred Qualifications

• Databricks Certified Data Engineer Professional certification.
• Experience in Databricks migration and modernization.
• Hadoop-to-Databricks migration experience.
• Cloud data warehouse-to-Databricks migration experience.
• Multi-cloud architecture exposure.
• Unity Catalog implementation experience.
• Data governance and streaming architecture experience.
• Strong technical leadership and mentoring experience.