Databricks Architect

Role Overview:

• We are looking for an experienced Databricks Lead Developer / Architect with 7+ years of experience in data engineering, cloud data platforms, and distributed data processing. The ideal candidate will have strong hands-on expertise with Databricks, Apache Spark, Python, SQL, and modern data lake/lakehouse architectures, with the ability to lead technical design and development initiatives.
• Experience working with AWS cloud services and a Databricks certification will be an added advantage.

• Design and architect scalable, secure, and high-performance Databricks/Lakehouse solutions.
• Lead the development of enterprise-grade data engineering pipelines using Databricks, Apache Spark, Python, and SQL.
• Define technical architecture, development standards, best practices, and reusable frameworks.
• Design and implement data ingestion, transformation, integration, and processing pipelines.
• Work extensively with Delta Lake and develop optimized data models and lakehouse architectures.
• Optimize Spark and Databricks workloads for performance, scalability, reliability, and cost.
• Provide technical leadership and mentoring to data engineers and development teams.
• Collaborate with business stakeholders, data architects, analysts, and engineering teams to translate requirements into technical solutions.
• Implement data quality, validation, monitoring, logging, and error-handling frameworks.
• Design solutions for batch and, where applicable, streaming data processing.
• Establish CI/CD and deployment practices for Databricks workloads.
• Ensure solutions follow enterprise standards for security, governance, access control, and data management.
• Troubleshoot complex production issues and provide technical guidance to development teams.
• Evaluate new Databricks and cloud capabilities and recommend improvements to the data platform.

• 7+ years of overall experience in data engineering, software development, or data platform engineering.
• Strong hands-on experience with Databricks and Apache Spark.
• Strong programming experience in Python and SQL.
• Solid understanding of Delta Lake, Lakehouse architecture, data warehousing, and data lake concepts.
• Experience designing and developing scalable ETL/ELT data pipelines.
• Strong understanding of Spark architecture, optimization, partitioning, caching, joins, and performance tuning.
• Experience with data integration and processing of structured and semi-structured data.
• Experience with Databricks Workflows/Jobs, notebooks, clusters, and deployment processes.
• Good understanding of data modeling and distributed data processing.
• Experience with version control and CI/CD practices, preferably using Git and automated deployment pipelines.
• Strong problem-solving, analytical, communication, and technical leadership skills.
• Ability to independently own architecture and technical decisions for complex data engineering solutions.

• Experience working with AWS-based data platforms is highly desirable, including one or more of:
  • Amazon S3
  • AWS Glue
  • AWS Lambda
  • Amazon Redshift
  • Amazon EMR
  • AWS IAM
  • Amazon CloudWatch
  • AWS networking and security concepts
• Experience integrating Databricks with AWS services and designing cloud-native data architectures will be an added advantage.

• Databricks certification such as Databricks Certified Data Engineer or Databricks Certified Data Engineer Professional.
• Experience with Delta Live Tables / Lakeflow Declarative Pipelines.
• Experience with Unity Catalog and Databricks governance.
• Experience with Databricks SQL and BI/data analytics integrations.
• Experience with real-time/streaming technologies such as Kafka or Spark Structured Streaming.
• Experience with orchestration tools such as Airflow or similar platforms.
• Knowledge of infrastructure-as-code tools such as Terraform.
• Experience implementing enterprise data governance, security, and observability.
• Experience working in Agile/Scrum development environments.

• Act as the technical lead and subject-matter expert for Databricks and data engineering initiatives.
• Review technical designs and code and establish engineering best practices.
• Mentor developers and help build technical capability within the team.
• Drive architecture decisions and provide guidance on scalability, performance, security, and maintainability.
• Communicate complex technical concepts effectively to both technical and non-technical stakeholders.

Education: Bachelor s or Master s degree in Computer Science, Information Technology, Engineering, or a related field is preferred.

Experience Level: 7+ years of overall experience, with significant hands-on experience in Databricks and data engineering and preferably experience in AWS cloud environments.