SENIOR DATA ENGINEER - Databricks

Senior Data Engineer (6–8 years)Role Overview Hands-on IC role focused on building reliable, performant data pipelines on Azure Databricks, implementing Delta Lake patterns, and contributing to production-grade data products for analytics. Core Responsibilities • Build/maintain batch & basic streaming pipelines using PySpark/Spark SQL and Delta Lake following Medallion principles. • Ingest via Autoloader, ADF, REST, and file sources (e.g., SFTP/SharePoint). • Apply schema enforcement/evolution; add basic data quality checks and validations. • Use Unity Catalog for catalog/schema/table permissions; handle basic secrets via Key Vault/secret scopes. • Author clear code & docs; participate in PRs; contribute utilities; triage job issues and first-line performance fixes. • Operate simple Databricks Workflows (or ADF) and Git-based CI/CD basics. Must-have Skills (Screening Focus) • Strong hands-on PySpark, Spark SQL, Delta Lake fundamentals; solid SQL. • Understanding of partitions, shuffles, joins, caching; Medallion basics. • Unity Catalog basics (catalog/schema/table permissions). • Autoloader basics; ADF familiarity or Workflows basics. • Git workflows; basic tests and logging; runbook hygiene. Nice-to-have (Prefer, not Mandatory) • Exposure to DLT and Workflows orchestration patterns. • Photon awareness; cost-aware design exposure. • Healthcare/PII familiarity; Power BI integrations. 90-day Outcomes (Examples) • Ship 2 Silver?Gold pipelines with <1% failure rate; produce runbook. • Add basic data expectations (null, duplicate, type checks) with alerting on a critical table