Senior Software Engineer - Python, Data Engineering, AI

Responsibilities

• Design, build, and operate batch and incremental ETL/ELT pipelines in Python (Glue Python-shell, PySpark, AWS Batch/Docker, Lambda).
• Model curated and Iceberg tables; write and tune Trino/Athena SQL; own partitioning, compaction, and cost/performance of the lake.
• Build ingestion from external APIs (Salesforce, Adyen, Intercom, Jira, New Relic, etc.) with incremental anchors, retries, and idempotent writes.
• Extend the data-quality framework and anomaly-detection checks; own alerting and on-call for pipeline health.
• Lead workstreams on the Databricks-on-Azure lakehouse: Delta Lake tables, Databricks Workflows/Jobs, Unity Catalog, and migration of existing curation logic.
• Ship via PRs with tests, infra-as-code (CloudFormation / Terraform), and CI/CD; participate in code review and design reviews.
• Partner with Product, Finance, and Customer Success analysts on metric definitions; expose datasets to QuickSight and to the AI/MCP layer with clear, documented semantics.
• Mentor junior engineers and raise the bar on engineering practices (testing, observability, documentation).

Must have

• 5-7 years of data engineering in production: building, deploying, and operating pipelines that other teams depend on daily. Analyst, BI-developer, or drag-and-drop ETL-tool-only experience does not count toward this.
• Strong Python (3.x): pandas/pyarrow, packaging, virtual envs, writing testable modules and shared libraries - not just notebooks.
• Apache Spark / PySpark at scale: DataFrame API, partitioning, joins/skew, shuffle tuning, reading/writing Parquet.
• Advanced SQL on a distributed engine (Trino/Athena, Spark SQL, Databricks SQL, BigQuery, Snowflake, or Redshift): window functions, CTEs, incremental/merge patterns, query-plan-level tuning.
• Lakehouse fundamentals: columnar formats (Parquet), partitioning strategies, and hands-on experience with at least one open table format - Apache Iceberg or Delta Lake (schema evolution, time travel, compaction/OPTIMIZE, MERGE INTO).
• Cloud data platform on AWS or Azure - at minimum object storage (S3/ADLS), serverless or managed compute (Glue/EMR/Lambda or ADF/Synapse/Functions), a catalog (Glue Data Catalog / Unity Catalog / Hive), and IAM/RBAC basics.
• Orchestration of DAG-based workflows with dependency management, retries, and failure alerting (Step Functions, Airflow, Databricks Workflows, Dagster, or equivalent).
• Incremental ingestion from REST APIs and databases: pagination, rate limits, watermark/anchor-based CDC, idempotent upserts, backfill design.
• Git + PR-based workflow + CI/CD - youve shipped through a review gate and a promotion path (dev qa prod) and can debug a failing build.
• Data quality observability mindset: row-count/freshness/schema checks, alerting on failures, and root-causing a bad number in a dashboard back to its source.

Good to have

• Databricks hands-on (any cloud): Delta Live Tables / Lakeflow, Unity Catalog, Workflows, Photon, cluster/SQL-warehouse sizing, Databricks Asset Bundles. Databricks Data Engineer Associate/Professional certification is a plus.
• Azure data stack: ADLS Gen2, Azure Data Factory, Azure Key Vault, Entra ID service principals, Azure DevOps or GitHub Actions for deployment.
• AWS depth: Glue (Python-shell and Spark), Athena v3/Trino internals, Iceberg on Athena, Step Functions, Batch, ECR, CloudFormation, CodeBuild/CodePipeline, Secrets Manager.
• Infrastructure as code: CloudFormation, Terraform, or Bicep.
• Docker for packaging batch jobs; basic familiarity with .NET-based jobs coexisting in a Python pipeline.
• Migration experience - moving pipelines/data between clouds or from a hand-rolled lake to a managed lakehouse, including parity validation.
• Streaming/near-real-time: Kafka, Kinesis, Event Hubs, Spark Structured Streaming.
• Relational sources: SQL Server / MySQL extraction (pyodbc, CDC), reverse-ETL back into an application DB.
• BI serving: QuickSight, Power BI, or Tableau - dataset design, row-level security, SPICE/import vs direct query trade-offs.
• AI/LLM data surfaces: exposing governed datasets to agents via MCP or similar, vector stores (Pinecone), metadata/catalog curation for LLM consumption, dbt-style semantic modeling.
• Statistics for data quality: anomaly detection, churn/adoption scoring, or similar analytical pipelines.
• SaaS-domain familiarity: subscription billing (Zuora), payments (Adyen), CRM (Salesforce), or support/telephony data (Intercom, Gong, RingCentral).
• Experience mentoring or leading a small pod of engineers.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.