Software Engineer II (Data Engineering)

As a Software Engineer II on the Data Ingestion team, you'll drive the design, development, and maintenance of large-scale data ingestion pipelines that power MiQ's programmatic advertising business. You will build automated, scalable, and resilient pipelines that bring data in from a wide variety of external partners (DSPs (The Trade Desk, DV360, Amazon, Meta, Yahoo, Google Ads, Beeswax, Index Exchange, Magnite/ClearLine, Adsmovil, OMC, and more), SSPs, and measurement and enrichment partners (Pixalate, Pixability, Vizio, Evertune, Lifesight, Circana, and Experian)) into MiQ's centralized Databricks lakehouse and Unity Catalog.

Responsibilities:

• Building and productionizing ingestion pipelines on Databricks (PySpark notebooks, SDP, Autoloader, Delta, etc. ) and orchestrating them via our scheduling and ingestion services.
• Contributing to our Unified Ingestion Service and our vision for self-serve ingestion.
• Onboarding new DSPs, regions, and third-party feeds; expanding reporting granularity and coverage across regions.
• Strengthening data health, lineage, and incident management workflows.
• Compliance and governance workflows.
• Release management and platform hygiene: Bitbucket/GitHub, CMRs, Terraform-managed IAM, Snyk/Akto vulnerability remediation.
• Key stakeholders include MiQ's Trading and Programmatic Ops teams, Analytics and Data Science, Product Management, Data Management and Data Governance, the Platform and IT/DevX teams, Regional Delivery and Account Management, external DSP/SSP and measurement partners, and other Data Ingestion pods responsible for downstream reporting and activation.

Requirements:

• 2-3 years of experience in data engineering and data integration, with hands-on exposure to building and managing data pipelines on cloud platforms (AWS preferred; Azure or GCP acceptable).
• Extensive hands-on experience with PySpark notebook development (preferably on Databricks) and advanced SQL to build and optimize large, high-throughput data pipelines.
• Strong working knowledge of the modern lakehouse stack Delta Lake, Delta Live Tables (DLT), Autoloader, Unity Catalog, and Hadoop-era tooling (Hive, Spark) is a plus for migration work.
• Solid understanding of data modeling, schema evolution, and data transformation techniques, with a bias for data quality, reliability, and integrity.
• Comfortable with ingestion patterns across APIs, S3, SFTP/FTP, email drops, event streams, and CDC.
• Well-versed in SDLC, release management, and incident management using Bitbucket/GitHub, JIRA, and change-management (CMR) workflows; basic software engineering fundamentals (version control, code review, testing, CI/CD) are expected.
• Good understanding of infrastructure component clusters, nodes, and compute and storage needed to run and tune data pipelines cost-effectively (bonus: exposure to Terraform/IaC and IAM).
• Familiarity with observability and monitoring tooling (Grafana, job-level metrics, data health checks) and basic security hygiene (Snyk/Akto-style vulnerability remediation).
• Excellent problem-solving and analytical skills, with the ability to anticipate roadblocks, diagnose production issues, and generate effective solutions.
• Outstanding stakeholder management and communication skills, comfortable working with a wide variety of business and technical stakeholders across regions.