Backend Developer Lead

Role overview
We are seeking a Lead Backend Engineer with strong production-level Python experience and deep expertise in large-scale search platforms, preferably SolrCloud/Lucene, to help integrate and scale two large healthcare data platforms.The role will focus on enabling enriched emergency medical records from one platform to be consumed effectively by another. The platform currently manages approximately 500 TB of data and processes around 50 GB of new data daily, with the initial implementation focused on one data source and plans to onboard approximately ten additional sources.You will work closely with a small, senior engineering team and take ownership of complex technical challenges across data ingestion, enrichment, modeling, indexing, APIs and search architecture.
Key responsibilities:
1. Data architecture and lifecycle

• Own the lifecycle of data attributes across the technology stack, from definition and modeling through ingestion, indexing, APIs and presentation.
• Define scalable data models and attribute structures to support evolving healthcare data requirements.
• Develop and maintain data ingestion and enrichment workflows using Python.
• Work closely with Java/Spring services to expose and serve enriched data through APIs.
• Collaborate with front-end teams to ensure data is effectively presented and consumed.

2. Metadata-driven framework

• Design and implement a metadata-driven framework for managing data attributes across the technology stack.
• Create a single attribute definition that can generate the required configurations and artifacts currently maintained across multiple components.
• Establish reusable patterns that simplify the introduction and management of new attributes.
• Improve consistency, maintainability and operational efficiency across the data platform.

3. Data source onboarding

• Design repeatable processes and frameworks for onboarding new data sources.
• Reduce data-source onboarding time from weeks or days to hours wherever practical through automation and metadata-driven configuration.
• Develop scalable ingestion, transformation, enrichment and indexing workflows.
• Support the initial implementation for one data source and subsequent onboarding of approximately ten additional sources.

4. Solr architecture and search performance

• Redesign the existing Solr architecture to support continued growth beyond the current fixed three-shard layout.
• Define and optimize SolrCloud sharding, routing, indexing and query strategies.
• Design solutions for high-volume indexing and low-latency search at scale.
• Diagnose and resolve search performance, indexing and scalability challenges.
• Establish best practices for Solr schema design, collections, replicas, shards and query optimization.
• Ensure the search architecture can scale reliably as data volumes and sources increase.

5. Backend engineering

• Develop scalable, maintainable and production-ready backend components using Python.
• Contribute confidently to Java/Spring-based services and APIs.
• Design robust data-processing pipelines for ingestion, enrichment and serving.
• Write clean, testable and maintainable code with appropriate documentation and engineering practices.
• Take ownership of technical solutions from architecture and design through implementation, testing and production deployment.

6. Technical leadership and collaboration

• Work closely with a small, experienced engineering team in an autonomous and fast-moving environment.
• Take ownership of complex technical problems and drive them through to resolution.
• Participate in architecture and design discussions and contribute to technical decision-making.
• Collaborate with engineering, product and other stakeholders to translate requirements into scalable technical solutions.
• Work through short development and feedback cycles to deliver production-ready improvements quickly.

Required technical skills and competencies:

• Strong production-level experience with Python.
• Deep hands-on experience with Apache Solr/Lucene, preferably SolrCloud 9 or equivalent large-scale deployments.
• Strong understanding of Solr architecture, including sharding, routing, indexing, replication, schema design and query optimization.
• Strong Elasticsearch experience may be considered for candidates who are willing and able to transition to Solr.
• Working knowledge of Java and the ability to contribute confidently to Spring-based backend services.
• Strong SQL and data-modeling skills.
• Experience designing scalable data-ingestion and data-processing pipelines.
• Experience with data enrichment, indexing and search-serving workflows.
• Strong understanding of distributed systems and large-scale data platforms.
• Ability to troubleshoot and optimize high-volume indexing and search workloads.
• Demonstrated ability to take ownership of complex engineering problems from design through production deployment.

Must have skills:

• Strong Python development experience.
• Hands-on experience with Solr/Lucene at scale.
• Strong understanding of SolrCloud, sharding, routing and indexing strategies.
• Experience with Elasticsearch can be considered with willingness to transition to Solr.
• Good working knowledge of Java and Spring.
• Strong SQL and data-modeling skills.
• Experience building scalable data ingestion and processing workflows.
• Strong problem-solving and system-design capabilities.
• Experience working with large-scale data platforms and production environments.

Good to have skills:

• Experience working with healthcare data.
• Knowledge of NEMSIS data and emergency medical records.
• Experience with Databricks.
• Front-end development experience.
• Experience building metadata-driven platforms or configuration frameworks.
• Experience working with very large-scale datasets and distributed search infrastructure.

Desired attributes:

• Strong ownership mindset with the ability to independently drive technical initiatives.
• Excellent system-design and architectural thinking.
• Strong analytical and problem-solving skills.
• Comfortable working with ambiguity and evolving requirements.
• Ability to balance long-term architecture with short-term delivery requirements.
• Strong communication and collaboration skills.
• Comfortable working in a small, senior and highly autonomous engineering team.
• Passion for building scalable, maintainable and production-ready systems.
• Ability to learn and adapt quickly to new technologies and domain requirements.

Interview Process2-3 rounds of discussion.