The candidate will join a European health‑technology team as a Senior AI Engineer responsible for ensuring the evaluation and quality assurance of a newly developed AI‑driven module used by more than 2.5 million users across Europe. The module is certified as a Medical Device Class IIa and is built on a healthcare‑specific large language model (LLM) platform.
Responsibilities
- Develop and maintain production code for RAG pipelines, LLM orchestration, prompt engineering and agent workflows on the healthcare LLM platform.
- Design, implement and own the evaluation framework for model outputs, including accuracy metrics, hallucination detection, clinical safety checks and regression testing across releases.
- Create and maintain ground truth datasets for evaluation in collaboration with clinicians and content partners, ensuring data quality and traceability.
- Implement production monitoring and alerting for model outputs: drift detection, quality regressions and anomaly reporting.
- Produce evaluation evidence and traceability required for regulatory and quality processes, including planned AI and Quality Control certifications alongside existing ISO 27001 and ISO 27701 controls.
- Design robust API contracts between Python‑based AI services and the .NET / Angular platform deployed on Azure; collaborate on integration and handoffs.
- Pair with the architecture‑focused Senior AI Engineer on day‑to‑day implementation, share knowledge with the in‑house engineering team and document evaluation practices.
- Engage with clinicians and non‑technical stakeholders to translate clinical safety requirements into measurable evaluation criteria and test plans.
- Contribute to software engineering best practices: code review, automated testing, version control and CI/CD for AI services.
Qualifications
- 6 years of hands‑on software engineering experience, with at least 3 years focused on production AI and LLM applications.
- Extensive experience with Python as the primary language for building AI services, RAG pipelines, agents and evaluation tooling.
- Proven track record integrating LLM APIs and SDKs in production environments.
- Hands‑on expertise in prompt engineering, RAG architectures and LLM orchestration frameworks.
- Demonstrable experience designing and operating LLM evaluation frameworks in production: accuracy metrics, hallucination detection, regression testing, ground truth dataset design and output monitoring.
- Nice to have: familiarity working with .NET and C# at the integration level; ability to design clear Python‑to‑.NET API contracts and collaborate on a .NET codebase when required.
- Strong software engineering fundamentals: API design, automated testing, version control and CI/CD pipelines.
- Effective communication skills, able to engage clinicians and non‑technical stakeholders on practical implications of model evaluation.
- Fluency in English (working language of the team).
- Nice to have: experience in healthtech, medtech or regulated industries; background in MLOps or production AI monitoring; familiarity with Medical Device Regulation (MDR), the EU AI Act or quality management for AI systems; experience with clinical safety frameworks or human‑in‑the‑loop systems.
Benefits
- Opportunity to shape the AI architecture of a Medical Device Class IIa module used by millions of European users.
- Work alongside a strong in‑house engineering team and a dedicated second Senior AI Engineer focused on quality and evaluation.
- Significant technical ownership and influence over integration, orchestration and certification activities (MDR, AI Act alignment).
- Hands‑on exposure to state‑of‑the‑art LLM platforms, agent frameworks and Azure‑based production environments.
- Remote work, long-term engagement.
- Unique TEAL culture, relationship- and respect-driven community, non-corporate atmosphere.
- Agile approach and no bureaucracy.
- Outstanding integration trips to various places in Europe for all employees.
- Activities to support your well-being and health.
- Luxmed Gold Extended medical care and Multisport Plus benefit.
The organisation welcomes applications from candidates with diverse backgrounds and perspectives. Please apply via the application form.