Built an open-source eval framework for testing medical LLM agents on synthetic FHIR records (meva-health-ai)

[u/heytanz100]

Hey all,

I built MEVA (meva-health-ai), an open-source eval framework for testing medical LLM agents against synthetic FHIR records.

• Why: General LLM evals don't measure structured clinical extraction or FHIR schema compliance.
• What it does: Runs automated evals against synthetic patient records (Patients, Encounters, Observations) for deterministic accuracy and schema validation.
• Looking for: Feedback on evaluation metrics and open-source contributors/testers.

Repo: https://github.com/Tanz2024/meva-health-ai

Need help

[u/Aggravating-Sea-2888]

For anyone who hasn't touched FHIR before the biggest headache is always the structure mismatch between what a model outputs and what actual EHR systems expect. Did you set up the eval to flag only missing required fields or does it get picky about the optional extensions too? asking cause I've seen stuff pass strict schema tests but still break downstream when the extended fields are parsed wrong

Also synthetic data is tricky cause it can be a little too clean compared to real messy records. might be cool if later on you can inject some common errors like duplicate observations or weird date formats and see how the agent handles it

I'm not a heavy coder but I work with clinical data pipelines and this sounds useful for catching things early before they hit production

[u/heytanz100 → u/Aggravating-Sea-2888]

Yes for now MEVA not doing full FHIR profile validation. It mainly checks if the model claim is actually supported by the FHIR evidence.

I should make that clearer in the post.

The messy data idea is good too. Could add a separate stress test later with duplicate observations missing fields and weird dates.

What kind of messy FHIR issues you usually see in real systems?

[u/No_Amphibian_7037 → u/Aggravating-Sea-2888]

everything can look great in a clean test environment and then fall apart once you throw real-world inconsistencies at it. I’d be really interested to see how the agent handles duplicate observations, missing fields, weird dates, etc.

[u/heytanz100 → u/No_Amphibian_7037]

Yeah good point. Right now MEVA uses clean synthetic data as a baseline. Adding things like duplicate records missing fields and weird dates would be a good next step to test how it handles real-world cases.
Thanks for the suggestion.

[u/Moms_Cedar_Closet]

Do you include accessibility metadata also? I'm bilingual with a sign language and that modality is not encoded in my FHIR export (i'm aware there may be information limitations on apple health). 

[u/Hot-Praline7204]

Healthcare AI evaluations is my world. Curious to hear about your background and why you decided to develop this. Also, tangentially, what was your impression of Synthea? I spent days and days enhancing a fork of Synthea to try to make rich, realistic synthetic patients with LLM-generated clinical notes, but ultimately I found the longitudinal data to be lacking.

[u/heytanz100 → u/Hot-Praline7204]

Yeah thanks for sharing. My background is in AI and software engineering and I started MEVA because I’m interested in making medical AI agents more reliable.
For Synthea I think it’s a great starting point for synthetic FHIR data but creating realistic long term patient data with real-world messiness is still a big challenge. I’m also interested in exploring this area more in the future.

[u/crazy-dev3]

One useful evaluation dimension might be not only whether a claim is supported, but whether the agent points to the correct resource and field—and abstains when the evidence is incomplete or contradictory. Claim-level provenance, contradiction handling, temporal consistency, and calibrated refusal could complement the schema checks. Two agents can produce the same correct-looking answer while only one reached it from the right evidence, which matters a lot in a clinical context.

[u/heytanz100 → u/crazy-dev3]

Yeah I agree. Checking where the claim comes from and how the agent handles missing or conflicting evidence would make MEVA stronger. This is something I want to explore more and I would appreciate any ideas or contributions.

[u/TechnologyMatch]

this is useful, especially if the eval separates “found the right fact” from “returned valid FHIR.” a model can sound clinically convincing while still failing the handoff point that matters

I’d also test missing, conflicting, and time-shifted records, not just clean extraction. like a game tutorial versus a real match: the hard part is seeing whether the agent holds up when the map gets messy

[u/krasynEMR]

One test I would add: a negative-control pair. Give the agent two synthetic patients that differ by one relevant observation, then ask the same question. The answer should change when the observation appears and stop short when it is absent. Otherwise you can get perfectly valid FHIR from a system that never looked at the evidence. That failure deserves its own score.