Reinforcement Learning for Tool-Calling Agents in Fast Healthcare Interoperability Resources (FHIR)

Fast Healthcare Interoperability Resources (FHIR) is the dominant standard for interoperable exchange of healthcare data. In FHIR, electronic health records form a directed graph of resources. Answering clinically meaningful questions over FHIR requires agents to perform multi-step reasoning, filtering, and aggregation across multiple resource types. Prior work shows that even tool-augmented LLM agents (retrieval, code execution, multi-turn planning) often select the wrong resources or violate traversal constraints. We study this problem in the context of FHIR-AgentBench, a benchmark for realistic question answering over real-world hospital data, and frame reasoning on FHIR as a sequential decision-making problem over a queryable structured graph. We implement a multi-turn CodeAct agent and post-train it with reinforcement learning using a custom harness and tools. A LLM Judge provides execution-grounded rewards. Compared to prompt-based, closed-model baselines, RL post-training improves performance while enforcing data-integrity constraints. Empirically, our approach improves answer correctness from 50% (o4-mini) to 77% on FHIR-AgentBench using a smaller and cheaper Qwen3-8B model. We present an end-to-end post-training pipeline (environment building, harness construction, model training and custom evaluation) that reliably improves multi-turn reasoning over structured clinical graphs.

The problem: Hospital records (in a format called FHIR) are a messy web of connected pieces, and AI assistants are bad at answering questions about them. Even top commercial models get only ~50% right on the FHIR-AgentBench benchmark. But privacy rules mean hospitals want small AI they can run in-house, which usually performs worse. What we did: Took a small, free AI model and trained it using a "CodeAct" approach, where the AI writes and runs little bits of code to pull and analyze records step by step, letting it practice and rewarding correct answers. The result: The small model hit 77% accuracy, beating much larger, more costly commercial models. It's best used as a tool that supports clinicians, who verify its answers.