Beyond Agreeable Chatbots: Context-Aware Safety Oversight for Trustworthy Patient-Facing LLMs

Large language models (LLMs) are increasingly being integrated into patient-facing healthcare systems, yet their safe deployment in real-world settings remains an open challenge. Although existing alignment methods can reduce overtly harmful outputs, they often fail in realistic medical interactions where patient queries are incomplete, ambiguous, and context dependent. In such settings, models frequently default to agreeable responses rather than actively identifying missing clinical context or challenging unsafe assumptions. As a result, LLMs may generate responses that appear plausible and reassuring while remaining clinically unsafe or factually unreliable. In this work, we investigate whether inference-time safety oversight can improve the trustworthiness of patient-facing medical LLMs beyond conventional alignment alone. We introduce CareGuardAI, a context-aware multi-agent framework designed to study and mitigate two critical deployment risks: clinical safety risk and hallucination risk. The framework combines a controller agent for risk-aware triage and contextual screening with dual evaluator agents that independently assess Clinical Safety Risk Assessment (SRA) and Hallucination Risk Assessment (HRA). Responses exceeding predefined deployment thresholds are iteratively refined or blocked prior to release. We evaluate the framework on PatientSafeBench, MedSafetyBench, and MedHallu, covering adversarial medical safety scenarios and hallucination-focused evaluation. Across benchmarks, inference-time oversight substantially improves deployable response rates while reducing unsafe and hallucinated outputs compared to strong baseline models, including GPT-4o-mini. Through ablation studies and qualitative analysis, we further examine the contribution of contextual screening, evaluator-based oversight, and iterative refinement in reducing deployment risk. Our findings suggest that trustworthy deployment of patient-facing medical LLMs may require more than static alignment objectives alone. Instead, reliable real-world deployment may depend on context-aware inference-time monitoring systems that actively assess uncertainty, identify missing context, and constrain model behavior during interaction.