Failure-Aware Query Refinement for Reliable Open-Vocabulary Home-Robot Perception

Home robots in open-ended indoor environments must localize user- or task-specified objects using imperfect perception tools. Frozen open-vocabulary detectors are attractive, but class-name queries can miss the target, produce weak detections, or fire on distractors. We study failure-aware query adaptation for a frozen detector, modifying only the text query at test time. We propose Failure-Aware Multi-Agent Query Refinement, which diagnoses detector responses into GOOD, MISS, WEAK, and NOISY states, routes failures to state-specific query agents, and re-evaluates candidates by re-running the detector. On 9,740 ODSR-IHS image--target pairs, our method improves YOLOE from 0.6513 to 0.7667 AP50 and from 0.6129 to 0.7504 Top-1 accuracy, with the largest gains in WEAK and NOISY cases.