FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation

LLM-conditioned segmentation has recently advanced rapidly by coupling large language models with iterative mask generation frameworks. However, we identify a persistent failure mode in current propose-then-select pipelines. Although high-quality mask candidates are often generated, the final prediction may fail to match the given linguistic condition. This failure arises because language semantics are typically used as static prompts or post-hoc matching signals, rather than participating in the iterative mask generation process. Through systematic analysis, we show that many errors stem from semantic misalignment rather than poor mask quality. To address this issue, we propose FlowSeg, which introduces dynamic semantic guidance via a bidirectional semantic flow between intermediate decoding states and LLM-derived condition embeddings throughout the generation process. Language conditions actively guide mask refinement at each stage, while condition embeddings are progressively updated by emerging visual evidence. This design yields semantically grounded mask representations and visually aligned language conditions, enabling more reliable matching. We further incorporate a lightweight boundary-aware refinement to selectively enhance uncertain regions without perturbing confident interiors. Extensive experiments on referring expression segmentation and reasoning segmentation tasks demonstrate that FlowSeg consistently improves language–mask alignment and achieves state-of-the-art performance. Project page: https://zkzhang98.github.io/FlowSeg_page

Many AI tools can mark the exact part of an image that a person describes, such as “the small red car next to the bus.” However, these tools can still make a surprising mistake: they may find several reasonable object regions, but choose the one that does not match the person’s words. This problem becomes more common when the description is detailed, ambiguous, or requires reasoning. We propose FlowSeg, a new way for the language-understanding and image-understanding parts of the system to work together throughout the process. Instead of waiting until the end to compare the words with possible image regions, FlowSeg lets the words guide the search while the image also helps refine what the words refer to. We also add a small cleanup step to improve uncertain object boundaries without changing parts that are already clear. Across standard tests for language-guided image segmentation, FlowSeg more reliably selects the region described by the user and produces cleaner results. This can make image editing tools, visual assistants, and other systems that need precise language-based image understanding more dependable.