QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL

Offline goal-conditioned RL (GCRL) learns goal-reaching policies from static datasets, but real-world environments are often partially observable, so the collected trajectories are only partly consistent with the Markov assumption while other segments remain history-dependent. History-aware sequence models such as Decision Transformer (DT) are a natural fit for long-term dependency modeling, yet pure attention is inefficient and brittle when handling local Markovian structure and long-range context simultaneously. Although recent hybrid architectures (e.g., LSDT) introduce local extractors, their fixed-window extraction cannot adapt the effective memory to varying dependency lengths, often truncating long-range context instead of compressing it. Moreover, under sparse rewards, return-to-go (RTG) becomes non-discriminative across sub-trajectories, offering little guidance for stitching goal-reaching behaviors from diverse demonstrations. To address these limitations, we propose QHyer (Q-conditioned Hybrid Attention-Mamba Transformer), which replaces RTG with a Normalizing Flows (NFs) parameterized goal-reaching Q-estimator used directly as conditioning tokens, and a gated Hybrid Attention-Mamba backbone whose selective state-space dynamics enable content-adaptive history compression while attention captures global goal-directed dependencies. Extensive experiments on OGBench and D4RL demonstrate that QHyer achieves state-of-the-art performance on both non-Markovian and Markovian datasets, validating its effectiveness for diverse scenarios.

Imagine teaching a robot to perform complex tasks using only recordings of past attempts, with no opportunity to practice on its own. Current methods for this problem face two obstacles. The guidance signal they use indicates only whether an entire recorded attempt eventually succeeded, which is too coarse to tell which individual steps were valuable. Furthermore, these models cannot adapt how far back in time they look, because they rely on a fixed memory window. We propose QHyer, which addresses both issues. We estimate how promising each individual decision is for reaching the goal, using a probabilistic model that produces exact, mathematically grounded scores. We also design a neural network that combines long-range goal-directed planning with a mechanism that adaptively decides how much past context to retain. On standard robotics benchmarks, QHyer outperforms existing approaches, bringing us closer to agents that can learn complex skills directly from logged real-world experience.