What Do Agents Learn from Trajectory-SFT: Semantics or Interfaces?

Large language models are increasingly evaluated as interactive agents, yet standard agent benchmarks conflate two qualitatively distinct sources of success: semantic tool-use and interface-specific interaction pattern memorization. Because both mechanisms can yield identical task success on the original interface, benchmark scores alone are not identifiable evidence of environment-invariant capability. We propose PIPE, a protocol-level evaluation augmentation for diagnosing interface reliance by minimally rewriting environment interfaces while preserving task semantics and execution behavior. Across 16 environments from AgentBench and AgentGym and a range of open-source and API-based agents, PIPE reveals that task-specific trajectory-SFT can amplify reliance on training-time interface forms: in several environments, agents with trajectory-SFT degrade sharply under minimal interface rewrites, whereas other agents are often more stable. We further introduce Interface Reliance (IR), a counterbalanced alias-based metric that quantifies preference for training-time interfaces, and show that interface shortcutting exhibits environment-dependent, non-monotonic training dynamics that remain invisible under standard evaluation. Our code is available at https://github.com/ChengZe2005/What-Do-Agents-Learn-from-Trajectory-SFT-Semantics-or-Interfaces-.

Large language models are increasingly used as agents that interact with tools and environments to complete tasks. However, a high score on an agent benchmark does not always mean that a model truly understands how to use the tools. In some cases, the model may partly rely on the exact tool names, commands, or interaction formats it saw during training. We proposes a simple way to test for this issue by making small changes to the interface while keeping the task itself the same. Across many agent environments and models, we find that some agents trained on task trajectories perform much worse after these small interface changes, while other agents are more stable. These results suggest that agent evaluations should check not only whether a model succeeds on the original benchmark, but also whether its success remains robust when superficial interface details change.