A Systematic Study of Behavioral Cloning for Scientific Data Annotation

Scientific data annotation, such as tracking animals in video or proofreading neural reconstructions, remains bottlenecked by the “last mile” problem: even with strong automation, verification and correction consume substantial human effort. Standard approaches train models to directly predict annotations, discarding the rich supervision in how experts navigate, click, verify, and correct. We introduce a framework for studying behavioral cloning on scientific annotation: 9 synthetic tasks paired with synthetic annotations that simulate realistic human strategies including exploration, mistake correction, and strategic decision-making. Our experiments reveal several findings. First, skills emerge hierarchically: models learn GUI mechanics before task-critical decisions, and commit fewer mistakes than the training data while retaining the ability to correct errors when they occur. Second, scaling models on multi-task behavioral cloning shows that larger models are more data efficient within our scale range. Third, multi-task pretraining enables efficient fine-tuning to new tasks, while training from scratch fails entirely. Fourth, linear probes reveal that models internally represent latent variables of the annotation process such as task phase and data position; interestingly, we find a shared mistake representation that generalizes across different annotation tasks. Overall, our framework establishes systematic benchmarks and identifies key bottlenecks, providing a foundation for scaling behavioral cloning to real-world scientific data annotation.

Mapping a single fly's brain by tracing neurons through electron microscope images has required 33 person-years of human work, even with the best automation. Across science (outlining cells, tracking animals on video, labeling satellite imagery, etc.), a similar “last mile” of human verification remains the bottleneck. Today's AI is trained on the final labels alone, ignoring how an expert arrived at them. A natural alternative is to train AI by imitation, a well-known idea called behavioral cloning, where a model learns by watching an expert work. But it has been hard to study when this works for science, because real behavioral data is expensive to collect and varies in ways that are hard to control. So, we built nine synthetic annotation tools paired with a virtual annotator that exhibits realistic human strategies including mistakes and corrections. Models trained this way adapt to new tasks with 15× less data, and the same recipe traces real neurons in human-brain images at up to 95% accuracy.