Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition

Handwritten Text Recognition (HTR) is computationally imbalanced in two ways: most image pixels are background, and many width-axis sequence positions are blank-dominated. This creates a mismatch for Spiking Neural Networks (SNNs): handwriting is observed as a static image, whereas spiking computation unfolds over timesteps. We propose Spike-HTR, a hybrid spiking recognizer that controls both the number of spiking steps and the number of width positions processed by the deep sequence mixer. To make a static image suitable for short-horizon spiking inference, InkCoder converts it into a coarse-to-fine input stream, where early steps cover broad stroke regions and later steps emphasize sharper stroke details. To reduce sequence computation, a CTC-guided length reducer keeps likely character or uncertain positions and compresses long blank-dominated stretches before deep mixing. With $T{=}2$, Spike-HTR trains only on target data, decodes without language models or lexicons, and reaches validation/test CERs of 3.5/5.4, 2.3/2.5, and 4.2/3.9 on IAM, LAM, and READ2016. Codes are available at https://github.com/QomolangmaH/SpikeHTR.

Handwritten text recognition often wastes computation on empty background and blank parts of a line. Spike-HTR uses a brain-inspired spiking neural network, where information is processed through short pulses over a few steps. Instead of repeatedly seeing the same image, it gradually focuses from broad handwriting strokes to finer details, while also compressing long empty stretches. This allows accurate handwriting recognition with less unnecessary processing and without external dictionaries, language models, synthetic data, or large-scale pretraining.