Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment

The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of the component-wise quantization remains a bottleneck for optimal deployment. This paper presents a systematic evaluation framework for empirically validating five hypotheses across six quantization configurations on the Jetson Orin NX and AGX. By separating the vision encoder, projector, and large language model backbone yields the following results: (1) MoE architectures effectively mitigate INT4 quantization noise compared to dense backbones; (2) The SigLIP architecture exhibits significant incompatibilities between hardware and software within the BitsAndBytes framework, resulting in a disproportionate latency overhead; (3) Although INT4 quantization of LLMs greatly reduces VRAM consumption, it also causes slower token generation due to dequantization overhead; (4) The impact of multimodal quantization is generally additive, whereas in modality alignment it leads to performance degradation, which depends on the model; (5) The intelligence-per-joule profile varies significantly across platforms owing to memory bandwidth constraints.