RTInfer: Real-Time Inference of Multiple DNNs on Edge GPUs

While edge GPUs are increasingly used for latency-critical DNN tasks, limited resources often fail to meet strict real-time (RT) requirements under concurrent workloads. Existing preemption and early-exit mechanisms often underutilize GPU resources through single-task queuing and sacrifice excessive accuracy during task bursts. To address this, we propose RTInfer, a novel system that enables concurrent RT task execution while balancing throughput and accuracy. RTInfer integrates an accuracy-calibrated lightweight variant co-optimization to generate efficient models, a memory-layout-aware scheduler to mitigate fragmentation during preemption, and an on-demand loading strategy to minimize host-to-GPU latency. Extensive evaluations demonstrate that RTInfer outperforms state-of-the-art methods by reducing average deadline miss rate (DMR) from 32.8\% to 0\% and improving accuracy by up to 56.5\%.

Modern cars, drones, and robots often need to run several AI programs at once, such as finding obstacles, understanding scenes, and responding to commands. Because these devices are small, they have limited computing power and memory. When many tasks arrive together, some may finish too late or produce less reliable results. RTInfer is a system that helps these devices manage such busy moments. It prepares smaller but still accurate versions of AI programs, decides which jobs can run together, arranges memory to avoid wasted space, and loads only the pieces that are needed. In our tests on a small computer like those used in robots and vehicles, RTInfer completed urgent jobs on time more often and kept answers more accurate than existing methods. This can help make AI in vehicles, drones, and robots faster, safer, and more practical without depending on large remote servers.