Google has introduced a new accelerator for LiteRT, called Qualcomm AI Engine Direct (QNN), to enhance on-device AI performance on Qualcomm-powered Android devices equipped with Snapdragon 8 SoCs. The accelerator delivers significant gains, offering up to a 100x speedup over CPU execution and 10x over GPU.
While GPU hardware is widely available on modern Android devices, relying on them exclusively for AI tasks can introduce performance bottlenecks, according to Google software engineers Lu Wang, Wiyi Wanf and Andrew Wang. For example, they note that “running a compute-intensive, text-to-image generation model on-device, while simultaneously processing the live camera feed with an ML-based segmentation” can overwhelm even high-end mobile GPUs. The result may be a jittery user experience and dropped frames.
However, many mobile devices now include a neural processing units (NPUs) which are custom-designed AI accelerators that can significantly speed up AI workloads compared to a GPU, while consuming less power.
QNN was developed by Google in close collaboration with Qualcomm as a replacement for the previous TFLite QNN delegate. It provides developers with a unified and simplified workflow by integrating a wide range of SoC compilers and runtimes and exposing them through a streamlined API. It supports 90 LiteRT operations with the goal of enabling full model delegation, which is a key factor for achieving optimal performance. QNN also includes specialized kernels and optimizations that further boost the performance of LLMs such as Gemma and FastLVM.
Google benchmarked QNN across 72 ML models, with 64 of them successfully achieving full NPU delegation. The results showed performance gains of up to 100x compared to CPU execution and 10x compared to GPU.
On Qualcomm’s latest flagship SoC, the Snapdragon 8 Elite Gen 5, the performance benefit is substantial: over 56 models run in under 5ms with the NPU, while only 13 models achieve that on the CPU. This unlocks a host of live AI experiences that were previously unreachable.
Google engineers also developen a concept app that leverages an optimized versions of Apple’s FastVLM-0.5B vision-encoding model. The app can interpret the camera’s live scene almost instantly. On the Snapdragon 8 Elite Gen 5 NPU, it achieves an impressive time-to-first-token (TTFT) of just 0.12 seconds on 1024×1024 images, over 11,000 tokens/sec for prefill, and more than 100 tokens/sec for decoding. Apple’s model was optimized with int8 weight quantization and int16 activation quantization. According to Google’s enginner, this is the key to unlocking the NPU’s most powerful, high-speed int16 kernels.
QNN only supports a limited subset of Android hardware, primarily devices powered by the Snapdragon 8 and Snapdragon 8+ SoCs. To get started, visit the NPU acceleration guide and download LiteRT from GitHub.
