A three-layer priority scheduler cuts on-device language-model latency from more than a second to under two-tenths of a second, keeping chat apps responsive even when the phone is busy with background work. Built for a Tensor G3 chip running a 3 billion-parameter model, it slices latency from 1,420 ms to 161 ms without killing background jobs.
Why on-device LLMs struggle
Running a large language model on a mobile processor is a resource squeeze. On the Tensor G3, the 3 B model already gobbles about 85 % of the neural-processing unit (NPU). When a low-priority task—such as an offline indexer—runs at the same time a user opens a chat window, the perceived response time jumps from roughly 140 ms to 1,400 ms, a ten-fold slowdown users notice instantly.
The problem isn’t just speed. Mobile devices must juggle UI smoothness, battery life, and multiple apps that all request compute. A naïve queue that processes jobs in order forces the UI thread to wait for background work, turning a conversational assistant into a sluggish experience.
How the three-layer scheduler works
The new scheduler inserts three coordinated components into the inference pipeline:
- Priority Queue – a min-heap that orders incoming jobs by a static importance level.
- Preemption Controller – when a higher-priority request arrives, it pauses lower-priority jobs instead of discarding them.
- Token Budget Governor – caps the number of tokens a job may generate based on the app’s lifecycle state.
Together they let a foreground chat request leap to the front of the line, while background jobs linger in a parked state, ready to resume when resources free up.
Priority tiers and preemption
Four tiers define what can be interrupted:
| Tier | Description |
|---|---|
| Foreground Chat | Critical UI interaction |
| Inline Suggestion | Autocomplete-style hints |
| Background Summary | Periodic content summarisation |
| Offline Indexing | Bulk data processing |
The scheduler never aborts a low-priority job. Instead, it snapshots the model’s key-value (KV) cache—a structure that holds intermediate attention results—and parks the job. When the high-priority request finishes, the controller restores the snapshot and lets the background task pick up where it left off. This “pause-and-resume” approach avoids the costly recomputation that would occur if the job were restarted from scratch.
Partial KV-cache eviction further trims waste. The static system prompt stays in cache, while only dynamic conversation turns are evicted. The result is a 40 %–60 % reduction in the cost of re-prefilling the model after a pause.
Managing token budgets without timers
Many implementations rely on timers to guess when a job should yield CPU or NPU time. Timers are blunt; they can either starve the UI or underutilise the chip. The scheduler swaps timers for Android’s ProcessLifecycleOwner, which emits lifecycle events that reliably indicate when the app is in the foreground or background.
- ON_RESUME – the app regains full compute budget, allowing pending foreground jobs to run unimpeded.
- ON_STOP – the app throttles background tasks to roughly 25 % of their normal token budget, preserving headroom for any sudden UI request.
By tying resource allocation to lifecycle events, the system reacts to real user behavior rather than arbitrary time slices.
Performance gains and trade-offs
Under a naïve first-come-first-served queue, a background task pushes foreground chat latency to about 1,420 ms. With the priority scheduler active, the same chat request completes in roughly 161 ms, a ten-fold improvement that restores a fluid user experience.
Resuming a paused job adds about a 22 % increase to its total execution time. Since background work is non-critical, the trade-off remains acceptable, especially when the UI stays snappy.