Scheduling & Batching

JustAPI’s inference scheduler implements continuous batching, prefix caching, and disaggregated prefill/decode for maximum LLM throughput.

Unlike static batching (wait for all sequences to finish), continuous batching interleaves prefill and decode steps:

Time →  |--- Batch 1 (prefill) ---|---- Batch 1 (decode) ----|
         |--- Batch 2 (prefill) ---|---- Batch 2 (decode) ----|
                                    |--- Batch 3 (prefill) ---|

This eliminates the “wait for straggler” problem and maximizes GPU utilization.

justapi serve --model my-model.gguf \
  --max-seqs 64 \             # Max concurrent sequences
  --pool-blocks 8192 \        # KV-cache pool size
  --scheduled                 # Use continuous batching scheduler

The disaggregated scheduler (PdScheduler) separates prefill and decode onto dedicated resource pools:

• Prefill pool: Handles prompt processing (compute-bound)
• Decode pool: Handles token generation (memory-bound)

┌──────────────┐    ┌──────────────┐
│ Prefill Pool │    │ Decode Pool  │
│ (GPU batch)  │───▶│ (GPU batch)  │
└──────────────┘    └──────────────┘

This prevents decode starvation during heavy prefill loads.

The radix prefix cache (RadixPrefixCache) reuses KV-cache prefixes across requests:

Request 1: "What is the capital of France?"
Request 2: "What is the capital of Germany?"
            └─────────────────┘
             Shared prefix: "What is the capital of"

The shared prefix is computed once and reused, saving prefill computation.

The /metrics endpoint exposes prefix cache statistics:

A smaller draft model proposes tokens, and the target model verifies them in parallel:

Draft:        [a] [b] [c] [d]
Target:       ✓   ✓   ✗   —
Accepted:     [a] [b] [c] → then resample

The tree-based decoder (spec_decode_tree) constructs a draft tree and verifies multiple branches simultaneously:

justapi serve --model my-model.gguf \
  --gamma 4 \                 # Speculation window size
  --branch 2                  # Tree branching factor

justapi serve --model llama-7b.gguf \
  --gpu \
  --scheduled \
  --max-seqs 64 \
  --pool-blocks 4096 \
  --gamma 4 \
  --branch 2

• Inference Overview — Architecture and components
• LLM Serving API — API endpoints
• GPU & CUDA Setup — Hardware configuration