llama.cpp PR #23198 eliminates redundant logit copies, speeds prompt processing

A performance optimization landed in llama.cpp this week that trims prompt processing time by skipping unnecessary logit copies during the decode phase of multi-token prefill. Pull request #23198, merged by contributor am17an, targets the prompt-ingestion path when the engine is running in multi-token processing mode — a scenario where the model processes dozens or hundreds of tokens before generating the first output token. The change eliminates a memcpy that was firing on every prompt token, a small overhead that accumulated quickly on long context windows.

The improvement is most visible on hardware with limited memory bandwidth or when prompt lengths stretch into the thousands of tokens. Users running large context workloads — RAG pipelines, long-document summarization, or multi-turn chat with extensive history — should see faster time-to-first-token without touching their command-line flags. The optimization is transparent; no new switches or config knobs required. The PR landed in the main branch this week, so anyone pulling the latest llama.cpp source or waiting for the next official release will pick it up automatically.

The change is narrow in scope, touching only the prompt-decode code path and leaving generation-phase logic untouched. It's a reminder that even mature inference engines still carry low-hanging fruit in their hot paths. The next round of llama.cpp performance work is likely to focus on batched prefill and speculative decoding improvements, both of which have active development branches. For now, the advice is simple: update your build and enjoy the faster prompt churn.