Prompt Compression: Techniques, Tradeoffs and Tools
Written By:
Matt Tanner
Published on

Alt text (deployment only): Manual prompt compression reduces a support prompt from 421 to 208 tokens, approximately 51%, while preserving six rules.
Ask LLMLingua-2 to cut a short refund policy in half, and it keeps "share internal discount codes" while dropping the "Never" in front of it. We ran it on a CPU with the open-source llmlingua package. That one result sums up prompt compression: the prompt gets shorter, the input bill can get smaller, and occasionally the one word that mattered disappears.
That tradeoff is why prompt compression sits on every LLM cost optimization list with an asterisk. This guide covers manual and automated techniques, compaction for long sessions, and the quality you can expect to lose. It ends with how to test a compressed prompt against the original prompt before it ships.
What is prompt compression?
Prompt compression is the practice of shortening the text you send a language model while keeping the information the task needs. It cuts prompt length, and with it token count, by removing redundant wording, pruning low-information tokens, or summarizing older context. The goal is the same answer from fewer input tokens, at lower cost and latency.
Research splits the field into two families. Hard prompt methods remove low-information tokens or paraphrase, so the result is still text that black-box API models accept. Soft prompt methods compress the prompt into a few special tokens, learned vectors a model has to be built to accept, so they won't work with a hosted chat API. This guide covers hard compression, the kind you can apply to a commercial model today.
Hard compression is lossy: once a token is deleted, the model can't read it.
Why prompts get long
Few prompts start long. They grow one fix at a time, and the growth usually comes from four places:
Redundant instructions. Two people add the same rule in different words, months apart.
Verbose examples. A few-shot example written as a full, polite customer exchange when one line would show the pattern.
Stale conversation history. With a stateless chat API, history is resent on every turn, so turn 40 carries the first 39, tool output included.
Retrieved context. A RAG pipeline passes its top-k passages whether or not they're relevant, and raising k to fix one recall miss adds tokens to every call.
Deciding what belongs in each call is the job of context engineering. Compression comes after that decision: it shrinks what you've already chosen to send.
Long prompts cost you twice. Every input token is billed, and uncached input has to be processed before the model writes anything. Prefill processes the input prompt and decode generates the output, so a bloated prompt shows up in time to first token as well as on the invoice. Our guide to LLM latency optimization lists long uncached input as one cause of a slow first response. Long prompts also eat into the context window.
Manual compression techniques
Start by hand. Manual compression costs nothing at request time, the change is reviewable in a pull request, and it doesn't put another model in your request path.
Here's a support system prompt for a fictional cloud storage company. The original prompt is 421 tokens. After a manual pass it's 208 tokens, counted with tiktoken's o200k_base encoding, and every rule survived.

A manual rewrite cuts this example from 421 to 208 tokens while preserving its six rules.
Alt text (deployment only): Selected before-and-after edits remove repeated politeness, duplicate instructions, and an expired promotion. A six-rule checklist remains, while the example prompt shrinks from 421 to 208 tokens; fewer tokens alone do not prove quality held.
Four edits did most of the work:
Remove duplicates. The original told the model not to invent information twice, in two different paragraphs. One rule, stated once, is enough.
Prefer structure over prose. Five rules buried in paragraphs became a five-item list. Lists drop filler like "Please remember that" and make each rule easy to check in review.
Shorten examples. The example answer kept the facts the model should copy (two weeks, the 30-day window, eligible) and lost the pleasantries.
Drop stale instructions. A December 2024 holiday promotion was still in the prompt. Stale rules waste tokens and invite the model to mention offers that no longer exist.
What makes manual compression safe is a checklist. Write down every rule before you edit, then confirm each one is still in the compressed prompt. Ours had six: answer only from the documents, escalate when the answer isn't there, refunds within 30 days, no refunds after 30 days, never share discount codes, and plain text under 120 words.
Manual compression runs out of road when the long part of the prompt changes on every request, like retrieved passages or conversation history. You can't hand-edit those, so you reduce prompt length with a model instead.
Automated and token-level compression
Automated compressors run a smaller model over the prompt and pass the shorter text to your target model. The families differ in how they score what to drop:
Selective context. Selective Context scores tokens, phrases or sentences by self-information computed with a base causal language model and drops the least informative units.
LLMLingua. LLMLingua uses a small model such as GPT-2 or LLaMA to compute perplexity, on the idea that low-perplexity tokens contribute least and can be removed with minor impact. A budget controller gives the instruction, demonstrations and question different compression ratios within the overall token budget. The paper reports up to 20x compression with little performance loss on GSM8K, BBH, ShareGPT and Arxiv-March23.
LLMLingua-2. LLMLingua-2 treats compression as token classification. A Transformer encoder, trained on labels distilled from an LLM, decides keep or drop for each token using context on both sides. Its authors report it runs 3x to 6x faster than earlier compression methods and cuts end-to-end latency by 1.6x to 2.9x at compression ratios of 2x to 5x.
LongLLMLingua. The question-aware member of the family compresses documents relative to the user's question and reorders them to reduce position bias, which makes it a natural fit for RAG.
All four are extractive compression at the token or sentence level: they keep a subset of the original text, which is why their output reads like a telegram. Abstractive compression writes new text instead. RECOMP trains both kinds for retrieved documents: an extractive compressor that selects useful sentences and an abstractive one that writes a summary across documents.
Here's LLMLingua-2 on the refund paragraph from the support prompt above. Setting rate=0.5 asks for a compressed prompt about half the original size, because the library defines rate as compressed size over original size.
With llmlingua 0.2.2 on CPU, two runs gave identical output:
The 30-day window survived. "Only" didn't, and the discount-code rule flipped from a prohibition to an instruction. Adding force_tokens, which keeps the listed tokens in the output, put "Never" back for one extra token. That fixed this rule, but force_tokens only protects the words you thought to list, so it's a patch rather than a policy.
Summarization and compaction for long sessions
Token pruning works on a passage, but a long chat or agent session has a different problem: turns pile up. The usual answer is a rolling summary that replaces older turns and keeps recent ones verbatim, which APIs and agent frameworks call compaction. Two implementations show the range:
Anthropic compaction replaces the older turns of a conversation with a summary Claude writes on the server, either on demand or when input tokens reach a threshold you set. Both modes are in beta. The compaction docs suggest writing your own summarization prompt when the default summary drops something a later turn needs, which tells you exactly what to test for.
LiteLLM's
compress()is in beta and takes a retrieval route. It replaces low-relevance messages with short stubs, keeps the originals in a cache, and gives the model alitellm_content_retrievetool to request one back. System messages and the last user and assistant messages are always preserved. It only acts abovecompression_trigger, which defaults to 200,000 tokens, with a target of 70% of the trigger, so set both explicitly, or a 20,000-token test conversation will pass through untouched.
Summaries fail in a predictable way. The account ID a user typed on turn 3, or the refund amount an agent promised on turn 12, gets folded into "the user asked about a refund." If later turns depend on identifiers, numbers, or commitments, keep them in a structured field outside the conversation history, or tell the summarizer to carry them forward.
Related approaches that are not compression
Two techniques show up in nearly every prompt compression roundup, and both cut cost or noise without rewriting the text that ends up in the prompt.
Prompt caching keeps the prompt identical and reuses the provider's processing of a matching prefix. OpenAI turns it on by default and needs the entire rendered prefix to match. Its minimum is 1,024 tokens for GPT-5.6 and later, and cached reads bill at 0.1x the normal input rate on most of those models, per its prompt caching guide.
Anthropic makes caching opt-in with cache_control, with minimums from 512 to 4,096 tokens depending on the model, and processes shorter prompts without caching or an error.
Caching and compression interact, and not always in your favor:
Minimums. Compress a 1,500-token cached prefix to 800 tokens, and it falls under a 1,024-token minimum, so that prefix stops getting the discount.
Prefix changes. OpenAI's guide notes that summarization, compaction, or truncation can change the prefix and reset cache reuse. If compressed text sits inside the prefix you want cached, any change in it breaks the match.
Question-aware methods. LongLLMLingua's authors note that it has to recompress for each new question, even with the same context, which prevents caching that context.
A workable split is to cache the stable prefix (system prompt, tool definitions) and compress what changes per request (retrieved passages, older turns). Our guide to LLM cost optimization covers caching alongside the other cost levers.
Retrieval filtering and reranking decide which documents enter the prompt at all; compression decides how much of each survives. Filter and rerank first, because compressing an irrelevant passage still sends noise, just less of it. If you're still deciding whether to retrieve or pass whole documents, start with long context vs RAG.
How much quality do you lose?
It depends on the ratio, the method, and the task, and the published numbers come from benchmarks that may look nothing like your traffic.
Start with units, since "compression ratio" means three different things. For an 8,000-token context compressed to 2,000 tokens:
Where you see it | Definition | Value for 8,000 to 2,000 |
|---|---|---|
Papers ("20x compression") | original รท compressed | 4x |
LLMLingua | compressed รท original | 0.25 |
LiteLLM | fraction of tokens removed | 0.75 |
What the papers report. Beyond the LLMLingua figures above:
Selective Context cut context cost by 50%, with a drop of .023 in BERTScore and .038 in faithfulness on four downstream applications.
LongLLMLingua improved question answering on NaturalQuestions by up to 21.4% while sending GPT-3.5-Turbo about 4x fewer tokens.
A 2026 study by Kummer et al. found LLMLingua delivered up to 18% end-to-end speed-ups when prompt length, compression ratio, and hardware were well matched, with quality statistically unchanged on summarization, code generation, and question answering. Outside that window, the compression step dominated and canceled out the gains.
Where it breaks. Benchmarks average over many questions, so they hide the single-word failures that matter in production. Our runs showed three, all with LLMLingua-2 on CPU:
Dropped constraints. "Never share internal discount codes" became "share internal discount codes" at rate 0.5.
Lost qualifiers. "Refunds are only available within 30 days" lost "only" in the same run.
Broken formats. In a 41-token instruction requesting JSON with "answer" and "confidence" keys, rate 0.5 dropped "not" from "Do not add text before or after JSON" and removed the exampleโs opening brace. At 0.7, "not" survived, but the example still lost required quotation marks.
How the techniques compare:
Technique | What it removes | Typical failure | Works with caching? |
|---|---|---|---|
Manual compression | Duplicates, filler, stale rules | A rule deleted by mistake | Yes, the prefix stays stable |
Token-level compression (Selective Context, LLMLingua, LLMLingua-2) | Low-information tokens | Negations, qualifiers, format syntax | If compressed text stays outside the cached prefix |
Question-aware compression (LongLLMLingua) | Text unrelated to the question | Context a follow-up needs | No, it recompresses per question |
Extractive or abstractive summarization (RECOMP) | Sentences, or all but a summary | Facts merged or dropped | Depends on placement |
Compaction | Older turns | Identifiers and earlier commitments | Resets reuse if the prefix changes |
How to test compression before shipping
Compression is a tradeoff you can measure. Run the original prompt and each compressed variant on the same fixed eval set, score them with the same checks, and compare quality against tokens and latency.
Build the eval set from your failure modes. Include cases where one dropped word flips the answer: a refund request on day 45, a request for a discount code, a question the documents don't answer, and a reply that must parse as JSON.
Score outcomes directly. Pick evaluation metrics that check the outcome itself: an exact label, schema validation, or a rubric.
Change one knob per variant. Compression rate, method, or prompt variant, with everything else held fixed.
Count all the work. The number of tokens sent, the compressor's own run time, and any extra calls a retrieval tool triggers.
This is the kind of experiment RapidFire is built to run. Our open-source engine shards your eval data and runs configurations side by side on the same hardware, so every compression setting sees the same examples. In run_evals, each leaf config can have its own preprocess_fn, which runs before generation and returns the reserved "prompts" key. That's where a compressor belongs. The evals functions docs define the contract, and RFGridSearch accepts a plain list of config dicts:
Here, api_config is an RFAPIModelConfig for your provider, and config_group goes to run_evals with your dataset (num_shards defaults to 4, and the docs recommend at least 4). On the refund paragraph, the rate 0.7 variant kept "Never share internal discount codes" and the 0.5 variant didn't. Finding that threshold is the point of the comparison.
A few things to know when you read the results:
To watch estimates update as shards finish, set up online aggregation. Confidence intervals that narrow shard by shard come from online aggregation, which needs metrics typed as algebraic or distributive with a value range and an
accumulate_metrics_fn. Without that function, metrics are computed once over the whole dataset at the end. Shuffle the dataset first if it's sorted.Built-in telemetry covers model calls. For API generators, the dashboard's latency, token, and spend metrics come from traced generator calls, and tokens and spend are projected to the full dataset. That's why the snippet times the compressor itself.
Stopping is up to you in the open-source engine. Stop a weak variant or Clone-Modify it with a different
preprocess_fnfrom the dashboard or a notebook panel using IC Ops. Our Converge platform automates those calls: it stops what isn't working, clones what is, explains each decision, and never exceeds the time or spend you set.
To try the pattern, open the Colab notebook from the RAG getting-started guide. It needs no local setup but is limited to small models and datasets, so use it to wire your compressor into a preprocess_fn before you scale up.
Prompt compression tools
Here's where the main options fit. None of them tells you whether the compressed prompt still answers correctly on your data; that's the eval set's job.
Tool | Type | What it does | Watch for |
|---|---|---|---|
LLMLingua ( | Token-level compression, MIT license | From Microsoft Research; ships LLMLingua, LongLLMLingua and LLMLingua-2 compressors | Runs a local model; LLMLingua-2 base ran on CPU in our tests |
Selective Context ( | Token, phrase or sentence pruning | Self-information from a small causal LM, GPT-2 in its README | Research code; tune |
LiteLLM | Conversation compression, beta | Stubs low-relevance messages, retrieval tool for originals | Default trigger of 200,000 tokens |
Anthropic compaction | Provider-side summarization, beta | Server-written summary replaces older turns | Claude API feature; check what the summary keeps |
PromptLayer | Prompt management and evals | Versioned prompt registry and version comparison | Doesn't compress prompts itself; useful for reviewing manual compression |
FAQ
What is prompt compression?
Prompt compression shortens a model's input while keeping the information the task needs, by editing the prompt by hand, pruning low-information tokens with a tool like LLMLingua, or summarizing older context. It reduces token count, cost, and prompt processing time, at some risk to answer quality.
What is lossless compression in simple words?
Lossless compression shrinks data so the original can be rebuilt exactly, the way gzip and PNG work. Lossy compression throws some data away for good. Prompt compression is almost always lossy, because deleted tokens are gone. LiteLLM's compress() is a partial exception, since it keeps the original messages in a cache the model can retrieve from.
What is prompt compaction?
Prompt compaction, also called context compaction, replaces older turns in a long conversation or agent session with a summary so the session keeps fitting in the context window. Anthropic offers it as a beta API feature, or you can run your own summarizer.
What are two types of compression?
In general data compression, the two types are lossless and lossy. In prompt compression research, the split is between hard and soft methods. Hard prompt methods keep natural-language tokens and work with any text API, while soft prompt methods encode the prompt into learned vectors that the model must be built to accept.