SLM vs LLM: Differences, Costs, and How to Choose (With Real Benchmark Data)

A large model API is a useful starting point when you need an application to handle many kinds of requests. But what if most calls do the same narrow job, such as classifying a support ticket or picking database columns? A smaller model trained for that job may be worth testing.

The SLM vs LLM decision comes down to whether the smaller model can meet your quality requirements at a useful cost and speed. Our schema-linking comparison tested fine-tuned open SLMs against seven closed APIs. This guide explains the differences, when each fits, and where the small models still lost.

SLM vs LLM: The Short Answer

A small language model (SLM) is a compact model, usually under about 10 billion parameters, that's cheaper and faster to run and often small enough to self-host. A large language model (LLM) has tens to hundreds of billions of parameters or more and handles a broader range of tasks. Fine-tuned SLMs can get close to LLMs on narrow, well-defined tasks.

What Is a Small Language Model (SLM)?

There's no official cutoff. Belcak et al.'s working definition uses deployment constraints: an SLM fits on a common consumer device and can serve one user's agent requests at practical latency. Their 2025 paper treats most models below 10 billion parameters as SLMs. The older range of millions to tens of millions excludes the larger models discussed here.

Current open small language models make the range concrete. SmolLM2 comes in 135M, 360M, and 1.7B sizes built to run on-device, and Gemma 4's smallest models, E2B and E4B, target mobile, edge, and browser deployment. Meta's Llama 3.2 ships 1B and 3B models, and Meta labels those sizes as LLMs, illustrating that the SLM/LLM distinction is a working definition rather than a fixed parameter boundary.

A smaller model can still learn from a huge amount of text. SmolLM2-1.7B saw 11 trillion tokens in pretraining, and Llama 3.2's 1B and 3B models saw up to 9 trillion. Size describes the number of model parameters, not the amount of training data.

The trade-off is often breadth. On the MMLU knowledge benchmark, Llama 3.2 1B Instruct scores 49.3, compared with 63.4 for the 3B and 69.4 for Llama 3.1 8B. Those scores don't tell you which will handle your specific task well enough. They show why a compact model is worth testing for a focused job rather than assuming it can replace a larger model everywhere.

What Is a Large Language Model (LLM)?

Under that working definition, an LLM exceeds the practical size or latency envelope of an SLM. Frontier closed models such as GPT-5.5, Claude Fable 5, and Gemini 3.1 Pro preview sit behind an API and bill per token. Open-weight LLMs like Llama 3.1 405B can run on your hardware, but that means a multi-GPU server (our guide to self-hosting a model covers what that takes).

Consider larger models for varied requests requiring broad knowledge or multi-step reasoning, and evaluate them on the same inputs as your smaller candidates.

Key Differences Between SLMs and LLMs

Compare the deployment and workload constraints alongside model size.

Cost per query depends on utilization, covered with numbers below. Accuracy depends on the task and training: evaluate the untuned and fine-tuned candidates separately on the same held-out inputs.

Are SLMs Faster and Cheaper to Run?

Faster, usually, on the same hardware. Meta measured both Llama 3.2 sizes on a OnePlus 12 phone. In BF16, the 1B decodes 19.2 tokens per second with a 1.0-second time to first token, while the 3B manages 7.6 tokens per second and takes 3.0 seconds. Belcak et al., whose working definition we used above, estimate that serving a 7B SLM is 10 to 30 times cheaper than serving a 70B to 175B LLM in latency, energy, and FLOPs.

A token-metered API charges for requests as they arrive. Dedicated GPU capacity incurs cost while idle, so self-hosted cost per query depends on how much of that capacity you use.

Our July 2026 schema-linking case study, linked in the introduction, shows why utilization matters. At 60% utilization, the fine-tuned 1.5B model's estimated serving cost was $419.56 per million queries. That was about 58 times cheaper than GPT-5.5 and 2.1 times cheaper than Gemini 2.5 Flash at $876.

Now suppose traffic halves while the GPU bill and the work required per query stay the same. Each query must cover twice as much of that bill. Under those assumptions, the estimated break-even utilization against Flash is:

0.60 × $419.56 ÷ $876 ≈ 0.29, or 29

0.60 × $419.56 ÷ $876 ≈ 0.29, or 29

0.60 × $419.56 ÷ $876 ≈ 0.29, or 29

Below that level, the self-hosted 1.5B model loses its estimated cost advantage. This is a best-case calculation, not a measured threshold: the study's unbatched serving costs were higher than its 60% utilization estimate. Use your expected traffic and serving setup when making the comparison.

How SLMs Close the Gap: Fine-Tuning, Distillation, and Quantization

Fine-tuning, distillation, and quantization address different constraints: task behavior, knowledge transfer, and serving memory. They can be combined, but each change needs evaluation.

Fine-Tuning on Your Task Data

Supervised fine-tuning (SFT) teaches a model the task using examples of the inputs it will receive and the outputs you want. Our schema-linking case study used 301 such pairs.

LoRA makes that training lighter by leaving the base model's weights unchanged and training small additional matrices, called adapters (Hu et al., 2021). You can merge an adapter into the base weights without adding inference latency, or keep adapters separate so one base model can switch between tasks.

Fine-tuning also improves output format, which an untuned small model can get badly wrong: in the case study below, every response became parseable JSON after training. If you already train with Hugging Face, our drop-in RFSFTConfig, RFDPOConfig, and RFGRPOConfig wrappers let you fine-tune with Hugging Face TRL across several configs at once. If you're still choosing between retrieval and training, read RAG vs. fine-tuning first.

Distillation From a Larger Model

Distillation trains a small "student" to imitate a larger "teacher." For a task-specific model, one approach is to have a strong LLM label your examples, then fine-tune the SLM on those labels. Check your API provider's terms first, because some restrict using their output to build competing models.

A richer version teaches from the teacher's predictions, rather than only its final answers. Hinton, Vinyals, and Dean described using output probabilities as "soft targets" in 2015. Meta used a related approach for Llama 3.2: the 1B and 3B models learned from the token-level scores, or logits, produced by Llama 3.1 8B and 70B during pretraining.

Quantization for Cheaper Serving

Quantization reduces the memory needed to store model weights by using fewer bits per value, such as int8 or int4. That can make a model easier to run on limited hardware. In Meta's phone test, a SpinQuant build of Llama 3.2 1B generated 50.2 tokens per second, 2.6 times the BF16 baseline, from a file 54.1% smaller.

The smaller footprint can come with trade-offs. Naive post-training quantization dropped the 1B's MMLU score from 49.3 to 43.3. Quantization-aware training with LoRA preserved more of that score, at 49.0. The quantized variants also list an 8k context window instead of 128k. Evaluate the actual quantized version you plan to deploy, rather than relying on the original model's scores.

Case Study: Replacing Frontier APIs With a Fine-Tuned SLM

Can a well-defined enterprise task move off closed APIs and onto an open model you fully control without giving up much quality? We tested that on schema linking: picking which tables and columns a natural-language question touches before any SQL is written. It runs on every query, so API bills grow with schema size, and every call ships your schema offsite.

Experiment setup

We used a 17-schema view of the SNAILS text-to-SQL benchmark for our table-and-column selection task. It contains 503 question-SQL pairs across 615 tables and 13,247 columns. We used 301 pairs for training, 101 for validation, and 101 for the final test.

The score combines the share of selected tables and columns that were correct (precision), the share of required ones found (recall), and a measure that balances both (F1). These become a composite score from 0 to 1, so higher scores indicate better selections on this task.

Baseline results

Seven proprietary models ran through the same prompts and grader for $52.34 of live spend. GPT-5.5, Gemini 3.1 Pro preview, and Claude Fable 5 all landed between 0.912 and 0.917. Untuned and zero-shot, Qwen2.5-Coder-1.5B-Instruct scored 0.351, worse than a script that returns every table and column (0.367), and emitted invalid JSON on 59.4% of questions.

Fine tuning sweep

We ran SFT across 16 configurations (4 base models, 2 schema serializations, and 2 LoRA ranks) with RapidFire, which split all 16 runs across 2 A100 GPUs and trained them concurrently. When four SmolLM2 runs hit memory limits, the other 12 kept going. The sweep took 3 hours, 29 minutes of wall time and about $7.52 of compute.

Costs below are the study's July 2026 estimates: API costs are metered, and self-hosted costs are amortized GPU costs at 60% utilization.

Fine-tuning made the 1.5B model a more credible candidate: its score roughly doubled from 0.351 to 0.705, and strict JSON validity rose from 40.6% to 100% on the evaluated questions. It still trailed Gemini 2.5 Flash by 0.135.

The study reports that its second fine-tuned model came within 0.027 of GPT-5.4-mini at 2.8 times lower serving cost. The source labels it 7B but lists no matching checkpoint in its configuration table, so its parameter count is unconfirmed.

The result is evidence that a self-hosted small model can be feasible for this task, with a quality trade-off. It isn't proof that small models generally match frontier APIs: the study itself calls the sweep "a small, initial experiment" with one run per configuration.

Figure 1. Composite score vs. cost per million queries (log scale), using only figures published in the case study. Self-hosted costs assume 60% GPU utilization; API costs are metered.

Cheap training compute doesn't remove the need for ML expertise. An API needs little model engineering to start; fine-tuning requires someone to design and run experiments, evaluate the results, and maintain the model.

When to Use an SLM vs. an LLM

Choose an SLM when:

• The task is narrow, repeatable, and domain-specific, like classification, routing, sentiment analysis, extraction, or schema linking.

• Volume is high and steady enough to keep a GPU busy.

• Latency is strict, or the model has to run on edge devices.

• Data has to stay in-house. A Fortune 500 enterprise tech team explored a small model it could run locally to keep sensitive customer data off external APIs, then scaled to 2,000+ fine-tuning configurations on four GPUs.

• Every call has to return the same strict output format, which fine-tuning can make far more reliable.

Choose an LLM when:

• Requests are open-ended or span many topics, as with a general-purpose assistant.

• The task needs multi-step reasoning or broad general knowledge.

• Volume is low or spiky.

• You're still prototyping, and the task keeps shifting.

• Nobody has time to build an eval set and fine-tune.

Hybrid Setups

You don't have to choose between an SLM and an LLM for the whole system. One option sends narrow, high-volume calls to a fine-tuned SLM and everything else to an LLM. A text-to-SQL service, for example, could let an SLM do schema linking and fall back to an LLM whenever the SLM's JSON fails validation. Belcak et al. propose this mixed architecture: SLMs handle routine work, with LLMs reserved for calls that need broader capabilities.

SLMs in Agentic AI

Repeated, specialized calls give an agent system places to test smaller models. Belcak et al.'s proposed conversion process groups recurring requests—such as intent recognition or data extraction—and fine-tunes an SLM for each group with LoRA or QLoRA. This is a proposed architecture, not a guarantee that every agent workload can migrate. Wiring those models together with tools and checks is the agent harness's job.

Decide With Evidence, Not Benchmarks Alone

Public leaderboards won't tell you how LLMs and SLMs do on your task. In the schema-linking study, Gemini models beat their Claude counterparts on both cost and accuracy for schema linking, a task-specific result you only find by testing. Run your own bake-off:

1. Build an eval set from real requests with known-good answers, and hold part of it out.

2. Score closed APIs and fine-tuned SLM candidates with the same prompts and grader.

3. Put serving cost at your expected volume next to every score.

4. Pick on validation, then confirm on the held-out set.

Our eval workflows accept both closed-model APIs and self-hosted open models as the generator, so you can score both with the same prompts and eval functions. Our guide on how to evaluate models on your task goes deeper on the first two steps.

Testing a fine-tuned candidate also means paying for the experiments that produce it. Each SFT configuration requires a training run. We built RapidFire for this offline, pre-deployment work: you can run an SFT sweep across base models and LoRA adapters concurrently.

Interactive controls, called IC Ops, let you stop weak runs early, give promising ones more training time, and copy a configuration to try a modified version. Those new variants can optionally start from the parent run's learned weights. Compare several choices before deciding what matters: in the schema-linking study, only the base model consistently separated the runs.

Conclusion

Model size alone won't settle the LLM vs SLM decision. The questions that do are how narrow the task is, how much traffic it gets, and whether your data can leave the building. A fine-tuned SLM can get close to frontier quality on one job at a fraction of the serving cost. Check both quality and serving cost on your data at your expected volume.

To try the fine-tuning side yourself, open our fine-tuning notebook on Colab. It fine-tunes Qwen3-0.6B on customer support data and trains two LoRA configs side by side, with no local setup. For the full results, read our schema-linking case study linked in the introduction. If you're weighing a move off closed APIs for a production workload, you can talk to our team.

SLM vs LLM FAQ

Is Copilot SLM or LLM?

Microsoft powers its Copilot experiences with large language models (LLMs) for language understanding and generation. Microsoft also builds SLMs, including the Phi family. Phi Silica, a local model on the NPU of Copilot+ PCs, is being replaced by an on-device model called Aion Instruct.

Are SLMs Faster Than LLMs?

Usually, when both run on the same hardware. In Meta's phone test, Llama 3.2 1B generated about 2.5 times as many tokens per second as the 3B.

What Is an SLM in AI?

An SLM is a language model small enough to run on a single GPU or consumer device, generally under about 10 billion parameters. With fewer parameters, it trades breadth of knowledge for speed, cost, and control.

Are SLMs Cheaper to Run?

Per query, usually, but only at steady volume. A self-hosted model costs money every hour, so at low utilization, a pay-per-token API can be cheaper.

Can an SLM Replace an LLM?

For a narrow task, a fine-tuned SLM may be a useful replacement if it meets your quality and cost requirements. In the schema-linking study, the 1.5B model's score rose from 0.351 to 0.705 after fine-tuning, but it still trailed Gemini 2.5 Flash at 0.840. Test the replacement on your own workload.