When not to use an LLM

Every few years a tool arrives that's so capable it bends the way we frame problems. Large language models are the latest. They're genuinely remarkable, and that's exactly why they get reached for too often, on problems a plain function, a lookup table, or a small classifier would handle faster, cheaper, and more reliably.

Choosing not to use an LLM is one of the most underrated calls in applied AI. Here are four times it mattered for us, the quick test we run before adding a model, and where they genuinely belong.

When the answer is computable

We once built a file-import flow: a file comes in, the UI runs some validation, then each row gets parsed and its gaps filled in. We handed the whole thing to an LLM, including three columns bound by a fixed relationship. Given any two of them, the third was simple arithmetic. We were, in effect, asking a language model to do sums.

It mostly got them right, which is the worst possible outcome. "Mostly" on arithmetic means a quietly wrong number slips through now and then, and a quietly wrong number in an imported record is the kind of bug nobody notices until it's expensive. We pulled those three columns out of the prompt and wrote the formula instead. It's instant, free, exact, and it will never drift. The model still does the genuinely fuzzy parsing; it just no longer does the maths.

If a problem has a correct answer you can compute, computing it beats predicting it.

When you're not allowed to send the data

Capability isn't the only question. Sometimes the model would handle the task fine and you still can't use it, because of what the data is. We work with health data, which means PHI under HIPAA, which means every service that touches it has to sign a BAA: the business-associate agreement that makes a provider legally accountable for protecting it.

Plenty of the most convenient options don't sign one. OpenRouter, for instance, is a lovely way to reach a dozen models behind a single API, and it doesn't offer a BAA, so for anything carrying PHI it is simply off the table. No benchmark score changes that. The data draws the boundary before capability gets a vote, and "but this model is so much better" is not an argument you want to be making to a regulator.

When the blast radius is the whole system

The third case is scope. A multi-step refactor across several services is where an LLM looks most impressive and is most dangerous. Ask it to restructure a module or rename a field and it will do it cleanly, in the file you showed it. What it can't see is the other service that reads that field by name, the migration that assumed the old shape, the cache key quietly built from it.

A small structural change ripples out through hidden dependencies, and you get runtime crashes that are miserable to trace precisely because nothing broke where the change was made. The model has no model of your system, only of the snippet in front of it. We're glad to let it draft a piece in isolation. We don't let it drive a change that reaches across systems it can't see.

A quick test before you reach for a model

Before we add an LLM to a system, we put the problem through three questions:

• Is the answer deterministic? If the same input should always give the same output, a rule or a lookup is more honest than a model.
• Can you tolerate being wrong? If a confident, incorrect answer causes real harm, you need verifiability, not fluency.
• Is the input genuinely unstructured? Free-form language, messy documents, ambiguous intent: that's where models shine. Clean, structured data rarely needs one.

Where LLMs genuinely earn their place

None of this is anti-LLM. When the problem is actually a language problem, nothing else comes close. We reach for them with confidence when the work involves:

• Turning messy, unstructured text into structure a system can use.
• Drafting, summarising, or rephrasing where a human stays in the loop.
• Open-ended reasoning over context no rule could anticipate.

The one that isn't about the task

A last "when not to," and it's aimed at anyone earlier in their career. The tempting move is to have the model generate the whole solution and move on. Resist it more often than you give in. The struggle you skip, sitting with a problem, getting it wrong, debugging your own mess, is the part that actually builds the skill.

Lean on generation too early and the bill comes later: you ship code you didn't really write, and then you can't read it, can't maintain it, and can't debug it when it breaks at two in the morning and the model is having an off day. An LLM is a fine pair-programmer and a poor substitute for ever having learned to program. Use it to go faster at things you already understand, not to skip understanding them.

The craft was never about using the most powerful tool available. It's matching the tool to the problem, and having the discipline to reach for the small, dependable one, or for no tool at all, when that's the right call. That discipline is most of what separates an AI demo from a system you can run for years.