Classic NLP techniques: when you don't need an LLM

23 techniques that are faster, cheaper and more predictable than a language model, each explained with a diagram and runnable Python, plus the decision ladder for knowing which one to reach for.

By Shafin ZamanLast updated 14 September 2026Runnable Python throughout

Every technique and code sample as one Markdown file.

TL;DR

Classic NLP still wins for extraction, ranking, tagging and pattern-matching: regex, spaCy, TF-IDF, BM25 and sentence embeddings run in microseconds to milliseconds on CPU, cost nothing, and return the same answer every time. An LLM is the right tool for open-ended understanding and generation. The rule that keeps systems fast and cheap: reach for the smallest tool that works, and let the LLM be the last step, not the first.

There is a reflex right now where every text problem becomes a prompt. Extract an order number? Prompt. Rank some documents? Prompt. Tag a ticket? Prompt. It works, which is exactly why it is dangerous: it works while quietly costing a hundred times more, running a thousand times slower, and returning a different answer on Tuesday than it did on Monday.

None of this is an argument against language models. They are genuinely extraordinary at the thing they are for: open-ended understanding, generation, and reasoning you could not specify in advance. It is an argument against using them as the first tool instead of the last one. Most production text pipelines are ninety percent boring, deterministic work with one genuinely open-ended step at the end. Classic NLP does the ninety percent for free.

Below: how to choose, then 23 techniques with runnable Python and an honest caveat for each, then hybrid search, and a real pipeline where only the last step is generative.

How to choose: climb only as far as you have to

Start at the top. Stop at the first rung that solves your problem. Every rung you skip past costs you latency, money and determinism.

Every rung you climb costs more latency, more money and more unpredictability. Stop at the first one that solves your problem.

1
Does the thing have a fixed, reliable shape?
A regular expression
Order IDs, dates, error codes, postcodes. Microseconds, deterministic, unit-testable, free.
2
Is it a standard entity type?
spaCy NER
People, organisations, places, money, dates. A trained model, milliseconds on CPU.
3
Is it still extraction, but with your own labels?
GLiNER
Define labels in plain English like a prompt, keep encoder speed and encoder cost.
4
Are you matching on exact words, codes or names?
BM25
Rare tokens and identifiers are where lexical search is genuinely unbeatable.
5
Are you matching on meaning, where the words differ?
Sentence embeddings and cosine similarity
Paraphrase, synonyms, and questions that share no vocabulary with their answer.
6
Honestly, both? (This is most real search.)
Hybrid: BM25 and vectors, fused
Each one covers the other's blind spot. Usually a bigger win than a better model.
7
Is it open-ended understanding, generation or reasoning across sources?
Now use an LLM
This is the job it is genuinely best at, and now it is doing it on a small, clean, pre-filtered input.

Notice what has happened by the time you reach rung seven: the input is small, clean and pre-filtered. That is why the last step is cheap, fast and far less likely to hallucinate. Teams that start at rung seven pay for all of it and get none of it.

Where each one actually wins

Use classic NLP when

The output must be the same every single time
You need to explain, in court or in a postmortem, why it did that
Volume is high and margin per item is thin
Latency is measured in milliseconds, not seconds
The data cannot leave your machine
The pattern is known and stable (IDs, codes, formats)
You need a unit test, not an eval suite

Use an LLM when

The task is open-ended: what is this person actually upset about?
You need generation: a summary, a reply, a rewrite, a translation with nuance
The schema is unknown until you see the input
Reasoning has to span several documents at once
Volume is low and variety is high, so engineering time costs more than tokens
You are prototyping and want an answer today, not a pipeline
The edge cases are genuinely unbounded

The difference that matters most in production is the failure mode. Classic NLP fails loudly and predictably. An LLM fails confidently. A regex that stops matching throws zero results and your tests go red. A model that drifts returns something plausible, and you find out from a customer.

Clean and split

Get raw text into consistent, countable units. Everything downstream inherits the mistakes you make here.

Text normalisation

Before you count anything, make the same word look the same. Unicode normalisation, case folding and whitespace collapsing turn Café's, CAFE'S and café's into one token instead of three.

import re, unicodedata

def normalise(text: str) -> str:
    text = unicodedata.normalize("NFKC", text)  # curly quotes, ligatures, full-width
    text = text.casefold()                      # more aggressive than .lower()
    text = re.sub(r"\s+", " ", text)            # collapse all whitespace
    return text.strip()

normalise("  The   Café's   RESUMÉ ")
# "the café's resumé"

Use when: Always, as step one of any matching, search or counting pipeline. It is the cheapest accuracy win in NLP.
Caveat: Normalisation destroys signal as well as noise. Case separates Apple from apple and US from us; punctuation carries sentiment. Normalise the copy you match on, and keep the original for display and for anything a model reads.

Tokenisation

Splitting text into the units you actually work with. Word tokenisation is what classic NLP uses. Subword tokenisation (BPE, WordPiece) is what every LLM uses underneath, which is why your prompt's word count never matches its token count. Character tokenisation is the typo-proof fallback.

Word

unbelievableone unit — unseen words are simply unknown

Subword

un##believ##ablewhat an LLM sees — unseen words still decompose

Char

unbelievabletypo-proof, but the sequence gets long fast

The same word, split three ways. Token counts differ per tokeniser, which is why a word count never equals a token count.

# Word-level — respects contractions, URLs, punctuation
import spacy
nlp = spacy.load("en_core_web_sm")
[t.text for t in nlp("Don't email me at a@b.com!")]
# ['Do', "n't", 'email', 'me', 'at', 'a@b.com', '!']

# Subword — what an LLM actually sees
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
tok.tokenize("tokenisation")
# ['token', '##isation']

Use when: Word tokens for anything lexical (BM25, TF-IDF, regex rules). Subword when you are budgeting or truncating for a model.
Caveat: text.split() is not tokenisation. It breaks on contractions, punctuation, URLs and every language that does not use spaces. Token counts also differ per tokeniser, so never size an LLM prompt by word count.

Stemming

Chop affixes off with fixed rules so running, runs and runner collapse to one index term. No dictionary, no context, just a fast rule cascade.

Input	Stemmer	Lemmatiser
running	run	run
runs	run	run
ran	ran	run
better	better	well
universal	univers	universal

Stemming chops by rule, so it is fast but wrong at the edges. Lemmatisation uses a dictionary and part of speech, so it returns real words.

from nltk.stem import PorterStemmer

ps = PorterStemmer()
[ps.stem(w) for w in ["running", "runs", "ran", "university", "universal"]]
# ['run', 'run', 'ran', 'univers', 'univers']

Use when: Search indexing at scale, where recall matters more than pretty output and you need microseconds per token.
Caveat: It is a crude chop and it shows: ran never reaches run, while university and universal both collapse to univers. Fine inside an index nobody reads, wrong anywhere a human sees the output.

Lemmatisation

The dictionary-aware version of stemming. It uses part of speech and a lexicon to map a word to its real base form, so mice becomes mouse and were becomes be.

import spacy
nlp = spacy.load("en_core_web_sm")

[(t.text, t.lemma_) for t in nlp("The mice were running better")]
# [('The','the'), ('mice','mouse'), ('were','be'),
#  ('running','run'), ('better','well')]

Use when: Whenever the normalised form is shown to a human or fed into a rule: analytics dashboards, tag clouds, entity dictionaries.
Caveat: It needs part-of-speech context, which makes it roughly an order of magnitude slower than stemming and dependent on a model for your language. On informal text the tagger's mistakes become the lemmatiser's mistakes.

N-grams

Consecutive runs of n tokens. Unigrams throw word order away entirely; bigrams and trigrams put just enough of it back to tell machine learning from learning machine.

newyorkpizzadelivery(new, york)

newyorkpizzadelivery(york, pizza)

newyorkpizzadelivery(pizza, delivery)

A bigram is a window of two sliding one token at a time. Trigrams slide a window of three. This is how a bag of words gets a little word order back.

from nltk import ngrams

list(ngrams("new york pizza delivery".split(), 2))
# [('new','york'), ('york','pizza'), ('pizza','delivery')]

# In practice you get them free from the vectoriser:
TfidfVectorizer(ngram_range=(1, 2), min_df=2)

Use when: Any bag-of-words or TF-IDF model where multi-word terms carry the meaning: product names, place names, technical phrases.
Caveat: Vocabulary explodes combinatorially. Bigrams can be ten to a hundred times your unigram vocabulary and most appear exactly once. Cap with min_df or max_features or you will spend a lot of memory modelling noise.

Chunking

Cutting a long document into pieces small enough to embed or feed to a model. It sounds trivial. In RAG it is quietly the single biggest quality lever you control.

Fixed-size chunks with overlap. The shaded bands are the overlap that stops a sentence being cut clean in half at a boundary.

def chunk(text, size=800, overlap=100):
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

# Better: split on structure first, size-cap only the leftovers
import re
sections = re.split(r"\n#{1,3} ", markdown)   # headings

Use when: Before embedding anything longer than a paragraph. Split on structure (headings, paragraphs, list items) first and fall back to fixed size with overlap.
Caveat: Fixed-size chunking cuts sentences, tables and code blocks in half, and a chunk that starts mid-thought embeds to nonsense. Overlap papers over the seams; respecting document structure actually fixes them.

Read the grammar

Let the sentence's own structure tell you what is going on, instead of asking a model to guess.

Part-of-speech tagging

Label every token with its grammatical role: noun, verb, adjective. It is how you pull every adjective customers use about delivery without asking a model anything.

BookVERB

mePRON

aDET

flightNOUN

Every token gets a grammatical role. Note that 'Book' is a verb here and would be a noun in 'read the book' — resolving that is exactly the tagger's job.

import spacy
nlp = spacy.load("en_core_web_sm")

[(t.text, t.pos_) for t in nlp("Book me a flight")]
# [('Book','VERB'), ('me','PRON'), ('a','DET'), ('flight','NOUN')]

# Every adjective in a review corpus, in one line
[t.lemma_ for t in nlp(review) if t.pos_ == "ADJ"]

Use when: Feature extraction, keyword filtering (keep nouns, drop determiners), and as the input that lemmatisation and parsing depend on.
Caveat: Accuracy falls off a cliff on informal text, all caps and domain jargon. Book as verb-or-noun is exactly the ambiguity it resolves, and exactly what it gets wrong on short fragments with no surrounding context.

Dependency parsing

Builds the grammatical tree of a sentence: which word governs which. That gives you who did what to whom as structure, rather than as a guess.

The parse tree of a sentence: each arc runs from a head word to the word it governs. Read the nsubj and dobj arcs and you have who did what to whom, with no model guessing.

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Acme acquired Foo for $2M")

[(t.text, t.dep_, t.head.text) for t in doc]
# [('Acme','nsubj','acquired'), ('acquired','ROOT','acquired'),
#  ('Foo','dobj','acquired'), ('for','prep','acquired'), ...]

subject = [t for t in doc if t.dep_ == "nsubj"]
obj     = [t for t in doc if t.dep_ == "dobj"]

Use when: Relation extraction over structured, well-formed text: contracts, filings, news leads, clinical notes.
Caveat: Long or garden-path sentences derail it, and because you build rules on top of the tree, one bad attachment silently corrupts everything downstream. Check it against your own text before you trust it.

Named entity recognition

Spot and classify the named things: people, organisations, places, dates, money. A small statistical model does it in milliseconds on CPU.

ShafinPERSONshippedInboxWarmORGfromLahoreGPEin2026DATE

Spans are found and classified in one pass. The label set is fixed at training time, which is why your own entity types need GLiNER or a fine-tune.

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Shafin shipped InboxWarm from Lahore in 2026")

[(e.text, e.label_) for e in doc.ents]
# [('Shafin','PERSON'), ('InboxWarm','ORG'),
#  ('Lahore','GPE'), ('2026','DATE')]

Use when: Redaction, tagging, routing and anywhere you need the standard entity types across a large volume of text.
Caveat: The label set is fixed at training time: PERSON, ORG, GPE, DATE and friends. Your domain's entities (SKU, policy number, dosage) are simply not in it, and it will confidently mislabel them. That gap is exactly what GLiNER fills.

Collocations

Word pairs that appear together far more often than chance: machine learning, strong coffee, deliverability rate. It finds your domain's real vocabulary without you writing a dictionary.

from nltk.collocations import BigramAssocMeasures, BigramCollocationFinder

finder = BigramCollocationFinder.from_words(tokens)
finder.apply_freq_filter(5)          # ignore rare pairs — important
finder.nbest(BigramAssocMeasures().pmi, 10)

Use when: Bootstrapping a domain glossary, finding phrases worth indexing as one unit, or mining the terms your users actually pair together.
Caveat: Pointwise mutual information wildly over-rewards rare pairs: two words that each appear once, together, score highest of all. Without a frequency filter your top collocations are just typos.

Turn text into numbers

Search, similarity, clustering and classification all need vectors. These are the ways to make them.

Bag of words

Count how many times each vocabulary word appears in each document. That is it. The document becomes a vector of counts and word order is discarded completely.

Document	cat	dog	sat	the
the cat sat	1	0	1	1
the dog sat	0	1	1	1
sat the cat	1	0	1	1

Each document becomes a vector of counts. Look at rows one and three: different sentences, identical vectors, because word order is thrown away.

from sklearn.feature_extraction.text import CountVectorizer

X = CountVectorizer().fit_transform(["the cat sat", "the dog sat"])
X.toarray()
# [[1, 0, 1, 1],
#  [0, 1, 1, 1]]     vocab: cat, dog, sat, the

Use when: A fast, explainable baseline for classification (spam, routing, topic) that trains in seconds and that you can debug by reading the weights.
Caveat: Order is gone, so dog bites man and man bites dog are the identical vector. The vectors are huge and sparse, and any word unseen at training time is silently dropped at inference.

TF-IDF

Bag of words, weighted by how informative each term is. Term frequency says how much this document is about a word; inverse document frequency crushes words that appear everywhere. The scores near zero, your distinctive domain terms float to the top.

TermTFIDFWeight

the

flights

paris

Weight = term frequency x inverse document frequency. 'the' is everywhere so its IDF collapses to almost nothing; 'paris' is rare across the corpus, so it carries the signal.

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

docs = ["cheap flights to paris", "paris hotel deals", "flight booking guide"]
v = TfidfVectorizer(ngram_range=(1, 2))
X = v.fit_transform(docs)

# What actually distinguishes document 0?
terms = np.array(v.get_feature_names_out())
terms[np.argsort(X[0].toarray()[0])[-3:]]
# ['cheap flights', 'flights', 'cheap']

Use when: Keyword scoring, document similarity, classification features, and as the sane default whenever someone says we need search and the corpus is small.
Caveat: It is purely lexical, so car and automobile share nothing at all. Inverse document frequency is also corpus-relative, which means adding documents silently shifts every score you have already computed.

Word embeddings

Represent each word as a dense vector learned from the company it keeps, so that geometry encodes meaning. The famous demonstration: king minus man plus woman lands near queen.

Meaning becomes geometry. The arrow from man to king is the same direction and length as the arrow from woman to queen, which is why king minus man plus woman lands on queen.

import gensim.downloader as api
wv = api.load("glove-wiki-gigaword-100")

wv.most_similar(positive=["king", "woman"], negative=["man"])[0]
# ('queen', 0.77...)

wv.similarity("inbox", "email")
# 0.6...

Use when: Synonym expansion, fuzzy matching, and as features wherever you need meaning but not a full transformer.
Caveat: One vector per word, forever. Bank the riverside and bank the institution are averaged into a single blurred point, and the vectors bake in whatever bias was in the training corpus. Contextual models fixed the first problem, not the second.

Word2Vec (CBOW and Skip-gram)

The algorithm that made embeddings practical, in two flavours. CBOW predicts a word from its surrounding context and is fast on frequent words. Skip-gram predicts the context from the word: slower, but much better on rare terms and small corpora.

CBOW

thesaton

model

cat

Predict the missing word from its context. Faster, and better on frequent words.

Skip-gram

cat

model

thesaton

Predict the context from the word. Slower, but much better on rare words.

Same data, opposite directions. Both end up producing the word vectors; they just learn them from different prediction tasks.

from gensim.models import Word2Vec

sents = [["email","lands","in","spam"],
         ["warmup","improves","inbox","placement"]]

# sg=0 → CBOW      (context → word, fast)
# sg=1 → Skip-gram (word → context, better for rare terms)
model = Word2Vec(sents, vector_size=100, window=5, min_count=1, sg=1)
model.wv.most_similar("inbox")

Use when: You have a large in-domain corpus full of vocabulary pretrained vectors have never seen: internal jargon, product codes, a non-English domain.
Caveat: It needs a lot of text to beat off-the-shelf vectors. Under roughly ten million tokens you are almost always better off loading pretrained GloVe or fastText than training your own.

FastText

Word2Vec that also learns vectors for character n-grams, so a word is the sum of its pieces. That means it can embed a word it has never seen, including your typos and your long compound words.

Worddeliverybroken into character n-grams

Grams

Typodelivreyshares most of the same grams, so it lands nearby

A word is the sum of its character n-grams, so a word the model has never seen — including a typo — still gets a sensible vector built from familiar pieces.

from gensim.models import FastText

model = FastText(sents, vector_size=100, window=5, min_count=1)

model.wv["delivrability"]        # typo — still gets a sensible vector
model.wv.most_similar("deliverability")

Use when: Noisy user input, morphologically rich languages (Turkish, Finnish, Urdu), and any vocabulary that keeps growing after you ship.
Caveat: Bigger models and slower training than Word2Vec. Sharing character n-grams is also not the same as sharing meaning: deliverability and deliverable look similar to it whether or not they belong together.

Sentence transformers

Small transformer encoders fine-tuned so that whole sentences with similar meaning land close together. This is where practical semantic search actually comes from. all-MiniLM-L6-v2 is 22.7M parameters producing 384-dimension vectors, fast enough to run on CPU.

how do I cancel?where is the refund button?

encoder

[0.12, -0.04, ...][0.14, -0.02, ...]cosine = 0.60

No shared keywords at all, yet the two questions land close together — that is the whole point of a sentence encoder, and why semantic search finds answers that lexical search misses.

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")
emb = model.encode(["how do I cancel?", "where is the refund button?"])

util.cos_sim(emb[0], emb[1])
# tensor([[0.60]])  — no shared keywords, still related

Use when: Semantic search, deduplication, clustering, reranking, and the retrieval half of RAG. This is the workhorse of the whole list.
Caveat: There is a hard input ceiling: all-MiniLM-L6-v2 truncates at its maximum sequence length (128 tokens) and does it silently, so an unchunked document is quietly half-ignored. Similarity is topical, not factual, too, so a claim and its exact negation embed close together.

Cosine similarity

The standard way to compare two vectors: the cosine of the angle between them. Because it ignores magnitude, a three-word query and a thousand-word document can still score as a strong match.

Cosine measures the angle, not the length. The short and long vectors point the same way, so they score 1.0 — which is why a three-word query can match a thousand-word document.

import numpy as np

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

# If your vectors are already L2-normalised (most embedding APIs
# return them that way) cosine is just a dot product:
scores = embeddings @ query_vec

Use when: Every time you rank by vector similarity. Normalise once up front and the whole ranking collapses into a single matrix multiply.
Caveat: A high cosine means points the same way, not relevant and certainly not true. Near-duplicates, exact opposites on the same topic and boilerplate all score high, which is why serious retrieval adds a reranker or a lexical signal alongside it.

Find and extract

Pull the right document out of a pile, or the right field out of a sentence.

BM25

The ranking function that actually runs search. It is TF-IDF's better-behaved successor with two fixes that matter: term frequency saturates, so the tenth paris adds almost nothing over the third, and long documents are penalised so they cannot win on length alone. It is the default in Lucene, Elasticsearch and OpenSearch.

Raw term frequency (dashed) rewards repetition forever, so stuffing a word 50 times wins. BM25 (solid) saturates: the third mention already counts for most of what the tenth will.

from rank_bm25 import BM25Okapi

corpus = ["cheap flights to paris", "paris hotel deals", "flight booking guide"]
bm25 = BM25Okapi([d.split() for d in corpus])   # you do your own preprocessing

bm25.get_scores("paris flights".split())
# array([1.19, 0.35, 0.11])

bm25.get_top_n("paris flights".split(), corpus, n=2)

Use when: Exact terms, rare tokens and identifiers: error codes, SKUs, surnames, acronyms, legal citations. It is also a shockingly strong RAG baseline that most people skip straight past.
Caveat: Zero semantics: a search for car will never find automobile. It also has two tuning knobs, k1 and b, that are corpus-specific and that almost nobody ever actually tunes.

Regex extraction

If the thing you are extracting has a shape, a regular expression finds it in microseconds, deterministically, with a test suite around it. Order IDs, invoice numbers, dates, tickers, IP addresses, version strings.

import re

ORDER = re.compile(r"\bORD-\d{6}\b")
ORDER.findall("refunds for ORD-123456 and ORD-987654 please")
# ['ORD-123456', 'ORD-987654']

# Named groups turn a match straight into a record
M = re.compile(r"(?P<level>ERROR|WARN)\s+(?P<code>[A-Z]{2}\d{3})")
M.search("ERROR AB123 upstream timeout").groupdict()
# {'level': 'ERROR', 'code': 'AB123'}

Use when: Any fixed-format field. This is the most under-used tool on the page: people routinely spin up a model to find a pattern they could have matched exactly.
Caveat: Regex cannot count nesting or understand context, so it cannot parse HTML and it cannot fully validate an email address (the real grammar is far worse than you think). A clever regex is also write-only code, so comment it or use re.VERBOSE.

Find what you did not know

Unsupervised passes over a corpus you have not read yet.

Topic modelling (LDA)

Unsupervised discovery of the themes running through a corpus. Latent Dirichlet Allocation models each document as a mixture of topics and each topic as a distribution over words, so you get clusters nobody had to label.

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer(stop_words="english", max_df=0.9)
X = vec.fit_transform(docs)
lda = LatentDirichletAllocation(n_components=5, random_state=0).fit(X)

terms = vec.get_feature_names_out()
for topic in lda.components_:
    print([terms[i] for i in topic.argsort()[-8:]])

Use when: Exploring a large corpus you have not read: support tickets, survey responses, reviews. It tells you what is in there before you decide what to build.
Caveat: You must pick the number of topics up front, results shift between runs, and a topic is only a word list until a human names it. For most modern work, clustering sentence embeddings gives cleaner topics with far less tuning.

Keyword extraction

Pull the key phrases out of a single document with no training, no model and no corpus. YAKE scores candidates on statistical features alone: position, frequency, casing and how widely the term spreads through the text.

import yake

kw = yake.KeywordExtractor(lan="en", n=3, dedupLim=0.9, top=5)

for phrase, score in kw.extract_keywords(text):
    print(phrase, score)   # LOWER score = MORE relevant

Use when: Auto-tagging, generating meta keywords, summarising a document in five phrases, or pre-filtering before a more expensive step.
Caveat: The score is inverted: lower means more relevant, which trips up nearly everyone the first time. And it is purely statistical, so it surfaces what is frequent and distinctive, which is not always what is important.

Language detection

Work out what language a string is in before you send it down the wrong pipeline. A dedicated detector beats asking a model, and costs nothing to run.

from lingua import Language, LanguageDetectorBuilder

detector = (LanguageDetectorBuilder
            .from_languages(Language.ENGLISH, Language.FRENCH, Language.URDU)
            .build())

detector.detect_language_of("Bonjour tout le monde")
# Language.FRENCH

Use when: Routing multilingual input, picking the right stemmer or model, and filtering a scraped corpus. Restrict the candidate languages: it raises both accuracy and speed.
Caveat: Accuracy drops sharply on very short strings (under roughly 120 characters). Code-switched and transliterated text, such as Urdu written in Latin script, confuses every detector on the market.

The modern bridge

Small neural models that give you prompt-like flexibility at classic-NLP speed and cost.

Schema-guided extraction (GLiNER)

The bridge between the two worlds. You declare arbitrary entity labels at runtime, the way you would in a prompt, but it is a small encoder that runs on CPU in milliseconds instead of a generative model billed per token.

Labelsorder idpersoncitydeadlinedefined by you, at runtime

TextRefund ORD-123456 to Shafin in Lahore by Friday

OutORD-123456 = order idShafin = personLahore = city

You supply the label set at runtime, the way you would in a prompt, but the work is done by a small encoder rather than a generative model billed per token.

from gliner import GLiNER

model = GLiNER.from_pretrained("gliner-community/gliner_small-v2.5")

text = "Refund ORD-123456 to Shafin in Lahore by Friday"
labels = ["order id", "person", "city", "deadline"]

model.predict_entities(text, labels, threshold=0.5)
# [{'text': 'ORD-123456', 'label': 'order id'},
#  {'text': 'Shafin', 'label': 'person'}, ...]

Use when: Custom entity types at volume, which is exactly the job people spin up an LLM for. Define the schema in plain English and get close to spaCy-speed extraction.
Caveat: It extracts spans and nothing else: no reasoning, no normalisation, no generation. Quality also drops on labels far from its training distribution, so measure on your own data before trusting it. And it is still a neural model, so CPU-fast, not regex-fast.

The one that matters most

Hybrid search: BM25 and vectors together

BM25 and embeddings fail in exactly opposite directions, which is why serious search runs both. Look at what happens to two different queries over the same corpus:

Query	BM25	Embeddings
ORD-123456	Exact hit, instantly	Near-random, IDs carry no meaning
my order never showed up	Misses “package not delivered”	Strong match, no shared words

BM25

1.D3
2.D7
3.D1
4.D9

Vectors

1.D7
2.D2
3.D3
4.D5

Fused (RRF)

1.D7in both
2.D3in both
3.D2vectors only
4.D1BM25 only

Reciprocal Rank Fusion combines ranks, not scores, so the two retrievers never need their numbers normalised. Documents that both retrievers liked (D7, D3) rise to the top.

So run both and fuse the results. Reciprocal Rank Fusion is the standard way, and the reason is practical: BM25 scores are unbounded while cosine similarity sits between minus one and one, so you cannot simply add them. RRF throws the scores away and combines the ranks instead.

def rrf(rankings, k=60):
    """Fuse ranked ID lists. k=60 is the value from the original paper."""
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

final = rrf([bm25_top_ids, vector_top_ids])[:10]

Why this matters for RAG: most “our RAG does not work” problems are retrieval problems, not model problems. Adding BM25 alongside your vector search is usually a bigger quality win than upgrading the model, and it costs nothing to run.

Caveat: RRF deliberately discards score magnitude, so a document that scraped into one list still earns credit. If one retriever is clearly stronger on your data, weight the lists rather than fusing them evenly.

What this looks like in a real pipeline

A support-ticket router, end to end. Four of the five steps are free, deterministic and unit-testable. Only the last one is generative.

1
Normalise and detect languageno model
Fold case, collapse whitespace, identify the language and route anything non-English to its own path. Microseconds.
2
Regex out the structured fieldsno model
Order IDs, invoice numbers, dates. These have a shape, so they are matched exactly and never guessed.
3
GLiNER for the custom entitiesno model
Product name, plan tier, urgency marker. Your labels, defined in plain English, at encoder speed.
4
Hybrid retrieval over resolved ticketsno model
BM25 catches the exact error code, embeddings catch the paraphrase. Fuse and keep the top five.
5
Now call the LLM, once
Hand it the extracted fields and those five tickets, and ask for one drafted reply. Small prompt, grounded input, far less room to hallucinate.

Most teams build step five and nothing else, then wonder why it is slow, expensive and unpredictable. The pipeline above is not more work, it is less: steps one through four are a few dozen lines, and they make step five smaller, cheaper and more accurate.

The tooling, and what each is for

spaCyProduction pipelines

Tokeniser, tagger, parser and NER in a single fast pass. Opinionated, with one obvious way to do things. Reach for it when you are shipping.

NLTKBreadth and teaching

Stemmers, corpora, metrics, collocations, and the long tail of classic algorithms. Slower and more assembly required, but it has the piece spaCy does not ship.

GensimVectors and topics at scale

Word2Vec, FastText and LDA over corpora too big for memory, plus a downloader for pretrained vectors.

scikit-learnThe glue

CountVectorizer and TfidfVectorizer, classifiers, clustering and LDA. Where most classic NLP pipelines actually live.

rank_bm25BM25 without a server

Okapi BM25 and four variants in a few lines, for when a full search engine is overkill.

sentence-transformersEmbeddings

Small, fast encoders for semantic search, clustering and reranking. CPU is fine.

Every technique, side by side

The whole toolkit in one table. “Beats an LLM at” is the job where reaching for a model is the wrong call.

Technique	What it's for	Beats an LLM at	Cost
Text normalisation	Making identical words look identical	Always, it is a prerequisite not an alternative	Free · microseconds
Tokenisation	Splitting text into countable units	Always, it is the substrate everything sits on	Free · microseconds
Stemming	Collapsing word forms, fast	Bulk index normalisation	Free · microseconds
Lemmatisation	Correct dictionary base forms	Any time you need real words out	Free · milliseconds
N-grams	Capturing short-range word order	Phrase-aware features on a budget	Free · microseconds
Chunking	Splitting documents for retrieval	Always, it sits upstream of the model	Free · microseconds
Part-of-speech tagging	The grammatical role of each word	Bulk filtering by word class	Free · milliseconds
Dependency parsing	Grammatical relations between words	High-volume relation extraction on clean prose	Free · milliseconds
Named entity recognition	Finding standard named entities	High-volume tagging with known entity types	Free · milliseconds
Collocations	Discovering multi-word terms	Mining domain vocabulary from a corpus	Free · seconds per corpus
Bag of words	Turning documents into count vectors	Explainable classification baselines	Free · milliseconds
TF-IDF	Weighting terms by informativeness	Keyword scoring and light similarity	Free · milliseconds
Word embeddings	Encoding word meaning as vectors	Cheap synonym and similarity lookups	Free · microseconds after load
Word2Vec (CBOW and Skip-gram)	Learning embeddings from your own corpus	Domain vocabulary no public model knows	Free · minutes to train
FastText	Embeddings that survive unseen words	Typo-tolerant matching at scale	Free · minutes to train
Sentence transformers	Sentence-level meaning as vectors	Semantic matching at scale, on CPU	Free · milliseconds on CPU
Cosine similarity	Comparing two vectors	Always, it is the measurement not the model	Free · microseconds
BM25	Ranking documents by keyword match	Exact terms, IDs, codes, rare words	Free · milliseconds
Regex extraction	Extracting fixed-format fields	Anything with a reliable shape	Free · microseconds
Topic modelling (LDA)	Finding themes without labels	Corpus-scale exploration, cheaply	Free · seconds to minutes
Keyword extraction	Key phrases from a single document	Bulk auto-tagging	Free · milliseconds
Language detection	Identifying the language of text	Always, it is a solved and cheap problem	Free · microseconds
Schema-guided extraction (GLiNER)	Zero-shot extraction with your own labels	Custom entity extraction at volume	Free · tens of milliseconds

The takeaway

Reach for the smallest tool that works. Regex before a model, a model before a large model, and a large model only for the part that genuinely needs open-ended understanding. You end up with a system that is faster, cheaper, testable, and that you can actually explain when someone asks why it did that.

Frequently asked questions

Do I still need classic NLP in the LLM era?+

Yes, for most production text work. Classic NLP techniques are faster (microseconds to milliseconds versus hundreds of milliseconds to seconds), free to run, deterministic, and explainable. An LLM is the right tool for open-ended understanding and generation, but for extraction, ranking, tagging and pattern-matching, classic NLP is usually cheaper and more reliable. The practical rule is to reach for the smallest tool that works: regex for fixed shapes, spaCy for standard entities, BM25 and embeddings for retrieval, and an LLM only for the open-ended step at the end.

BM25 or embeddings: which is better for search?+

Neither alone. BM25 is a lexical ranking function, so it excels at exact terms, rare tokens and identifiers such as error codes, SKUs, surnames and acronyms, but it cannot match car to automobile. Embeddings capture meaning, so they match paraphrases and synonyms, but they perform poorly on identifiers and rare tokens that carry no semantic signal. They fail in opposite directions, so production search almost always runs both and fuses the rankings, typically with Reciprocal Rank Fusion. If your retrieval-augmented generation system is underperforming, adding BM25 alongside vector search is usually a bigger improvement than upgrading the model.

What NLP can I do without a GPU?+

Almost all of it. Every technique on this page runs on CPU. Regex, normalisation, tokenisation, stemming, n-grams, bag of words, TF-IDF and BM25 are pure CPU and take microseconds to milliseconds. spaCy's tagging, parsing and NER run comfortably on CPU. Sentence transformers such as all-MiniLM-L6-v2 are only 22.7M parameters and embed sentences on CPU in milliseconds. GLiNER is explicitly optimised for CPU and consumer hardware. You only need a GPU for training your own transformer or running a large generative model locally.

What is the fastest way to extract fields from text?+

Match the tool to how predictable the field is. If it has a fixed shape, such as an order ID, date or error code, use a regular expression: it runs in microseconds, returns the same answer every time and can be unit-tested. If it is a standard entity such as a person, organisation, place or amount of money, use spaCy's named entity recognition, which takes milliseconds on CPU. If it is a custom entity type specific to your domain, use GLiNER, where you declare labels in plain English at runtime and still get encoder speed. Only reach for an LLM when the field is genuinely ambiguous or the schema is not known in advance.

Is TF-IDF obsolete?+

No, though BM25 has largely replaced it for ranking. TF-IDF is still an excellent, explainable way to score keywords, build classification features and measure document similarity on a small corpus, and it trains in seconds. BM25 is the better choice specifically for search ranking because it adds term-frequency saturation and document-length normalisation. Where TF-IDF genuinely falls short is semantics: it has no concept of synonyms, so pair it with embeddings when meaning matters more than exact wording.

When should I actually use an LLM?+

When the task is open-ended understanding, generation or reasoning that you cannot specify in advance. Good cases include summarising, drafting replies, rewriting, nuanced translation, deciding what a frustrated customer actually wants, and reasoning across several documents at once. It is also the pragmatic choice when volume is low and variety is high, because engineering a pipeline costs more than the tokens. Use it as the last step in a pipeline rather than the first: let classic NLP clean, extract and retrieve, then hand the model a small, grounded input. That is faster, cheaper and hallucinates less.

I post these on LinkedIn, one practical breakdown a week. Follow along for the next one.

Follow me on LinkedIn Next: the AI engineer roadmap