LLM-Based vs Traditional NER Compared

Accuracy Comparison

On standard entity types (person, organization, location, date), fine-tuned NER models achieve 90 to 93% F1. LLM-based extraction achieves 88 to 92% F1 on the same types. The accuracy gap is small and shrinking, but NER models still have a slight edge because they are trained specifically for token-level entity boundary detection, while LLMs are general-purpose text generators adapting to an extraction task.

On domain-specific entity types (services, APIs, configuration keys, medical terms), the comparison reverses. Pre-trained NER models score 0% F1 on types they have never seen. Fine-tuned NER models score 82 to 88% F1 after 300+ labeled examples per type. LLM-based extraction scores 80 to 90% F1 on domain-specific types with zero training data, using only prompt descriptions and a few examples. The LLM's world knowledge and language understanding give it a strong starting point on any entity type, while NER models must be explicitly trained.

For relationship extraction, there is no comparison. Traditional NER does not extract relationships at all. It identifies entity mentions but says nothing about how entities connect. LLM-based extraction handles entity and relationship extraction in a single pass (or two sequential passes for higher quality), producing the triples that knowledge graphs need. If you need relationships, you need either an LLM or a separate relationship extraction model.

Cost Comparison

Traditional NER runs on your hardware with no per-document cost. A SpaCy model on a single CPU processes 200 to 10,000 documents per second depending on the model size. The cost is the compute infrastructure, which amounts to a few dollars per million documents on cloud hardware. Fine-tuning adds a one-time cost of $5 to $50 in GPU compute, plus the human cost of labeling training data (20 to 40 hours for a typical domain).

LLM-based extraction costs $0.003 to $0.015 per passage through API calls. Processing 100,000 documents costs $300 to $1,500. Processing 1 million documents costs $3,000 to $15,000. These costs repeat whenever you need to re-extract (for prompt improvements or new entity types). At scale, the cost difference between NER and LLM extraction spans multiple orders of magnitude.

The cost break-even depends on how many entity types require LLM extraction. If 80% of your entities are standard types that NER handles well and only 20% are domain-specific, a tiered approach (NER first, LLM for the remaining 20%) costs 80% less than running everything through the LLM.

Latency Comparison

SpaCy's non-transformer models add less than 1 millisecond per document. Transformer-based NER (SpaCy trf, Hugging Face models) adds 20 to 100 milliseconds per document on a GPU, or 200 to 500 milliseconds on a CPU. Either way, entity extraction adds negligible latency to a real-time pipeline.

LLM-based extraction adds 1 to 5 seconds per passage through API calls. Network latency, queue time, and generation time all contribute. For batch processing (extracting entities from a document corpus), this latency is manageable with parallelization. For real-time extraction (processing user input as it arrives), it is often too slow. A common pattern is to use NER for real-time extraction and run LLM extraction asynchronously in the background, merging the results when the LLM completes.

Flexibility Comparison

Traditional NER models have fixed entity types determined by training data. Adding a new type requires collecting labeled examples, fine-tuning the model, evaluating it, and deploying the new version. This process takes days to weeks depending on how quickly you can produce labeled data. Removing or modifying an existing type requires the same retraining cycle.

LLM-based extraction adapts instantly. Add a new entity type by adding its description to the prompt. Remove one by removing the description. Modify the extraction behavior by changing the prompt instructions. This flexibility is what makes LLMs ideal for exploratory phases where you are still figuring out what to extract, and for domains where entity types evolve faster than you can retrain models.

Coreference and Context

LLMs handle coreference naturally. When a text says "the authentication service processes requests. It depends on Redis for caching," an LLM understands that "it" refers to the authentication service and can extract the relationship correctly. Traditional NER identifies "it" as a pronoun and does not link it to any entity. You need a separate coreference resolution model (like SpaCy's neuralcoref or Hugging Face's coreference models) to resolve pronouns before running NER.

LLMs also handle implicit context better. "We use the same database as the payments team" implies a shared database entity without naming it. An LLM can note this implicit reference and either extract it with a caveat or flag it for resolution. NER models only extract explicitly named entities and miss implicit references entirely.

When to Use Each

Use traditional NER when: you process more than 10,000 documents per day, your entity types are stable, you need sub-100ms latency, and your entity types are well-covered by existing NER models or you have labeled training data for fine-tuning.

Use LLM-based extraction when: your entity types are domain-specific and evolve over time, you need relationship extraction alongside entity extraction, you are in an exploratory phase figuring out what to extract, or your volume is low enough that API costs are manageable.

Use both (tiered extraction) when: you need the throughput of NER for standard types and the flexibility of LLMs for domain-specific types. Run NER first at near-zero cost, then pass the passages with potential domain-specific entities through the LLM. This is the approach most production systems use in 2026, and it is the approach that Adaptive Recall implements automatically.