What Is Named Entity Recognition and How It Works

How NER Models Work

NER models process text token by token, predicting for each token whether it is part of an entity and, if so, what type of entity it belongs to. The standard labeling scheme is BIO: B (beginning of an entity), I (inside an entity, continuing from the previous token), and O (outside any entity, just regular text). Some systems use BIOES, adding S (single-token entity) and E (end of a multi-token entity) for finer granularity.

For the sentence "John Smith works at Acme Corp in New York," the BIO labels are: John (B-PERSON), Smith (I-PERSON), works (O), at (O), Acme (B-ORG), Corp (I-ORG), in (O), New (B-GPE), York (I-GPE). The model learns from thousands of labeled examples which token patterns correspond to which entity types. At inference time, it applies these learned patterns to new text.

Modern NER models use transformer architectures (BERT, RoBERTa, DeBERTa) that understand context bidirectionally. This means the model considers both the words before and after each token when making predictions. Context matters enormously for NER: "Apple" is a company when preceded by "shares of" but a fruit when preceded by "slice of." Pre-transformer NER models (CRFs, BiLSTMs) used limited context windows and performed significantly worse on ambiguous entities.

Standard Entity Types

The entity types recognized by general NER models come from the information extraction research community. The most widely used type system is from the OntoNotes corpus:

PERSON: People, including fictional characters. "Marie Curie," "Dr. Smith," "the CEO" (when a specific person is implied).

ORG: Companies, agencies, institutions, teams. "Google," "the United Nations," "platform engineering team."

GPE: Countries, cities, states, geopolitical entities. "France," "New York City," "the European Union."

DATE: Absolute or relative dates. "January 15, 2026," "last Tuesday," "Q3 2025."

MONEY: Monetary values. "$500," "12 million euros," "the $3.5B acquisition."

PRODUCT: Objects, vehicles, foods, named products. "iPhone 16," "Boeing 787," "Model S."

EVENT: Named hurricanes, wars, sports events. "World War II," "the Super Bowl," "Hurricane Katrina."

LAW: Named documents made into laws. "the Constitution," "GDPR," "Section 230."

These types cover general text well but miss the entity categories that matter for specific domains. Software engineering text needs types for services, databases, APIs, and configuration settings. Medical text needs types for diseases, symptoms, medications, and procedures. This gap between general types and domain needs is what drives the use of LLM-based extraction and fine-tuned NER models.

NER Model Architectures in 2026

SpaCy

SpaCy is the most widely used NER library for production applications. It provides pre-trained models for multiple languages with NER built in. The en_core_web_sm model uses a convolutional architecture that is extremely fast (10,000+ documents per second) but less accurate. The en_core_web_trf model uses a transformer backbone (RoBERTa) that is slower (50 to 200 documents per second) but achieves 89.8% F1 on the OntoNotes benchmark. SpaCy supports fine-tuning on custom entity types, making it a complete NER solution for teams that need both standard and domain-specific types.

Hugging Face Token Classification

The Hugging Face Transformers library provides NER through token classification models. You load a pre-trained model (bert-base-NER, dslim/bert-base-NER, Jean-Baptiste/roberta-large-ner-english), pass text through it, and get token-level entity predictions. These models achieve 91 to 93% F1 on standard benchmarks. The advantage over SpaCy is more flexibility in model selection and fine-tuning configuration. The disadvantage is more engineering work for production deployment (SpaCy handles tokenization, sentence splitting, and post-processing automatically).

GLiNER and Few-Shot NER

GLiNER is a newer approach that recognizes entity types from natural language descriptions rather than from training labels. Instead of fine-tuning on hundreds of labeled examples, you describe the entity types you want ("cloud infrastructure services like AWS, GCP, Azure") and the model identifies matching entities. Accuracy is lower than fine-tuned models (75 to 85% F1) but the zero-shot capability means you can start extracting new entity types immediately. This approach fills the gap between traditional NER (high accuracy, requires training) and LLM-based extraction (high flexibility, high cost).

NER Limitations

NER models have fundamental limitations that shape how they should be used in production systems. They recognize entity mentions (spans of text that refer to entities) but do not resolve which real-world entity a mention refers to (entity linking). Two mentions of "Smith" in a document might refer to different people, and a NER model does not distinguish between them.

NER models do not extract relationships. They tell you that "Redis" and "checkout service" are entities in a sentence, but they do not tell you that the checkout service depends on Redis. Relationship extraction requires a separate model or approach.

NER models struggle with entities that do not look like typical entity names. A configuration key like MAX_RETRY_COUNT or an API endpoint like /api/v2/orders does not resemble the person and organization names that NER models are trained on. Domain-specific fine-tuning helps, but these unusual entity forms often require LLM-based extraction for reliable detection.

For AI memory systems, NER is best used as the fast, cheap first pass in a tiered extraction pipeline. It catches standard entity types at near-zero cost and high throughput. LLM-based extraction then handles the domain-specific entities, relationship extraction, and coreference resolution that NER cannot do. Adaptive Recall implements this tiered approach automatically, extracting entities from every stored memory without requiring you to configure or maintain the extraction pipeline.