What Languages Does NER Support Well

Tier 1: Excellent Support (88-93% F1)

English has the most NER resources of any language. SpaCy, Hugging Face, and commercial APIs all provide pre-trained English NER models. Fine-tuning datasets (OntoNotes, CoNLL-2003) have been benchmarked extensively. Every new NER technique is first validated on English benchmarks. If you are building an English-only application, you can choose from dozens of models at every point on the speed/accuracy curve.

Tier 2: Strong Support (85-90% F1)

German, French, Spanish, Dutch, Italian, Portuguese: Major European languages have good pre-trained NER models available through SpaCy and Hugging Face. Multilingual BERT (mBERT) and XLM-RoBERTa provide cross-lingual transfer that achieves 85 to 88% F1 on these languages even without language-specific fine-tuning. Language-specific models fine-tuned on CoNLL-2002/2003 data reach 88 to 90% F1. SpaCy provides dedicated pipelines for German, French, Spanish, Italian, and Portuguese that handle language-specific tokenization and entity patterns.

Tier 3: Good Support (82-88% F1)

Chinese, Japanese, Korean: CJK languages have unique challenges: no whitespace word boundaries (Chinese, Japanese), mixed scripts (Japanese uses kanji, hiragana, katakana, and Latin), and complex entity naming patterns. Language-specific tokenizers and NER models handle these challenges. Chinese NER models trained on the OntoNotes Chinese corpus achieve 82 to 86% F1. Japanese NER models using GiNZA (SpaCy-compatible) achieve 83 to 87% F1. Korean NER benefits from the relatively regular Hangul script but entity names often mix Korean and English (brand names, technology terms), requiring mixed-script handling.

Tier 4: Workable Support (78-85% F1)

Arabic, Hebrew, Turkish, Russian, Hindi: These languages have NER models available but with less extensive training data. Right-to-left scripts (Arabic, Hebrew) require script-aware tokenization. Morphologically rich languages (Turkish, Finnish, Hungarian) have entity boundaries that are harder to detect because entity names are inflected, meaning the same entity appears in different surface forms depending on its grammatical role. Russian NER benefits from dedicated models trained on the FactRuEval corpus. Arabic NER has improved significantly with ArabicBERT and CAMeL Tools, reaching 82 to 85% F1 on Modern Standard Arabic, though dialectal Arabic (Egyptian, Gulf, Levantine) remains harder at 75 to 80%.

Tier 5: Limited Support (70-80% F1)

Low-resource languages: Many African, Southeast Asian, and indigenous languages have minimal or no dedicated NER training data. Traditional NER approaches require labeled examples that do not exist for these languages. Cross-lingual transfer from high-resource languages (training on English, applying to Swahili) achieves 70 to 78% F1, which is usable but not reliable for production applications. MasakhaNER provides training data for 10 African languages and is expanding, but coverage remains sparse compared to European and Asian languages.

The Multilingual LLM Advantage

Multilingual LLMs (Claude, GPT-4) provide a consistent extraction interface across all supported languages. You write the extraction prompt in English, provide the text in any supported language, and the model extracts entities with language-appropriate understanding. This achieves 75 to 85% F1 across 100+ languages without any language-specific model training, making LLMs the practical choice for multilingual entity extraction.

The accuracy varies by language. Languages well-represented in the LLM's training data (English, major European and Asian languages) achieve the high end. Languages with less training representation achieve the low end. But even the low end (75% F1) is often better than what is available from traditional NER for that language.

Cross-Language Entity Linking

For multilingual applications, the hardest problem is not extraction but entity linking: recognizing that "Google" in English, "Google" in Japanese katakana, and "Google" in Arabic script all refer to the same entity. Entity names that are transliterated (phonetically adapted to another script) rather than translated create surface forms that string matching cannot resolve. The solution is to embed entity names into a shared vector space using multilingual embeddings (XLM-RoBERTa, multilingual E5) and match entities by embedding similarity rather than string similarity. This handles transliteration, translation, and abbreviation across languages.

For multilingual AI memory systems, LLM-based extraction provides the most consistent cross-language experience. Adaptive Recall extracts entities from memories in any language, maintaining a unified knowledge graph regardless of the language each memory was stored in. An entity stored from English text is linked to the same graph node when mentioned in Spanish or Japanese text, because the extraction system resolves cross-language entity references to canonical identifiers.