language identification (text-classification) · text · WASM · ~270 MB

Language identification — what language is this?

Paste any text and this model tells you which of 20 languages it's written in, with a confidence score. It's a fine-tune of XLM-RoBERTa, a multilingual encoder trained on 100 languages, so every language shares one representation space — the model reads the actual linguistic signal (script, morphology, function words) rather than matching a keyword list. That's why it copes with short snippets and stays honest on mixed or ambiguous text. Nothing leaves your device.

At a glance

Model	onnx-community/xlm-roberta-base-language-detection-ONNX (ONNX of papluca/…-language-detection)
Task / pipeline	`text-classification` — one of 20 language labels + softmax scores
Languages	ar, bg, de, el, en, es, fr, hi, it, ja, nl, pl, pt, ru, sw, th, tr, ur, vi, zh
Architecture	XLM-RoBERTa-base (multilingual encoder, 100-language pretraining) with a classification head
Quantization	q8 (8-bit) ONNX
Backend	WebAssembly (runs anywhere; no WebGPU needed)
Download	~270 MB, cached after first load
License	MIT
Web APIs	WebAssembly, Web Workers — all Baseline widely available

Run it

Type or paste text (or try a sample). Detection runs as you stop typing.

Text

See inside — the probability distribution

The model doesn't just pick a language; it produces a softmax probability over all 20. A confident detection puts almost all the mass on one bar. On short or mixed text the mass spreads — that spread is the model telling you it's unsure, and it's genuinely useful signal. Below are the top languages for the current text. Try the Code-switch sample and watch two languages share the mass.

Enter some text above to see the distribution.

Why it matters

Language identification is the quiet first step in almost every multilingual text pipeline. Before you can translate, moderate, route, or index text, you have to know what language it's in — and you often have to do it on a single sentence, a search query, or a chat message. Running it on-device means you can branch instantly with zero latency and without sending user text to a server.

Route to the right model — pick the correct translator, spell-checker, or moderation model per message.
Pre-filter a dataset — keep only the languages you support, or split a mixed corpus by language before processing.
Set the UI language — infer a user's language from what they type and adapt on the fly.
Tag user-generated content — label reviews, comments, or support tickets so the right team or workflow picks them up.

Use cases

Basics

Detect the language of a sentence, with the full probability distribution.

Practical

Route incoming messages to the right per-language workflow, live.

Wild

Short, ambiguous, and code-switched snippets — watch the confidence honestly wobble.

Multi-model

Detect the language, then route to the matching Marian translator into English.

How the API works

It's a plain text-classification pipeline; ask for all labels to get the full distribution:

import { pipeline } from "@huggingface/transformers";

const detect = await pipeline(
  "text-classification",
  "onnx-community/xlm-roberta-base-language-detection-ONNX",
  { dtype: "q8" },
);

const out = await detect("Bonjour, comment allez-vous ?", { top_k: 20 });
// → [{ label: "fr", score: 0.99 }, { label: "ru", score: 0.003 }, … ]
// out[0].label is the detected language code; the rest is the confidence distribution.

The model outputs a fixed label set (the 20 ISO codes above). Passing top_k: 20 returns every language's probability, sorted — that's what powers the "see inside" bars. All inference runs in a Web Worker so detection never janks the page.