document-question-answering · document → answer (VQA) · WASM · ~215 MB
Donut reads documents the OCR-free way: there's no separate text-extraction step. A
vision transformer looks at the whole page and a decoder writes the answer to your question
directly — What is the total? → $65.40 — entirely on your device.
| Model | Xenova/donut-base-finetuned-docvqa |
|---|---|
| Task / pipeline | document-question-answering |
| Architecture | Swin Transformer image encoder → BART-style text decoder (VisionEncoderDecoder). No OCR engine. |
| Input | A document image + a natural-language question |
| Output | A short free-text answer, generated token by token |
| Quantization | q8 (8-bit) ONNX — separate encoder + decoder graphs |
| Backend | WebAssembly (runs anywhere; no WebGPU needed) |
| Download | ~215 MB, cached after first load |
| License | MIT (naver-clova-ix/donut-base-finetuned-docvqa) |
| Web APIs | WebAssembly, Web Workers, FileReader — all Baseline widely available |
Pick a document, type a question (or tap a preset), and hit Ask. The answer streams in as the decoder generates it — that's the real model, not an animation.
Donut turns "answer this question" into text generation. Your question is
wrapped in a fixed task template and fed to the decoder as its starting prompt; the model then
generates the answer tokens that come after <s_answer>, each conditioned on the
image and everything decoded so far. Below is the exact prompt Donut receives and
the answer streaming out, token by token with timing. This is the real output, no
OCR step in sight.
Ask a question to see the prompt Donut builds.
Most document automation is a brittle chain: OCR the page, then hunt through the raw text with regexes and rules. Donut skips the OCR step — you ask for what you want in plain language and it answers, which makes it robust to layout and doing it locally makes it free, private, and offline-capable.
Ask one question about a document and watch the answer stream in.
Extract a whole field set — total, date, number — from a receipt or invoice at once.
Have a back-and-forth "chat" with a document — ask follow-ups, build a Q&A log.
Donut answers → M2M100 translates the answer into another language.
The base call is a couple of lines with Transformers.js:
import { pipeline } from "@huggingface/transformers";
const docqa = await pipeline(
"document-question-answering",
"Xenova/donut-base-finetuned-docvqa",
); // downloads + caches ~215 MB once
const out = await docqa(documentImageURL, "What is the total due?", {
max_new_tokens: 64,
});
// → [{ answer: "$65.40" }]
Under the hood Donut wraps your question in a task prompt —
<s_docvqa><s_question>…</s_question><s_answer> — and generates
the answer that follows. For "See inside", this page passes a TextStreamer into the
same call so the answer renders as it forms and each token's arrival time is recorded, all off the
main thread in a Web Worker.