← All models

zero-shot-image-classification · vision + language · WASM · ~90 MB

CLIP — classify an image against any labels you invent

CLIP learned a shared space for pictures and words. Give it an image and a handful of labels you make up on the spot — a sleeping cat, a rainy street, cozy — and it scores how well each one matches. No training, no categories baked in, nothing leaves your device.

At a glance

ModelXenova/clip-vit-base-patch16
Task / pipelinezero-shot-image-classification
ArchitectureViT-B/16 vision encoder + text transformer → shared 512-d space
Quantizationq8 (8-bit) ONNX
BackendWebAssembly (runs anywhere; no WebGPU needed)
Download~90 MB, cached after first load
LicenseMIT (OpenAI CLIP)
Web APIsWebAssembly, Web Workers, FileReader — all Baseline widely available

Run it

Pick or drop an image, type comma-separated labels, and hit Classify. The bars are a softmax over CLIP's similarity between the image and each label.

Drop an image here, or click to choose
Two cats sleeping on a pink couch A cozy sunlit bedroom with a bookshelf A riverside at golden hour with a swan
Selected image preview

See inside

Under the hood CLIP turns your image into one 512-number vector and each label into its own 512-number vector, then measures the cosine similarity (the angle) between the image vector and every label vector. Those similarities are scaled (×≈100, CLIP's learned temperature) into logits, and a softmax turns the logits into the percentages above. Same picture, different words → different winner. That is the whole trick.

Run a classification to see the raw numbers.

Why it matters

Every other image classifier is stuck with the categories it was trained on. CLIP isn't: the label set is an input, decided at runtime, in the browser, for free. That flips a lot of problems from "collect a dataset and fine-tune a model" into "write a list of words."

Use cases

Basics

One image, your labels — feel zero-shot classification first-hand.

Practical

Auto-tagging & content moderation against a fixed label set.

Wild

Live webcam "vibe check" — classify the camera against moods in real time.

How the API works

The whole thing is three lines with Transformers.js. The label list is just an argument:

import { pipeline } from "@huggingface/transformers";

const classify = await pipeline(
  "zero-shot-image-classification",
  "Xenova/clip-vit-base-patch16",
); // downloads + caches ~90 MB once

const output = await classify(imageURL, ["a cat", "a dog", "a city street"]);
// → [{ label: "a cat", score: 0.94 }, { label: "a city street", score: 0.05 }, …]
//   score is the softmax over CLIP's image↔label cosine similarities

This page goes one level deeper for "See inside": it reuses the pipeline's own model, processor, and tokenizer to read the raw logits_per_image and the image/text embedding tensors from a single forward pass — the same call, just keeping the intermediate numbers. Inference runs in a Web Worker so the page never janks.

References