Introduction
The Clarifai platform has developed considerably. Earlier generations of the platform relied on many small, task-specific fashions for visible classification, detection, OCR, textual content classification and segmentation. These legacy fashions have been constructed on older architectures that have been delicate to area shift, required separate coaching pipelines and didn’t generalize nicely exterior their authentic circumstances.
The ecosystem has moved on. Trendy massive language fashions and vision-language fashions are skilled on broader multimodal information, cowl a number of duties inside a single mannequin household and ship extra steady efficiency throughout completely different enter sorts. As a part of the platform improve, we’re standardizing round these newer mannequin sorts.
With this replace, a number of legacy task-specific fashions are being deprecated and can now not be obtainable. Their performance continues to be totally supported on the platform, however is now supplied by extra succesful and basic mannequin households. Compute Orchestration manages scheduling, scaling and useful resource allocation for these fashions in order that workloads behave constantly throughout open supply and customized mannequin deployments.
This weblog outlines the core job classes supported at present, the really useful fashions for every and easy methods to use them throughout the platform. It additionally clarifies which older fashions are being retired and the way their capabilities map to the present mannequin households.
Really helpful Fashions for Core Imaginative and prescient and NLP Duties
Visible Classification and Recognition
Visible classification and recognition contain figuring out objects, scenes and ideas in a picture. These duties energy product tagging, content material moderation, semantic search, retrieval indexing and basic scene understanding.
Trendy vision-language fashions deal with these duties nicely in zero-shot mode. As an alternative of coaching separate classifiers, you outline the taxonomy within the immediate and the mannequin returns labels instantly, which reduces the necessity for task-specific coaching and simplifies updates.
Fashions on the platform fitted to visible classification, recognition and moderation
The fashions under supply robust visible understanding and carry out nicely for classification, recognition, idea extraction and picture moderation workflows, together with sensitive-safety taxonomy setups.
MiniCPM-o 2.6
A compact VLM that handles pictures, video and textual content. Performs nicely for versatile classification workloads the place pace, price effectivity and protection must be balanced.
Qwen2.5-VL-7B-Instruct
Optimized for visible recognition, localized reasoning and structured visible understanding. Robust at figuring out ideas in pictures with a number of objects and extracting structured data.
Moderation with MM-Poly-8B
A big portion of real-world visible classification work entails moderation. Many buyer workloads are constructed round figuring out whether or not a picture is secure, delicate or banned in accordance with a particular coverage. In contrast to basic classification, moderation requires strict taxonomy, conservative thresholds and constant rule-following. That is the place MM-Poly-8B is especially efficient.
MM-Poly-8B is Clarifai’s multimodal mannequin designed for detailed, prompt-driven evaluation throughout pictures, textual content, audio and video. It performs nicely when the classification logic must be specific and tightly managed. Moderation groups typically depend on layered directions, examples and edge-case dealing with. MM-Poly-8B helps this sample instantly and behaves predictably when given structured insurance policies or rule units.
Key capabilities:
-
Accepts picture, textual content, audio and video inputs
-
Handles detailed taxonomies and multi-level resolution logic
-
Helps example-driven prompting
-
Produces constant classifications for safety-critical use instances
-
Works nicely when the moderation coverage requires conservative interpretation and bias towards security
As a result of MM-Poly-8B is tuned to comply with directions faithfully, it’s fitted to moderation eventualities the place false negatives carry increased danger and fashions should err on the facet of warning. It may be prompted to categorise content material utilizing your coverage, determine violations, return structured reasoning or generate confidence-based outputs.
If you wish to show a moderation workflow, you’ll be able to immediate the mannequin with a transparent taxonomy and ruleset. For instance:
“Consider this picture in accordance with the classes Secure, Suggestive, Express, Drug and Gore. Apply a strict security coverage and classify the picture into probably the most applicable class.”
For extra superior use instances, you’ll be able to present the mannequin with an in depth set of moderation guidelines, resolution standards and examples that outline how every class must be utilized. This lets you confirm how mannequin behaves beneath stricter, policy-driven circumstances and the way it may be built-in into production-grade moderation pipelines.
MM-Poly-8B is accessible on the platform and can be utilized by the Playground or accessed programmatically by way of the OpenAI-compatible API.
Notice: If you wish to entry the above fashions like MiniCPM-o-2.6 and Qwen2.5-VL-7B-Instruct instantly, you’ll be able to deploy them to your individual devoted compute utilizing the Platform and entry them by way of API identical to some other mannequin.
The best way to entry these fashions
All fashions described above might be accessed by Clarifai’s OpenAI-compatible API. Ship a picture and a immediate in a single request and obtain both plain textual content or structured JSON, which is helpful while you want constant labels or need to feed the outcomes into downstream pipelines.
For particulars on structured JSON output, test the documentation right here.
Coaching your individual classifier (fine-tuning)
In case your utility requires domain-specific labels, industry-specific ideas or a dataset that differs from basic internet imagery, you’ll be able to prepare a customized classifier utilizing Clarifai’s visible classification templates. These templates present configurable coaching pipelines with adjustable hyperparameters, permitting you to construct fashions tailor-made to your use case.
Out there templates embrace:
-
MMClassification ResNet 50 RSB A1
-
Clarifai InceptionBatchNorm
-
Clarifai InceptionV2
-
Clarifai ResNeXt
-
Clarifai InceptionTransferEmbedNorm
You may add your dataset, configure hyperparameters and prepare your individual classifier by the UI or API. Try the Superb-tuning Information on the platform.
Doc Intelligence and OCR
Doc intelligence covers OCR, structure understanding and structured subject extraction throughout scanned pages, types and text-heavy pictures. The legacy OCR pipeline on the platform relied on language-specific PaddleOCR variants. These fashions have been slender in scope, delicate to formatting points and required separate upkeep for every language. They’re now being decommissioned.
Fashions being decommissioned
These fashions have been single-language engines with restricted robustness. Trendy OCR and multimodal programs assist multilingual extraction by default and deal with noisy scans, blended codecs and paperwork that mix textual content and visible parts with out requiring separate pipelines.
Open-source OCR mannequin on the platform
DeepSeek OCR
DeepSeek OCR is the first open-source possibility. It helps multilingual paperwork, processes noisy scans fairly nicely and may deal with structured and unstructured paperwork. Nevertheless, it’s not excellent. Benchmarks present inconsistent accuracy on messy handwriting, irregular layouts and low-resolution scans. It additionally has enter dimension constraints that may restrict efficiency on massive paperwork or multi-page flows. Whereas it’s stronger than the sooner language-specific engines, it’s not the most suitable choice for high-stakes extraction on advanced paperwork.
Third-party multimodal fashions for OCR-style duties
The platform additionally helps a number of multimodal fashions that mix OCR with visible reasoning. These fashions can extract textual content, interpret tables, determine key fields and summarize content material even when construction is advanced. They’re extra succesful than DeepSeek OCR, particularly for lengthy paperwork or workflows requiring reasoning.
Gemini 2.5 Professional
Handles text-heavy paperwork, receipts, types and sophisticated layouts with robust multimodal reasoning.
Claude Opus 4.5
Performs nicely on dense, advanced paperwork, together with desk interpretation and structured extraction.
Claude Sonnet 4.5
A sooner possibility that also produces dependable subject extraction and summarization for scanned pages.
GPT-5.1
Reads paperwork, extracts fields, interprets tables and summarizes multi-section pages with robust semantic accuracy.
Gemini 2.5 Flash
Light-weight and optimized for pace. Appropriate for widespread types, receipts and easy doc extraction.
These fashions carry out nicely throughout languages, deal with advanced layouts and perceive doc context. The tradeoffs matter. They’re closed-source, require third-party inference and are dearer to function at scale in comparison with an open-source OCR engine. They are perfect for high-accuracy extraction and reasoning, however not at all times cost-efficient for big batch OCR workloads.
The best way to entry these fashions
Utilizing the Playground
Add your doc picture or scanned web page within the Playground and run it with DeepSeek OCR or any of the multimodal fashions listed above. These fashions return Markdown-formatted textual content, which preserves construction resembling headings, paragraphs, lists or table-like formatting. This makes it simpler to render the extracted content material instantly or course of it in downstream doc workflows.
Utilizing the API (OpenAI-compatible)
All these fashions are additionally accessible by Clarifai’s OpenAI-compatible API. Ship the picture and immediate in a single request, and the mannequin returns the extracted content material in Markdown. This makes it simple to make use of instantly in downstream pipelines. Try the detailed information on accessing DeepSeek OCR by way of the API.
Textual content Classification and NLP
Textual content classification is utilized in moderation, subject labeling, intent detection, routing, and broader textual content understanding. These duties require fashions that comply with directions reliably, generalize throughout domains, and assist multilingual enter while not having task-specific retraining.
Instruction-tuned language fashions make this a lot simpler. They’ll carry out classification in a zero-shot method, the place you outline the courses or guidelines instantly within the immediate and the mannequin returns the label while not having a devoted classifier. This makes it simple to replace classes, experiment with completely different label units and deploy the identical logic throughout a number of languages. In case you want deeper area alignment, these fashions can be fine-tuned.
Under are the some stronger fashions on the platform for textual content classification and NLP:
-
Gemma 3 (12B)
A current open mannequin from Google, tuned for effectivity and high-quality language understanding. Robust at zero-shot classification, multilingual reasoning, and following immediate directions throughout diverse classification duties. -
MiniCPM-4 8B
A compact, high-performing mannequin constructed for instruction following. Works nicely on classification, QA, and general-purpose language duties with aggressive efficiency at decrease latency. -
Qwen3-14B
A multilingual mannequin skilled on a variety of language duties. Excels at zero-shot classification, textual content routing, and multi-language moderation and subject identification.
Notice: If you wish to entry the above open-source fashions like Gemma 3, MiniCPM-4 or Qwen3 instantly, you’ll be able to deploy them to your individual devoted compute utilizing the Platform and entry them by way of API identical to some other mannequin on the platform.
There are additionally many further third-party and open-source fashions obtainable within the Neighborhood part, together with GPT-5.1 household variants, Gemini 2.5 Professional, and several other high-quality fashions. You may discover these primarily based in your scale, and domain-specific wants.
Customized Mannequin Deployment
Along with the fashions listed above, the platform additionally enables you to deliver your individual fashions or deploy open supply fashions from the Neighborhood utilizing Compute Orchestration (CO). That is useful while you want a mannequin that isn’t already obtainable on the platform, or while you need full management over how a mannequin runs in manufacturing.
CO handles the operational particulars required to serve fashions reliably. It containerizes fashions routinely, applies GPU fractioning so a number of fashions can share the identical {hardware}, manages autoscaling and makes use of optimized scheduling to scale back latency beneath load. This allows you to scale customized or open supply fashions while not having to handle the underlying infrastructure.
CO helps deployment on a number of cloud environments resembling AWS, Azure and GCP, which helps keep away from vendor lock-in and provides you flexibility in how and the place your fashions run. Try the information right here on importing and deploying your individual customized fashions.
Conclusion
The mannequin households outlined on this information symbolize probably the most dependable and scalable option to deal with visible classification, detection, moderation, OCR and text-understanding workloads on the platform at present. By consolidating these duties round stronger multimodal and language-model architectures, builders can keep away from sustaining many slender, task-specific legacy fashions and as a substitute work with instruments that generalize nicely, assist zero-shot directions and adapt cleanly to new use instances.
You may discover further open supply and third-party fashions within the Neighborhood part and use the documentation to get began with the Playground, API or fine-tuning workflows. In case you need assistance planning a migration or deciding on the best mannequin to your workload, you’ll be able to attain out to us on Discord or contact our assist crew right here.
