NVIDIA PixelUMM releases pixel-space multimodal model

// 1h agoMODEL RELEASE

NVIDIA PixelUMM releases pixel-space multimodal model

PixelUMM is an encoder-free NVIDIA model that processes raw image patches and video tubelets directly, supporting image and video understanding plus generation through a shared Qwen3-8B-based Transformer. The checkpoint is available for non-commercial research and evaluation. [Project page](https://nv-tlabs.github.io/PixelUMM/) [Model card](https://huggingface.co/nvidia/PixelUMM)

// ANALYSIS

PixelUMM is a compelling architecture experiment, but its raw-pixel simplicity comes with substantial compute and quality tradeoffs.

–Removes both the vision encoder and VAE, using a shared pixel-space interface for understanding and generation.
–Handles text-to-image, image-conditioned text, video understanding, and video generation in one checkpoint.
–Uses 16×16 image patches, four-frame video tubelets, and separate understanding/generation experts with shared attention.
–Developers should expect research-grade limitations, including prompt-following issues, temporal inconsistencies, patch artifacts, demanding NVIDIA GPU requirements, and non-commercial model weights. [Technical details](https://github.com/nv-tlabs/PixelUMM)

// TAGS

pixelummllmmultimodalvisionimage-genvideo-genopen-sourceresearch

DISCOVERED

1h ago

2026-10-04

PUBLISHED

1h ago

2026-10-04

RELEVANCE

9/ 10

AUTHOR

AI Search

// KEEP READING

More AI developer news from the feed

EXPLORE FULL FEED

MODEL1h ago

Whistle Packs Multilingual Speech-to-Text into 16.9 MB.

Cactus Compute released Whistle, an open CPU speech-recognition model that transcribes seven languages, supports 30-second inputs, word timestamps, speech embeddings, and on-device processing. It shares the Needle runtime and weighs just 16.9 MB. [Cactus launch post](https://www.cactuscompute.com/blog/whistle)

RESEARCH1h ago

TactileStep Makes Humanoid Landings Softer

Tsinghua’s TactileStep adds sole-pressure feedback to Unitree G1 locomotion policies, using contact force, center of pressure, and loaded-area signals to regulate touchdown and stance. Tests across varied terrain report up to 48.8% lower touchdown force and 23.8% greater stance contact area. [Official project page](https://tactilestep.github.io/) · [Paper](https://arxiv.org/abs/2609.28959)

MODEL1h ago

Phonon-2 Releases 164MB Open ASR

Fermion Research released Phonon-2, an open English speech-recognition model with a 164 MB download, 5.21% average word error rate, and 174× realtime inference on an M5 MacBook Air. It runs locally across macOS, Linux, Windows, and NVIDIA GPUs. [Announcement](https://www.fermionresearch.com/research/phonon-2/)