HiUni

Harmonizing Understanding and Generation in Shared Visual Space

ERNIE Team

Baidu Inc.

Created with HiUni Text to imageEditing

Can a pretrained VLM become a unified model without splitting its visual pathway?

Unified multimodal models (UMMs) aim to support visual understanding and generation within a single framework. However, prevailing approaches often separate their visual representations or backbone computation to accommodate the different requirements of the two tasks, which limits the degree of unification.

We present HiUni, which revisits this design choice and explores whether a pretrained VLM can be directly extended into a unified model with a high degree of sharing. HiUni reuses the same language-aligned ViT and dense Transformer backbone for both capabilities, while confining task-specific computation to the output side.

Such a design introduces two challenges: direct generation in pixel space is difficult to optimize, and the visual compression and semantic abstraction in pretrained VLMs may discard fine spatial details required for synthesis. HiUni introduces a Pixel Diffusion Head tailored to direct pixel-space optimization and a lightweight HiGate pathway that provides fine-grained spatial details near the output stage.

Experiments show that HiUni achieves a similar average understanding score to the pretrained VLM while attaining competitive performance on image generation and editing. Controlled comparisons further demonstrate that a single dense backbone can effectively support both capabilities without task-specific backbone computation, highlighting a simpler path toward unified multimodal modeling.

Share the visual pathway, specialize only the output

Existing UMMs either keep separate encoders for understanding and generation, route the two tasks through different experts (MoT), or cascade a VAE with a ViT. HiUni instead reuses the VLM's own visual pathway end to end: the same ViT tokens enter the same dense backbone, and only the generation head and HiGate are new.

Figure 1.Comparison of UMM designs. (a) Bagel uses separate encoders and MoT. (b) TUNA cascades a VAE and a ViT. (c) HiUni reuses the VLM visual pathway, and HiGate supplies pixel details only to the generation head.
HiUni model architecture: shared ViT and dense backbone, Pixel Diffusion Head and HiGate
Figure 2.Overview of HiUni. Understanding, text-to-image generation and editing share one ViT and one dense backbone. Backbone features are decoded into RGB patches by the Pixel Diffusion Head; HiGate injects the noisy pixels through a token-wise gate.
Inference Process
01Understanding
AaQuestion
▦Input image
→
ViTLanguage-aligned ViTShared
→
ΣDense TransformerShared
→
LMLanguage head
→
AText answer
02Generation
AaPrompt + timestep
▦Noisy RGB
→
ViTLanguage-aligned ViTShared
→
ΣDense TransformerShared
→
GHiGate
DiTPixel Diffusion Head
→
RGBGenerated image

Understanding is preserved, generation and editing are competitive

Model size is reported as understanding (backbone) parameters + generation-specific parameters. ★ marks officially reported results.

Table 1.Image understanding

HiUni is initialized from Qwen3-VL-INS-4B. Gains appear on MathVista and HallusionBench, declines on MMBench and OCRBench; understanding remains at the starting model's level across seven benchmarks (75.56 for HiUni vs 75.61 for Qwen3-VL-INS-4B).

ModelSizeMathVistaMiniMMBenchENRealWorldQAMMStarHallusionBenchAI2DOCRBenchOmniDocBench ↓
Understanding-only models
Qwen3-VL-INS★4B73.785.170.969.857.684.188.10.24
Qwen3-VL-THK★4B79.586.773.273.264.184.980.80.23
Qwen3-VL-THK4B76.586.274.170.365.884.579.60.22
Unified multimodal models
Harmon★1.5B–65.549.835.3––11.2–
Cheers★1.5B50.570.460.950.9––58.4–
InternVL-U2B + 1.7B53.971.756.953.532.476.883.9★0.83
Janus-Pro7B44.779.2★58.048.336.471.359.00.81
Show-o27B51.879.3★64.7★56.636.278.6★32.40.84
TUNA★7B––66.161.2–79.374.3–
Bagel7B + 7B73.1★85.0★74.467.651.889.280.80.43
SenseNova-U18B + 8B84.2★90.3★73.6★78.3★67.8★91.7★82.1★0.28
HiUni4B + 0.9B79.080.772.270.566.086.074.50.24
Table 2.Text-to-image generation

Qwen-Image-Bench (Chinese and English prompts), OneIG overall and LongText. Bold marks the best result among unified models, underline the second best. HiUni has the best Chinese overall score among unified models.

ModelSizeQwen-Image-BenchZHQwen-Image-BenchENOneIGLongText
Qual.Aesth.Align.Real.Creat.OverallQual.Aesth.Align.Real.Creat.OverallZHENZHEN
Generation-only models
GLM-Image9B + 7B49.2650.6447.9044.6945.2348.1949.8649.9847.4944.2544.6747.860.511★0.528★0.979★0.952★
Qwen-Image★20B48.4452.2550.7243.1647.3049.2348.4551.1850.0443.4545.3748.480.5480.5390.9460.943
HiDream-O18B48.6149.7549.5343.9740.9647.5848.8449.6949.5343.8041.0547.590.5180.5210.978★0.979★
Unified multimodal models
InternVL-U2B + 1.7B42.3443.3840.3038.1334.3640.5443.9243.0341.0938.8732.3540.820.500★0.500★0.860★0.738★
Bagel7B + 7B46.1145.8840.2739.7432.6942.1545.9047.2043.4840.3034.8343.450.3700.3610.3100.373
SenseNova-U1★8B + 8B45.7947.0747.7842.5343.8845.9947.3050.3050.5843.1346.1548.280.5350.5490.9620.979
UniSpace8B + 8B47.2245.0248.5446.1940.4946.0547.4446.5448.9447.5443.8447.050.533★0.561★––
HiUni4B + 0.9B46.7350.0050.4543.0945.6247.9847.6350.5248.7843.7044.1547.700.5240.5380.9630.954
Table 3.Image editing

ImgEdit (0–5) and GEdit (0–10, VIEScore protocol), both judged by GPT-5.2. HiUni ranks second among unified models on ImgEdit and on both GEdit splits.

ModelSizeImgEditGEditCNGEditEN
AddAdjustExtractReplaceRemoveBackg.StyleHybridActionOverallSCPQOSCPQO
Editing-only models
Qwen-Image-Edit-251120B4.224.243.344.373.983.614.393.254.694.018.256.427.198.206.467.17
LongCat-Image-Edit6B4.154.243.394.473.953.574.283.394.704.028.136.447.158.056.447.10
Qwen-Image-2.17B4.174.242.164.484.133.754.273.152.893.697.606.456.737.916.406.92
Unified multimodal models
InternVL-U2B + 1.7B3.943.712.424.083.343.454.032.284.073.486.025.815.746.055.805.74
Bagel7B + 7B3.493.361.173.692.823.063.802.593.823.097.375.716.317.315.686.27
SenseNova-U18B + 8B3.823.962.444.203.113.594.473.094.203.658.196.207.058.046.146.94
UniSpace8B + 8B4.064.013.004.323.993.524.073.164.253.827.725.846.577.885.876.68
HiUni4B + 0.9B3.933.952.724.053.473.324.343.223.953.667.875.886.707.936.006.81
Table 4.Reasoning-based generation (WISE)

A chain-of-thought variant fine-tuned for 10K steps on UniReason-derived data, with no extra backbone or generation module.

ModelOverallCulturalTimeSpaceBiologyPhysicsChemistry
DeepGen 1.0★0.570.650.410.720.390.590.45
Bagel w/ CoT★0.630.780.630.570.380.550.51
SenseNova-U1-8B★0.680.690.600.740.630.720.69
SenseNova-U1-8B w/ CoT★0.790.800.740.860.760.780.80
Qwen-Image-2.0★0.800.820.650.900.790.800.75
GPT-Image-1.5★0.830.890.690.880.800.760.78
HiUni w/ CoT0.750.810.680.780.630.740.78
Bold: best among unified modelsUnderline: second best★: officially reported

Dense backbone vs. Mixture-of-Transformers

At the same total parameter budget, HiUni-Dense-4B beats HiUni-MoT-2B (two 2B branches) on all six understanding benchmarks (65.04 vs 59.23 average), at the cost of a ~9% lower OneIG average. Scaling the dense backbone from 2B to 4B gives +11.1% understanding and +7.9% OneIG.

Comparison between shared dense and task-routed MoT backbones under controlled settings. HiUni-Dense-2B and HiUni-MoT-2B differ only in backbone design while HiUni-Dense-4B evaluates a larger shared backbone under a similar total parameter budget. HiUni-Und-4B and HiUni-Gen-4B serve as understanding-only and generation-only baselines, respectively.
ModelUnderstandingGeneration
NameSizeMMBenchENRealWorldQAMMStarHallusionBenchAI2DOCRBenchAvg.OneIG Avg.
HiUni-Dense-4B4B+0.5B72.6065.4953.9348.9677.9571.3065.040.314
HiUni-Dense-2B2B+0.5B64.9460.3946.6738.7070.4770.1058.550.291
HiUni-MoT-2B4B+0.5B67.9763.2747.0035.3371.7670.0259.230.345
HiUni-Und-4B4B72.5263.7953.3346.9677.8573.0064.58–
HiUni-Gen-4B4B+0.5B–––––––0.390
Training curves comparing the dense backbone with the MoT backbone
Training trajectories during joint pretraining after warmup: average understanding and OneIG scores.

Where should HiGate take its input?

Both residual sources improve generation over removing the gate; raw pixel features work best (OneIG 0.267 → 0.284 with ViT features → 0.314 with pixels), while the understanding average barely moves (65.04–65.17). The learned gate stays weak at high noise and strengthens near the clean image, and early checkpoints rely on it more.

HiGate inputMMBRWQAMMStarHallBAI2DOCRBOneIG
Pixel Features72.6065.4953.9348.9677.9571.300.314
ViT Features74.6167.4553.0749.0577.8269.000.284
No Gate72.9165.8854.1346.7077.7573.000.267
HiGate injection strength across diffusion timesteps and training checkpoints
HiGate injection strength across diffusion timesteps and training checkpoints, normalized by the global maximum.

Cross-task interaction

Adding image-editing supervision raises the understanding average from 62.64 to 64.49 (gains on four of six benchmarks), while OneIG dips slightly, a mix of positive transfer and mild interference.

MMBRWQAMMStarHallBAI2DOCRBAvg.OneIG
w/o Editing data69.5864.5850.2048.4372.5470.5062.640.538
w/ Editing data72.2966.8055.0043.8379.2469.8064.490.528

Why no AdaLN in the head

With block-wise AdaLN, the Pixel Diffusion Head frequently produced abrupt loss spikes in pixel space. Replacing it with a <time> token and a single final modulation gives smoother training. The two runs come from different development stages, so this is a qualitative comparison.

Diffusion loss curves with and without AdaLN in the Pixel Diffusion Head
Diffusion training loss with AdaLN (left) and without AdaLN (right).

Citation

@article{hu2026hiuni,
  title         = {HiUni: Harmonizing Understanding and Generation in Shared Visual Space},
  author        = {Hu, Jiahao and Du, Jingdong and Zhang, Yichen and Song, Yi and
                   Zhang, Zhenyu and Niu, Jiaxin and Li, Ruihang and Hao, Xiangzhao and
                   Zhang, Yiqian and Huang, Yong and Zhang, Zefeng and Gu, Zheyuan and
                   Wang, Shuohuan},
  journal       = {arXiv preprint arXiv:2611.26107},
  year          = {2026},
  eprint        = {2611.26107},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2611.26107}
}

HiUni is built on Qwen3-VL. Training uses Megatron-Bridge, Megatron-LM and the NeMo container, with attention masks from FlashMask. The x-prediction objective follows JiT. We thank the authors of these projects and of the open benchmarks used in our evaluation.