FuseReg：缓解RAEs中的重建-生成差距

表示自编码器将来自多个编码器层的特征融合到一个共享的潜在空间中。

然而，重建和生成倾向于层次结构的不同部分：

解码器：偏好浅层、富含像素的特征。

DiT：偏好更深、更具结构化的语义特征。

这导致了重建与生成之间的不匹配。

FuseReg

FuseReg在训练期间用随机层子集采样取代固定的层融合。

对于采样的子集（S），

z_S = \frac{1}{|S|}\sum_{k\in S} h_k.

归一化采样保持潜在均值不变：

E[z_S | x] = \bar h.

在线性平方损失代理下，我们进一步表明，随机子集采样明确地正则化了跨层分歧。

简而言之，FuseReg discourages 模型不要过于依赖任何特定的层组合。

对层融合具有鲁棒性

单个FuseReg解码器在全量、稀疏甚至单层输入上均保持有效，而固定融合解码器在其训练融合范围之外会严重退化。

生成能力的提升

FuseReg可以独立应用于解码器和DiT。

仅替换解码器：无引导 gFID 3.01 → 2.21

DiT-Base，联合FuseReg：无引导 gFID 13.96 → 9.93

DiT-XL：无引导 gFID 2.91 → 2.38

这一趋势在不同预训练编码器家族中也同样成立。

一个解码器，多种层组合

相同的FuseReg解码器可以在不重新训练的情况下从不同的层子集进行重建，包括稀疏和单层输入。

发生了什么变化？

FuseReg使重建信息在编码器深度上分布得更加均匀：

对单个层的依赖性降低；

来自单层的重建能力增强；

在层组合变化下表示更加稳定。

FuseReg不再寻找单一的最优融合方式，而是训练模型在各种层融合的分布中保持鲁棒性。

FuseReg: Mitigating the Reconstruction–Generation Gap in RAEs

Representation Autoencoders fuse features from multiple encoder layers into a shared latent space.

However, reconstruction and generation prefer different parts of the hierarchy:

Decoder: prefers shallow, pixel-rich features.

DiT: prefers deeper, more structured semantic features.

This creates a reconstruction–generation mismatch.

FuseReg

FuseReg replaces fixed layer fusion with random layer-subset sampling during training .

For a sampled subset (S),

z S = 1 ∣ S ∣ ∑ k ∈ S h k .
z_S = \frac{1}{|S|}\sum_{k\in S} h_k.
z S ​ = ∣ S ∣ 1 ​ k ∈ S ∑ ​ h k ​ .

The normalized sampling keeps the latent mean unchanged:

E [ z S ∣ x ] = h ˉ .
\mathbb{E}[z_S \mid x] = \bar h.
E [ z S ​ ∣ x ] = h ˉ .

Under a linear squared-loss surrogate, we further show that random subset sampling explicitly regularizes cross-layer disagreement .

In short, FuseReg discourages models from relying too strongly on any particular layer composition.

Robust across layer fusions

A single FuseReg decoder remains effective across full, sparse, and even single-layer inputs, while fixed-fusion decoders degrade strongly outside their training fusion.

Generation improvements

FuseReg can be applied independently to the decoder and DiT.

Decoder replacement only: unguided gFID 3.01 → 2.21

DiT-Base, joint FuseReg: unguided gFID 13.96 → 9.93

DiT-XL: unguided gFID 2.91 → 2.38

The same trend also holds across different pretrained encoder families.

One decoder, many layer combinations

The same FuseReg decoder can reconstruct from different layer subsets without retraining, including sparse and single-layer inputs.

What changes?

FuseReg distributes reconstruction information more evenly across encoder depth:

lower dependence on individual layers;

stronger reconstruction from single layers;

more stable representations under changing layer compositions.

Instead of searching for one optimal fusion, FuseReg trains the model to remain robust across a distribution of layer fusions.