Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models

Luca Eyring1, Shyamgopal Karthik1,2, Alexey Dosovitskiy3, Nataniel Ruiz*4, Zeynep Akata*1
1TU Munich/MCML/Helmholtz Munich, 2University of Tübingen, 3Inceptive, 4Google Research
Paper arXiv Code
Noise Hypernetworks Concept

Noise Hypernetworks optimizes diffusion models to desired downstream rewards by amortizing test-time compute into post-training.

Abstract

The new paradigm of test-time scaling has yielded remarkable breakthroughs in Large Language Models (LLMs) and in generative vision models, allowing models to allocate additional computation during inference to effectively tackle increasingly complex problems. Despite the improvements of this approach, an important limitation emerges: the substantial increase in computation time makes the process slow and impractical for many applications. Given the success of this paradigm and its growing usage, we seek to preserve its benefits while eschewing the inference overhead. In this work we propose one solution to the critical problem of integrating test-time scaling knowledge into a model during post-training. Specifically, we replace reward guided test-time noise optimization in diffusion models with a Noise Hypernetwork that modulates initial input noise. We propose a theoretically grounded framework for learning this reward-tilted distribution for distilled generators, through a tractable noise-space objective that maintains fidelity to the base model while optimizing for desired characteristics. We show that our approach recovers a substantial portion of the quality gains from explicit test-time optimization at a fraction of the computational cost.

Comparison with other methods

Comparison with other methods Teaser Sana Geneval results

Algorithm

Algorithm

Redness reward

HyperNoise is able to learn the reward-tilted distribution whereby the generated images are more red, while still staying within the natural distribution of images. In contrast, direct reward fine-tuning ends up trivially overfitting to the reward while worsening the quality of the generated images.

Redness Reward Illustration Redness Tradeoff

BibTeX

@article{eyring2025noise,
  author    = {Eyring, Luca and Karthik, Shyamgopal and Dosovitskiy, Alexey and Ruiz, Nataniel and Akata, Zeynep},
  title     = {Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models},
  journal   = {arXiv preprint},
  year      = {2025},
}