Accepted at ICLR 2026 Adaptive compute for LLMs

LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation

LoopFormer trains looped Transformers on variable-length trajectories so the same model can reason under different compute budgets, stay stable across shortcut schedules, and keep improving as budget grows.

Ahmadreza Jeddi^1,2,3, Marco Ciccone^1,2, Babak Taati^1,2,3

¹University of Toronto • ²Vector Institute • ³University Health Network

Paper OpenReview Code

Models

Method snapshot

Time-aware shortcuts make looped depth elastic

Trajectory conditioning

Each loop step sees both internal time t and step size Δt.

Shortcut consistency

Short routes are aligned to full routes so reduced-budget runs remain informative.

Budget-conditioned inference

Choose the compute budget at test time without retraining the model.

Overview

Why LoopFormer matters

Elastic depth

Standard looped models are usually trained at one fixed unroll depth. LoopFormer is trained to operate across variable-length trajectories.

Stable shortcut paths

Coarser schedules can approximate finer ones, letting the model take larger jumps without drifting or collapsing.

Compute-aware reasoning

Inference can be adapted to a chosen budget, trading cost and quality smoothly instead of being locked to a single setting.

Looped (recurrent) Transformers are an efficient alternative to deep non-shared stacks, but in practice they are almost always trained and evaluated with a fixed number of unrolls. As a result, performance often collapses when running at shorter or longer depths—making looped models far less compute-elastic than they should be.

LoopFormer treats iterative refinement as a trajectory in representation space. Each loop step is conditioned on internal time t and step size Δt, so coarser schedules can mimic finer-grained ones with fewer steps.

Training uses a shortcut-consistency objective that aligns shorter routes to the final representation of the full route. At inference time, users choose a budget M ≤ L and a schedule, and the model scales gracefully with compute—without retraining.

LoopFormer training and inference algorithms — Train-time shortcut alignment and budget-conditioned inference for variable-length loop trajectories.

Results

Strong performance across budgets

Main benchmark comparison

LoopFormer is compared against vanilla non-looped Transformers, fixed-depth looped models, and adaptive early-exit baselines. The main takeaway is simple: making looped depth explicitly budget-aware improves the quality/compute trade-off rather than only the peak setting.

Stronger behavior under aggressive compute constraints
Smoother scaling as more loop budget becomes available
A cleaner path to practical compute-adaptive language modeling

LoopFormer compared with vanilla, fixed-loop, and early-exit Transformer baselines. — Main results across perplexity and reasoning-oriented evaluation settings.

Analysis

What changes as depth and trajectories vary?

Layers vs. loops

Under similar compute budgets, LoopFormer remains competitive while varying the number of blocks and loop steps. The plots below highlight two complementary views: language modeling quality through perplexity, and zero-shot reasoning performance across 10 language reasoning benchmarks.

Perplexity under different numbers of layers and loop steps. — Perplexity across compute-matched block/loop configurations.

Accuracy on 10 language reasoning tasks under different numbers of layers and loop steps. — Zero-shot reasoning accuracy under the same compute-matched settings.

Trajectory behavior

A core idea in LoopFormer is that iterative refinement follows a trajectory in representation space. Different shortcut schedules produce different trade-offs: the route that minimizes perplexity is not always the one that maximizes reasoning accuracy.

This makes trajectory design a real control knob for deployment. LoopFormer exposes that knob directly, instead of hiding it behind a single fixed-depth operating point.

Perplexity and reasoning accuracy across different LoopFormer trajectories. — Different loop trajectories lead to different quality/compute trade-offs.

Citation

BibTeX

@inproceedings{jeddi2026loopformer,
  title     = {LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation},
  author    = {Jeddi, Ahmadreza and Ciccone, Marco and Taati, Babak},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026},
  url       = {https://openreview.net/forum?id=RzYXb5YWBs}
}