Sideplane
Research Tools contact@sideplane.ai

Research

The Geometry of Updates: Fisher Alignment at Vocabulary Scale

Choosing which corpus will best fine-tune a model is often done by comparing internal representations, but representation similarity cannot determine transfer: two tasks can share identical activations while moving the output head in orthogonal directions. The paper shows that the quantity that does govern transfer, the alignment of the tasks' output-head Fisher matrices, is exactly a cosine between two compact summaries of their activations and errors, and that this identity extends from the head to the full network. FisherSketch estimates the cosine in a single streaming pass, replacing a covariance too large to materialize at a 128k-token vocabulary with a signature small enough to store beside a model hash, and it remains computable up to 70B-scale models.

The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order

When a model is fine-tuned on several sources in sequence, the order of those sources changes the final result, and identifying the best order has traditionally required training each ordering in turn. The paper shows that the order effect is, to leading order, a single computable quantity derived from the sources' gradients, so the better of two orders can be predicted from one inexpensive probe per source rather than by exhaustive search. On instruction and preference tuning it identifies the better pairwise order about 98% of the time after a single training step.

Commutator Memory: Sparse, Path-Local Reading and Steering in Language Models

Training a language model on two data sources in opposite orders yields different weights even with identical data, and a loss or benchmark comparison shows only that the two models differ, not where. The paper shows that this order dependence leaves a readable trace in the weights: to leading order the difference is the Lie bracket of the two sources' gradients, and projecting it through the output layer assigns each vocabulary token a share of the held-out loss gap between the two orders. The trace is concentrated on a few tokens, and downweighting the ten largest closes a median 32% of the measured gap in Qwen-3-4B fine-tuning. Projecting the difference between two trained models onto the bracket identifies which came from which order in 92% of cases across four language models.

Signed-Permutation Coordinate Transport for RMSNorm Transformers

Weight-space merging and alignment methods match a model's coordinates up to permutation, a symmetry that is incomplete for RMSNorm transformers. The relabelings that leave the model's function unchanged also include an independent sign flip on every coordinate, a far larger group; omitting it introduces error as large as the update itself, and the paper proves that permutation-only matching cannot exceed a fixed accuracy ceiling at any data budget. Recovering the full symmetry lets steering vectors, sparse-autoencoder features, and adapters transfer faithfully across checkpoints, where permutation-only alignment reverses or degrades them.

Preprint · June 2026

Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise

Fine-tuning results are commonly treated as insensitive to the order in which data is shuffled, on the assumption that reordering averages out. The paper shows this fails for optimizers with memory: because the optimizer's running average advances with the step count, reordering the same data changes the outcome at first order in the learning rate rather than the negligible second order, enough that a single-shuffle comparison of two close recipes can select the wrong one. The effect is confined to a training window and averages out across independent reshuffles; the paper gives a way to size it and to set the number of seeds a fair comparison requires.