The Geometry of Updates: Fisher Alignment at Vocabulary Scale
Choosing which corpus will best fine-tune a model is often done by comparing internal representations, but representation similarity cannot determine transfer: two tasks can share identical activations while moving the output head in orthogonal directions. The paper shows that the quantity that does govern transfer, the alignment of the tasks' output-head Fisher matrices, is exactly a cosine between two compact summaries of their activations and errors, and that this identity extends from the head to the full network. FisherSketch estimates the cosine in a single streaming pass, replacing a covariance too large to materialize at a 128k-token vocabulary with a signature small enough to store beside a model hash, and it remains computable up to 70B-scale models.
The Geometry of Sequential Learning: Lie-Bracket Prediction of Transfer Order
When a model is fine-tuned on several sources in sequence, the order of those sources changes the final result, and identifying the best order has traditionally required training each ordering in turn. The paper shows that the order effect is, to leading order, a single computable quantity derived from the sources' gradients, so the better of two orders can be predicted from one inexpensive probe per source rather than by exhaustive search. On instruction and preference tuning it identifies the better pairwise order about 98% of the time after a single training step.
Signed-Permutation Coordinate Transport for RMSNorm Transformers
Weight-space merging and alignment methods match a model's coordinates up to permutation, a symmetry that is incomplete for RMSNorm transformers. The relabelings that leave the model's function unchanged also include an independent sign flip on every coordinate, a far larger group; omitting it introduces error as large as the update itself, and the paper proves that permutation-only matching cannot exceed a fixed accuracy ceiling at any data budget. Recovering the full symmetry lets steering vectors, sparse-autoencoder features, and adapters transfer faithfully across checkpoints, where permutation-only alignment reverses or degrades them.
Task arithmetic, the standard way to add or remove a behavior by adding or subtracting a weight update, is geometrically incomplete. It treats the update as a fixed vector, but any training that follows transports that vector along the optimization path, which makes subtracting the original update provably only first-order accurate. The paper recovers the transported direction from the training run itself and restores the missing term, yielding an edit that is accurate to second order.
Optimizer Memory Makes Shuffle Order a First-Order Source of Fine-Tuning Noise
Fine-tuning results are commonly treated as insensitive to the order in which data is shuffled, on the assumption that reordering averages out. The paper shows this fails for optimizers with memory: because the optimizer's running average advances with the step count, reordering the same data changes the outcome at first order in the learning rate rather than the negligible second order, enough that a single-shuffle comparison of two close recipes can select the wrong one. The effect is confined to a training window and averages out across independent reshuffles; the paper gives a way to size it and to set the number of seeds a fair comparison requires.