A running research log on "RYS"-style layer duplication — the phenomenon, the apparatus built to study it on a single workstation, a set of pre-registered experiments, what adversarial review changed, and the evidence to date on whether the duplicated layers solve or merely sharpen.
RYS-style duplication ("Repeat Your Self") is a zero-training model surgery: take a pretrained transformer, pick a contiguous band of middle layers [i..j], and duplicate it in the forward pass — the residual stream runs through the same weights twice (or k times) before continuing to the top of the network. No weights change. No gradient steps. The only thing that changes is the computation graph.
The reason anyone cares: community self-merge models built this way (the RYS series by dnhkng) posted real benchmark gains, including a run at the top of the Hugging Face Open LLM Leaderboard (base model reportedly Qwen2-72B — flagged, verify before citing). The gains are documented at roughly 27B parameters and above. Below that scale, nobody has shown the trick does anything — a fact that turns out to govern most of this log's story.
Framed that way, layer duplication stops being a leaderboard curiosity and becomes a mechanistic-interpretability question with unusual leverage: the intervention is perfectly clean (no confound from training), perfectly reproducible, and parameterized by two integers. If we can say what the second visit computes, we learn something about what the middle of a transformer is for.
Every experiment in this program ultimately discriminates between two hypotheses about the duplicated band.
SOLVER. The band performs genuine additional computation on the second visit. The iterated map behaves like a crude fixed-point solver or recurrent refinement step: pass 2 produces answer information that was not present in pass 1's output ranking. The signature that only a solver can produce: an item where pass 1 was confidently wrong — low entropy, correct token nowhere near the top — and pass 2 rescues it with a token that wasn't in pass 1's top-k at all.
SHARPENER. The band amplifies what is already winning. Attention re-endorses the tokens it already attends to (the KV cache is frozen from pass 1, anchoring the loop), gated MLPs re-boost features that are already active, RMSNorm rescales the result — a rich-get-richer loop that produces tighter distributions, prettier convergence curves, cosines drifting toward 1, and even benchmark gains (better calibration on items where the right answer was already ranked first or nearly first) — all with no new computation.
The sharpener is the program's steel-man rival, and it is dangerous precisely because it predicts most of the same phenomenology the solver does: convergence, contraction-looking geometry, entropy collapse, "crystallizing" circuit edges. Two formal points sharpen the boundary between them:
Everything below — the lenses, the orbit renderings, the contraction tests, the flip census — is machinery for forcing these two hypotheses to make different predictions and then paying the cost of measuring honestly.
All experiments run on one Apple Silicon workstation with 128 GB of unified memory, inference-only, one model resident at a time. The software stack is a Go toolkit over MLX, against Qwen3 checkpoints in bf16 (and 4-bit at large scale). That constraint shapes everything: no training runs, no GPU fleet, and memory watchdogs on anything that touches Jacobians.
The core reading instrument is our implementation of the Jacobian lens, following Gurnee et al., "Verbalizable Representations Form a Global Workspace in Language Models" (2026), with the authors' public reference implementation (pinned at a fixed commit) as the numerical oracle. A Jacobian lens is a fitted linear transport: for a source layer ℓ, fit a matrix that maps the residual stream at ℓ into the model's final residual space, then read it out through the model's own final norm and unembedding. The transport is fitted by delta-injection vector–Jacobian products through the model, frozen after fitting, and applied as a single matrix multiply — nothing about the lens is learned at read time. Our exact fitter matches the reference implementation to relative Frobenius error ≤ 0.031 and matrix cosine ≥ 0.9995 on Qwen3-0.6B (§ 07).
Because a fitted lens can smuggle circularity into any claim it touches, the standing rule is that every lens-based result ships with a raw-readout null — the same measurement in raw residual space, no lens — and interpretive claims must survive both.
On Qwen3-0.6B, lens fitting identifies a workspace band at layers [20..25] — where fitted transports beat the identity baseline and where interventions land. An earlier representational-similarity (CKA) analysis suggested a much lower band, roughly [2..16]. The two priors disagree and the disagreement is deliberately kept open: every experiment that uses a band must name which analysis produced it, and no result may quietly assume one definition.
Duplicated forward passes run through a plan-based execution engine with one KV cache per plan step, which is what makes multi-visit runs cache-correct. Derivative work at small scale goes through a hand-written VJP seam that is currently checkpoint-family-specific. The toolkit also serves a public interpretability-server API surface end-to-end, so standard visualization tooling interoperates.
From mid-July the program runs as a two-agent workflow: one LLM agent executes runs and edits; an independent LLM reviewer verifies every claim against the filesystem before acceptance — artifact existence, row counts, number provenance, scale-scoping of citations — and sends back blocking fixes. Several substantive errors in this log's history (§ 05, § 10) were caught exactly this way. Standing epistemic rules: verdicts are checkpoint-scoped by name; every band names its source analysis; every lens claim ships a raw-readout null; every cosine ships both raw and mean-centered/rogue-dimension-corrected variants; blockers are recorded as fail-closed artifacts rather than implicit assumptions; and a "solver signature" measured at a scale with no behavioral gain characterizes nothing.
Work is organized as short pre-registered experiment designs, each with named verdicts, explicit controls, and falsification conditions written before any run. Designs gate one another — a downstream experiment may not run until an upstream verdict lands. The convention exists to fight two failure modes: post-hoc verdict selection, and "thesis elasticity" (verdict taxonomies with an escape hatch for every outcome).
The lane under heaviest review treats the duplicated band as an approximately contractive, steerable discrete dynamical system:
| Design | Instrument | Question |
|---|---|---|
| Cross-pass geometry | Cross-pass drift curve: cos(z₁ᵈ, z₂ᵈ) over relative depth d | Does visit 2 retrace visit 1's trajectory (transport-invariant loop), decay from it, or diverge chaotically? |
| Fixed-point analysis | k = 1..4 convergence at the depth of closest approach (d*); inter-pass steering; refusal-vs-k safety arm | Does iterating the band converge like a fixed-point solve, and can the iterate be steered between passes? |
| Gradient-circuit stability | Sequential Jaccard overlap of top-decile gradient-edge sets across passes | Do the same circuit edges "crystallize" across visits (compression) or churn? |
| Lens content calibration | Prose-vs-JSON concept-vector cosine with format ceilings/floors and a causal-equivalence anchor | Is the lens reading content, or format? |
| Orbit rendering | Whitened top-2 SVD rendering of the band's iterated trajectory, with magnitude anchor | What does the orbit of the iterated map actually look like? |
Before external review, this lane had already survived six internal review rounds that installed roughly nineteen controls — wrong-block-swap and scrambled-token nulls, attenuation-offset and greedy-autopilot checks, KV-cache-anchor nulls with duplication-consistent and zero-history arms, raw-residual identity readouts, a perturbation-growth contraction test, a Gram-matrix limit-cycle guard, coherent-counterfactual handoffs, magnitude anchors, and more. The external review's job (§ 05) was to find what all of that still missed.
A hostile external-style review was commissioned over the geometry lane, with instructions not to re-derive the installed controls but to find what remained. Six findings survived verification and reshaped the program.
The fixed-point design tested contraction via R = ‖f(z′)−f(z)‖ / ‖z′−z‖ < 1 — a Lipschitz-style bound on the residual update f. But the iterated map is g(z) = z + f(z), and convergence of iterates is governed by the spectral radius ρ(I + Df) < 1, not by ‖Df‖ < 1. The installed test is neither necessary nor sufficient: a map can pass it and diverge, or fail it and converge — and the inert case f ≈ 0 aces it, so the control certifies the null it was meant to catch. Repair (adopted): measure perturbation growth through the full map across passes (a finite-time Lyapunov exponent), or estimate ρ(Dg) directly by power iteration on Jacobian-vector products — the standard machinery of the Deep Equilibrium Model literature (Bai, Kolter & Koltun 2019).
Projecting smooth high-dimensional trajectories onto top principal components produces Lissajous-like orbits and rank-concentrated spectra even for random walks — the phantom-oscillation artifact (Antognini & Sohl-Dickstein 2018; Shinn 2023, flagged-verify). The orbit design had no null for this, which made its "orbital" verdict unreachable-by-failure: the instrument would render an orbit no matter what. Repair (adopted): surrogate-data calibration — phase-randomized and AR-matched surrogates (Theiler et al. 1992) define the orbit-score threshold a real orbit must clear, as a hard gate.
Residual streams are strongly anisotropic; a handful of rogue/massive-activation dimensions dominate norms and inflate every raw cosine (Ethayarajh 2019; Timkey & van Schijndel 2021; Sun et al. 2024). Cross-pass cosines of 0.98 can reflect shared rogue dimensions, not shared content. This confound cuts across every drift curve in the lane. Repair (adopted program-wide): per-depth corpus mean-centering plus projecting out the dominant singular direction, with results reported both raw and corrected — if a verdict flips under correction, that is itself the finding.
The circuit-stability census presumes there is a behavioral effect whose circuitry is worth compressing. With no demonstrated utility at 0.6B, a "crystallizing" edge set describes iteration mechanics, not a mechanism of benefit. Layered on top: gradient-attribution unfaithfulness at this granularity (Kramár et al. 2024) and edge-set overlap statistics that behaved like coin flips under resampling. The design is now gated behind a demonstrated behavioral effect.
Several verdict taxonomies contained escape hatches — most concretely, the orbit design's limit-cycle guard made the geometry design's "monotonic decay" verdict effectively unreachable until surrogate calibration landed. Repair: quantitative pre-registration (numeric thresholds, k = 8–16 iterates rather than 4, multi-language distributions of the depth of closest approach, cluster-robust bootstrap CIs over template families) so that each verdict corresponds to a region of measurement space fixed in advance.
The review's constructive output was the iterated-sharpener rival (§ 02) plus the single experiment that separates it from the solver: an answer-flip census (§ 06). A second review pass over the census design then caught its own flaws before running: the temperature-matched control was vacuous (monotone in logit space — it can never change an argmax, so it can never explain a flip), and one claim overreached ("architecturally incapable" — feature-space sharpening can flip confident items; the claim was weakened to a statistical asymmetry). The controls were replaced with two that live in feature space: a first-pass-update extrapolation arm (push the residual further along pass 1's own update direction — "more of the same computation") and a magnitude-matched random perturbation arm (noise floor at ‖δ‖ = ‖z_pass2 − z_pass1‖).
The census is deliberately boring: run each item once normally (k=1) and once with the band iterated (k=3), and census what happens to the answer token. Its power comes from the cross-classification, because the two hypotheses disagree in exactly one cell.
Around the 2×2 sit the guards: the two feature-space controls (§ 05.6) as attribution floors, a right→wrong asymmetry census (a sharpener entrenches errors as readily as it entrenches correct answers; a solver should be asymmetric), winner's-curse split-sampling (cells selected on one half, rates estimated on the other), re-run noise floors, and template-cluster-robust paired-bootstrap confidence intervals. Verdicts are scoped to the checkpoint by name — e.g. "consistent with sharpening at 32B-4bit" — so no result at one scale can quietly annex another. All cutoffs and rules, including the minimum count of sharp-entropy wrong items required for adequate power (30), were fixed in a written pre-registration before the large-scale run.
A first behavioral utility sweep at 0.6B (2026-07-15) persisted 104 accepted rows across width-1 and width-3 duplication candidates: 32 positive rows, maximum accuracy delta +2/3 on a three-item pilot bank. The artifact explicitly demotes all band-selectivity claims — a random-range baseline of +0.4 nearly matches the winners — and pre-registered a held-out recheck before any interpretation. A census of the raw generations then found that of 96 baseline records in positive rows, 64 produced no answer at all (echo/stall) and 32 buried the right answer in rambling — zero were confidently wrong. Every apparent 0.6B "gain" is a format/verbosity rescue: the duplicated pass un-jams degenerate decoding on content the model already knows. The held-out recheck closed the loop the same day: on a frozen 30-item set at the fixed [20..22] range, baseline scored 0/30 and duplication 2/30 (+0.067, below the pre-registered +0.10 gate) — no held-out replication. The pilot's positive rows do not generalize.
Taken together, the two scales tell one coherent story so far: at small scale there is no real utility to explain, and what looks like utility is decoding-mode rescue; at documented scale the model's errors overwhelmingly already contain the right answer just below the top, which is exactly the population a sharpener helps and a solver is not needed for.
The lens implementation is externally anchored: the exact fitter matches the published reference implementation on Qwen3-0.6B. A finite-rank sketched variant of the fitter, however, is not a numerical substitute for the exact Jacobian — a result with methodological consequences (§ 08).
| Path | rel. Frobenius | matrix cosine | Verdict |
|---|---|---|---|
| Exact fit (two measured transports) | 0.00515 / 0.03114 | 0.999988 / 0.999515 | parity |
| Sketch, rank 32 | 5.616 | 0.174 | non-parity |
| Sketch, rank 64 | 3.980 | 0.242 | |
| Sketch, rank 128 | 2.827 | 0.333 |
Relative Frobenius error above 1.0 means the rank-128 approximation is worse than the zero matrix at matrix level: these Jacobians are high-rank and diffuse in ambient space. The sketch's demonstrated value is downstream and corpus-averaged (its band-detection stack beats the identity baseline), not per-prompt matrix recovery. Separately, exact fitting is intractable at Qwen3-4B on this host (killed at ~66.6 GB wired, no artifact) while the rank-128 sketch completes in 274 s — so any Jacobian statistic at 4B+ must come from O(100)-probe estimators, never materialization. A 468-cell viewer comparison against the reference visualization recorded zero rank-array mismatches (11 near-tie top-1 swaps), retiring the rendering path as a suspect.
After the inconclusive small-scale census, three label-free discriminator ideas were proposed — designs that would not need a large correct-answer dataset. Each was audited adversarially, on statistics and causal logic, under the rule "assume it is subtly wrong until it survives."
Proposal: inject the midpoint of two prompts' residuals into the band and measure the nonlinearity deviation ‖G(z_mid) − ½(G(z_A)+G(z_B))‖; near-zero ⇒ sharpener, large ⇒ solver.
The audit: this quantity is exactly the Hessian quadratic form of the block along the chord — a curvature meter, guaranteed positive for any smooth nonlinear map and scaling with ‖z_A−z_B‖². Worse, midpoints of two prompts sit near basin boundaries, and the mechanistic sharpener (softmax-driven winner-take-all) is violently nonlinear precisely there — so the sharpener produces its maximum deviation exactly where the test looks, making a false "solver" the expected outcome. "Commutes with interpolation" is the signature of an affine toy nobody's hypothesis asserts. And the map isn't a function of z alone: injecting a midpoint state over one prompt's KV cache is incoherent unless the pair shares its prefix.
Surviving form (renamed, because it no longer discriminates): a label-free basin-geometry probe with pre-registered dual basin-assignment functions, an α-sweep along the chord, hysteresis under loop count, matched-chord scrambled and random-direction floors, and a norm-restoration control. It answers "what shape is the dynamics," not "does it compute."
Executed at 0.6B (30 shared-prefix pairs × 21 α values × loop counts {1,2,3,4,8} × 4 conditions = 12,600 rows, with the iterated-band plans frozen as literal layer sequences before the run). The headline result is a null with teeth: real pairs show a basin boundary at median α ≈ 0.85 across loop counts — but the scrambled-pair floor shows the same boundary location, so boundary position is a property of the ambient state geometry, not of the specific pair's contrast. Random-direction chords never cross into the far basin at any loop count. The two pre-registered assignment functions agreed on 9,371 rows, disagreed on 440, and the raw-readout assignment was non-assignable on 2,789 (the readout's top token matched neither endpoint identity — itself a finding about how far interpolated states sit from either endpoint's decode). Reported geometry-only, as pre-registered: none of this is evidence for or against solving.
Proposal: hypothesize duplication fails below ~1B because of feature superposition; compute the participation ratio PR = (Σσ²)²/Σσ⁴ of an input–output Jacobian across a model-size ladder and look for a fracture co-located with the onset of duplication gains.
The audit: a Jacobian's spectral PR is an effective-rank/sensitivity measure, not a superposition measure (superposition in the Elhage et al. 2022 sense lives in the representation; no established mapping connects the two). The ladder has 4–6 points that are different models, not one model rescaled; distinguishing a fracture from a smooth power law with five heterogeneous points is statistically hopeless; and a co-located kink would be pure correlation — everything fractures somewhere on a scale ladder.
Surviving form: drop the ladder. Within one checkpoint, correlate per-item Jacobian PR with per-item duplication utility, cluster-robust over template families, with the sign pre-registered — the superposition story actually predicts harm on high-interference items, a sharper falsifiable claim than absence-of-gain. Estimator: Hutchinson-only — both spectral sums are trace estimates from probe products (tr JᵀJ = E‖Jv‖², tr (JᵀJ)² = E‖JᵀJv‖²), with probe-bootstrap confidence intervals. Rank-truncated sketch spectra are banned for PR on our own evidence (§ 07.3: the sketch is matrix-level non-parity, so a truncated spectrum measures the sketch, not the model). The design is written but gated: it requires a checkpoint with demonstrated item-level utility, which nothing on this host currently has.
Proposal: even if duplication is "just" a sharpener, exploit it — use the cross-pass entropy drop ΔH = H(k=1) − H(k=3) to filter speculative-decoding drafts, or as a context-sensitivity reranker for retrieval.
The audit: under anything close to monotone sharpening, H(k=3) is approximately a deterministic function of the pass-1 distribution's shape, so ΔH is largely a reparametrization of H(k=1) — high mutual information, near-zero incremental signal. The decisive test is cheap: on logged speculative-decoding tokens, compare the acceptance-prediction AUC of {H(k=1)} against {H(k=1), ΔH} (DeLong test) with a pre-registered minimum increment. Even if it passes, the economics bind: ΔH costs k−1 extra band passes per drafted token — the compute the filter exists to save — so only a two-stage design (threshold on H(k=1) first; extra pass only in a pre-registered ambiguous band) can break even. The retrieval variant is contrastive-context confidence with a duplication costume on; it carries no duplication-specific content. This pre-check is now a written design: logged-tokens-only, 4B+ scope (at 0.6B the quantity is numerical noise), a pre-registered absolute AUC increment of 0.03 with paired DeLong at α = 0.05, an explicit break-even inequality that must be evaluated with observed acceptance precisions, and verdict semantics under which a failed gate declares both applications dead in the design itself.
Inventory executed, fail-closed: a sweep of all existing 4B+ logs found no rows carrying the required unit — the same drafted token with both-pass entropies, an acceptance label, position, and checkpoint identity. The nearest records (4B answer-flip banks with pass-1/pass-2 top-k fields) were tabled with content hashes and their exact missing fields, and the schema any future qualifying log must record is now specified. No analysis was run on non-qualifying data; the pre-check waits for logs that actually contain its unit.
The standing epistemic rules travel with all of it: verdicts are checkpoint-scoped; every band names its source analysis; every lens claim ships a raw-readout null; every cosine ships mean-centered and rogue-dimension-corrected variants; and a "solver signature" measured at a scale with no behavioral gain characterizes nothing.
Citations this work leans on, with confidence flags. "Verified" means checked against the literature during review; "flagged" means cited from memory and awaiting verification — flagged items must be verified before appearing in anything formal.
| Claim it anchors | Source | Status |
|---|---|---|
| Spectral-radius convergence criterion; power iteration for iterated blocks | Bai, Kolter & Koltun, Deep Equilibrium Models, NeurIPS 2019 | verified |
| PCA phantom oscillations from smooth trajectories | Antognini & Sohl-Dickstein, arXiv:1806.08805 | verified |
| Surrogate-data significance testing | Theiler et al., 1992 | verified |
| Residual-stream anisotropy; rogue dimensions; massive activations | Ethayarajh 2019; Timkey & van Schijndel 2021; Sun et al., arXiv:2402.17762 | verified |
| Gradient-attribution unfaithfulness at circuit granularity | Kramár et al., AtP*, arXiv:2403.00745 | verified |
| Superposition as features-per-dimension with interference | Elhage et al., Toy Models of Superposition, 2022 | verified |
| Stochastic trace estimation; randomized range-finding | Hutchinson 1990; Halko, Martinsson & Tropp 2011 | verified |
| Jacobian-lens method and reference implementation | Gurnee et al. 2026, Verbalizable Representations…; authors' public implementation, pinned commit | pinned |
| Phantom-oscillation follow-up in neuroscience | Shinn, PNAS 2023 | flagged — verify |
| RYS-XLarge base model identity (Qwen2-72B) | dnhkng model cards / leaderboard records | flagged — verify |