Speech, squeezed to 6.25 Hz and back

A Conformer runs every block at 25 Hz. Zipformer subsamples only by 2 up front, so stack 1 sees 50 Hz, and the factors (1, 2, 4, 8, 4, 2) set the rest. Between stacks the sequence is back at 50 Hz; a final Downsample gives a 25 Hz output, the rate a Conformer hands its decoder.

Downsampling by k averages each group of k frames with k softmax-normalized learned weights. Upsampling repeats each frame k times. A Bypass with a learned per-channel weight c mixes the stack's output with its input:

c starts limited to [0.9, 1.0] and the floor drops to 0.2 after 20,000 steps (Section 3.2), so every stack has to learn something before it may switch itself mostly off.

Widths differ per stack, largest in the middle where frames are cheapest; the sequence is truncated or zero-padded between them. Try any clip length and model size:

Self-attention does two quadratic things: it computes attention weights and it uses them to mix frames. Zipformer splits these into a Multi-Head Attention Weight module (MHAW) and Self-Attention modules (SA). The paper gives weight reuse as the main reason the deeper block is affordable (Section 3.2). Step through it in code order; the 3D stack above follows along.

Non-linear attention

NLA uses the weights for more than value averaging. With projections A, B, C of the input, each 3/4 of its width (paper Figure 2, right):

Here attention means mixing over time with one head of the shared weights, the first (attn_weights[0:1]). It is a convolution module with the convolution swapped for attention-weighted mixing.

What the ablations say

Zipformer-M, LibriSpeech test-clean / test-other WER, paper Table 5. Full model: 2.21 / 4.79. Two Conformer-style blocks in place of each Zipformer block: 2.18 / 4.95, with 73.9M parameters instead of 65.6M. No NLA: 2.16 / 4.97. No Bypass: 2.25 / 4.86. No NLA and no weight sharing: 2.20 / 5.10, slower with no gain, which the authors read as the block's two attention patterns being similar enough to share.

The authors saw two habits in trained Conformers that they blame on LayerNorm (Section 3.3). Some set one channel to a large constant, for example 50, to keep length information alive. Some modules go "dead", with outputs near 10^-6: the LayerNorm scale $\gamma$ drifts toward zero and, if it oscillates around zero, the module's gradient keeps flipping sign.

The per-channel bias b only enters the RMS, so a large entry plays the constant channel's role without using a real one. $\exp(\gamma)$ is always positive, so the scale cannot flip sign, and there is no mean subtraction. Zipformer has one BiasNorm per block and one in Conv-Embed, and no per-module LayerNorms.

SwooshR's constant puts it through the origin. SwooshL's was tuned and slightly beat the value that would, so SwooshL(0) is about -0.017. The letter names which zero crossing sits near x = 0 (Section 3.4).

The left-hand slope is the point. With Swish, a unit whose input stays negative gets almost no gradient, and the paper notes that the Adam denominator can then get dangerously small.

Placement follows a training observation. With SwooshR everywhere, the modules behind a bypass, feed-forward and the ConvNeXt layer, learned a large negative bias to be "normally off". SwooshL is near zero around x = 0, so they get SwooshL; convolution modules and the rest of Conv-Embed get SwooshR. On Zipformer-M, SwooshR everywhere costs 0.11 / 0.42 WER (test-clean / test-other), and Swish everywhere gives 2.27 / 5.37 against 2.21 / 4.79 (Table 5).

Without per-module normalization, something has to keep activations in range. Balancer and Whiten add the gradient of an auxiliary loss to the real gradient, kept small next to it (Appendix A.3). The paper says their placement follows no clean rule: they were added where a module trained badly or diverged.

Balancer

Per channel, it bounds the proportion of positive values and the mean absolute value, to catch values that grow too large or small (a mixed-precision risk) and "dead" channels that are almost always negative. Under a Gaussian assumption these become limits on $\mathrm{E}[x]/\sqrt{\operatorname{Var}[x]}$ and $\operatorname{RMS}[x]$ (Equation 9). The extra gradient is scaled to 0.04 times the real gradient's magnitude at each element, so padding frames get tiny corrections.

Whiten

The second failure is a layer whose covariance is dominated by one or a few directions, seen in models about to diverge. Whiten measures it as

over the eigenvalues of the mean-subtracted covariance C: 1 when white, D when one direction holds everything, and computable without an eigendecomposition. It penalizes only above a limit: 10 in the paper, a schedule from 4 to 12 over 20,000 batches for the block output in the code.

Adam's step on each element is roughly the learning rate whatever the parameter's size: large in relative terms for a tensor with RMS 0.01, small for one with RMS 1. ScaledAdam multiplies the update by the tensor's current RMS r_t-1 (Section 3.5, Equation 6):

It also learns the scale directly: treat each tensor as $\theta = r \cdot \theta'$, run Adam on r with gradient $h_t = g_t \cdot \theta_{t-1}$, and step along the tensor itself (Equation 7):

Shrinking a tensor with plain gradient steps is hard, since step noise tends to grow its norm. This term handles it, and it is why Zipformer can drop most normalization layers. The code adds limits the paper does not spell out: scale updates every 4 steps, each clamped to [-0.1, 0.1]; tensor RMS kept in [1e-5, 3.0]; scalars get plain Adam at 0.1 times the learning rate; gradients clipped at twice the recent median norm (clipping_scale=2.0).

The Eden schedule

ScaledAdam is less prone to divergence, so it needs no long warm-up, and its learning rates are numerically larger because parameter RMS values are usually well below 1. Eden decays with both step t and epoch e (Equation 8):

Two factors keep parameter change per hour of data roughly constant when the batch size changes. The paper uses base 0.045, start 0.5 and a 500-step warm-up; the icefall recipe uses lr_batches = 7500, lr_epochs = 3.5 and a warm-up start of 0.1. The epoch counter is 0 during the first epoch (scheduler.step_epoch(epoch - 1)).

The result in Table 5: with Adam, Zipformer-M needed a BiasNorm in every module to avoid divergence, and its best run reached 2.38 / 5.51 WER on test-clean / test-other against 2.21 / 4.79 with ScaledAdam. Learning-rate factors 0.025, 0.035, 0.045 and 0.055 were tried for ScaledAdam and 2.5, 5.0, 7.5 and 10.0 for Adam; the best of each is reported.

The paper evaluates full-context models only; streaming lives in the icefall code. With --causal 1, each batch draws a chunk size from --chunk-size (default 16,32,64,-1) and a left context from --left-context-frames (default 64,128,256,-1), in 50 Hz frames; -1 means no limit. One model can then run at several chunk sizes.

The mask is built at 50 Hz and each stack takes every k-th row and column, so chunk boundaries line up across stacks. Hence the code's asserts: the chunk size must divide by every downsampling factor, and the left context must cover half of each convolution kernel.

Convolution without lookahead

Convolutions look ahead too: kernel 31 sees 15 frames into the future. Causal Zipformer swaps in a chunk-causal depthwise convolution: a causal convolution of half the kernel width plus one that runs only inside the current chunk, scaled by a learned, position-dependent factor. A frame can still use frames to its right within its own chunk, never beyond.

What a streaming step costs and carries

Each call takes $T = 2 \cdot \text{chunk} + 13$ feature frames: twice the chunk because Conv-Embed halves the rate, plus 7 for its convolutions and 6 for the 3 frames of right padding its ConvNeXt layer needs at 50 Hz. It returns chunk / 2 frames at 25 Hz. Every layer carries six tensors between calls: cached keys, NLA inputs, values for both SA modules, and the left context of both convolutions, with left_context_frames // k frames for a stack with factor k. The ONNX export adds a Conv-Embed cache and a processed-frame count.

Smaller chunks mean lower latency and less right context; longer left context means more state and more attention work per step. The accuracy cost of a chunk size is something to measure on your own data.

Every number here is copied from the paper (arXiv 2310.11230v4): pruned transducer unless stated, beam search of size 4, no external language model. The authors' summary: "In terms of efficiency, Zipformer converges faster during training and speeds up the inference by more than 50% compared to previous studies while requiring less GPU memory." (Section 1)

The paper flags a baseline problem: neither its own Conformers nor the WeNet and ESPnet ones reach the original Conformer paper's 2.1 / 4.3 for Conformer-L. Its longer-trained Zipformer-L reaches 2.00 / 4.38, "the first model to approach Conformer-L to the best of our knowledge."

Zipformer also trains with other heads. In Table 8, CTC-only Zipformer-S/M/L reach 2.85 / 6.91, 2.51 / 6.02 and 2.49 / 5.7, and CTC/AED versions with a 6-layer Transformer decoder reach 2.46 / 6.04, 2.22 / 4.97 and 2.09 / 4.59.

Mandarin

Training lives in icefall; deployment usually goes through k2-fsa/sherpa-onnx. A streaming export bakes one chunk size and left context into the graph.

Export a streaming transducer

The architecture flags must match training. Chunk size and left context are picked from the values the model was trained with (from 16,32,64 and 64,128,256 for the default recipe):

# in icefall/egs/librispeech/ASR, with $repo holding exp/epoch-99.pt
./zipformer/export-onnx-streaming.py \
  --tokens $repo/data/lang_bpe_500/tokens.txt \
  --use-averaged-model 0 --epoch 99 --avg 1 \
  --exp-dir $repo/exp \
  --num-encoder-layers "2,2,3,4,3,2" \
  --downsampling-factor "1,2,4,8,4,2" \
  --feedforward-dim "512,768,1024,1536,1024,768" \
  --num-heads "4,4,4,8,4,4" \
  --encoder-dim "192,256,384,512,384,256" \
  --query-head-dim 32 --value-head-dim 12 \
  --pos-head-dim 4 --pos-dim 48 \
  --encoder-unmasked-dim "192,192,256,256,256,192" \
  --cnn-module-kernel "31,31,15,15,15,31" \
  --decoder-dim 512 --joiner-dim 512 \
  --causal True --chunk-size 16 --left-context-frames 128

This writes an encoder, a decoder and a joiner. The encoder carries its streaming contract as ONNX metadata: model_type=zipformer2, decode_chunk_len (twice the chunk, in feature frames), T (decode_chunk_len + 13), left_context_len, and the per-stack layers, widths, kernels and head sizes. A runtime reads these instead of hard-coding them; check them first when a port produces garbage.

Decode a stream

import numpy as np, sherpa_onnx, soundfile as sf

rec = sherpa_onnx.OnlineRecognizer.from_transducer(
    tokens="tokens.txt",
    encoder="encoder-epoch-99-avg-1-chunk-16-left-128.onnx",
    decoder="decoder-epoch-99-avg-1-chunk-16-left-128.onnx",
    joiner="joiner-epoch-99-avg-1-chunk-16-left-128.onnx",
    num_threads=2, sample_rate=16000, feature_dim=80,
    decoding_method="greedy_search",
)
samples, sr = sf.read("test.wav", dtype="float32")
s = rec.create_stream()
s.accept_waveform(sr, samples)           # call repeatedly with live audio
s.accept_waveform(sr, np.zeros(int(0.66 * sr), dtype=np.float32))  # tail padding, as in the official example
s.input_finished()
while rec.is_ready(s):
    rec.decode_stream(s)
print(rec.get_result(s))

A CTC-only Zipformer uses OnlineRecognizer.from_zipformer2_ctc(tokens=..., model=...) instead, with one ONNX file. sherpa-onnx computes the 80-dimensional fbank features itself from 16 kHz audio, so the front end matches what icefall trained on.

On a device

• Memory per stream is the cache state from the streaming section, about 1.7 MiB in fp32 for Zipformer-M with 128 frames of left context, plus the activations of one chunk.
• Every encoder call takes exactly 2 x chunk + 13 feature frames and fixed-size caches, so the graph has fixed shapes. That suits accelerators that compile a static graph, and it means one export per chunk size.
• Features are Kaldi-style fbank (25 ms window, 10 ms shift, 80 bins). A different mel front end is a different input distribution; reproduce the training features exactly.

Everything in this section comes from the public model card of Quran-Lab/zipformer_p-arabic-v3. The current release is v3.1, a fine-tune of v3 that retunes madd (elongation) length; it keeps the architecture, streaming interface and token table, so it drops in for v3 in every format.

The card describes a 65.5M-parameter streaming phoneme recognizer for Hafs 'an 'Asim: Zipformer2 with a CTC head, emitting a 251-symbol Qur'anic phonetic alphabet that encodes madd length, gemination, ghunna, ikhfaa, qalqalah and the emphatic consonants. It uses no language model anywhere, because a language model silently corrects a reciter's mistakes, which an assessment system must never do.

How the architecture shows up in the release

• The streaming contract. The ONNX files are standard sherpa-onnx streaming-CTC zipformer2 exports with a chunk of 48 input frames, and the CoreML metadata gives T=61, decode_chunk_len=48. That is the formula from the streaming section: 48 = 2 x 24, so a chunk of 24 frames at 50 Hz, 0.48 s, and $T = 2 \cdot 24 + 13 = 61$.
• Trained chunk sizes. The card says the model was trained only at chunk sizes 8, 16 and 24, and that the best chunk size depends on the domain.
• The front end. Input is 16 kHz mono with 80-dimensional Kaldi fbank (povey window). The card is explicit that a mel-spectrogram front end is not a substitute.
• Swoosh in fp16. The icefall Swoosh code guards overflow by checking for inf. The card notes that fp16 hardware on the Apple Neural Engine saturates at 65,504 instead of producing inf, which silently corrupts an inf-compare guard; the CoreML graph uses a threshold compare instead, a max-subtracted log-softmax head, and no fp32 epsilon parameters.
• Splitting the encoder. The encoder is too large for one Neural Engine program, so the CoreML packages split it into two functions, front and back, sharing one weight file. The cut is at the narrowest point of the network, where a single 24 x 1 x 384 tensor crosses.
• Token table. The CTC blank is id 250, not 0; decoding must use the shipped tokens.txt.

The card also lists measured limitations, for example that a model trained almost entirely on correct recitation with canonical labels tends to transcribe the rule rather than a reciter's deviation from it. Read the card before building on it; the model is released under the Quran-Lab No-Profit License 1.2 with gated access.