Fine-tuning Offline FastConformer to Streaming Architecture

[Firaskraiem1]

I am trying to adapt nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0 (offline, att_context_size: [-1, -1]) to a streaming architecture (att_context_size: [70, 13], att_context_style: chunked_limited) using speech_to_text_hybrid_rnnt_ctc_bpe.py with init_from_pretrained_model (encoder only, decoder/joint reinitialized from scratch).
The RNNT decoder collapses immediately to a blank-only prediction strategy and never recovers, resulting in val_wer stuck at exactly 1.000 across all training runs regardless of hyperparameters.

init_from_pretrained_model:
  model0:
    name: nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0
    include:
      - encoder
    exclude:
      - conv.batch_norm
      - pre_encode.out.weight

@pzelasko
@chtruong814
@nithinraok

[nithinraok]

It would be helpful if you could post your full config (the model section + optim + the init_from_pretrained_model block as it actually ran), but in the meantime I'd suggest checking the following first:

1. Try loading the decoder/joint as well, not encoder-only.
stt_ar_fastconformer_hybrid_large_pc_v1.0 is already a hybrid RNNT-CTC model with the Arabic tokenizer, so if you're keeping the same tokenizer there's no need to start the decoder from scratch. A cold decoder + a perturbed encoder is the most common way to fall into blank collapse. Try:

init_from_pretrained_model:
  model0:
    name: nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0
    include:
      - encoder
      - decoder
      - joint
      - ctc_decoder      # the aux CTC head

(If you did intentionally change the tokenizer/vocab, then a cold decoder is expected — but in that case make sure joint.num_classes, decoder.vocab_size, and aux_ctc.decoder.num_classes all match the new vocab.)

2. Make sure the full streaming block is set, not just att_context_size.
Flipping only the attention context leaves the conv front-end in a regime it was never trained for (it'll leak future context across chunk boundaries). For a cache-aware streaming FastConformer all of these need to change together:

Param	Offline (the ar model)	Streaming
att_context_size	[-1, -1]	[70, 13]
att_context_style	regular	chunked_limited
conv_context_size	null	causal
conv_norm_type	batch_norm	layer_norm
causal_downsampling	false	true

Your exclude: conv.batch_norm suggests you already switched to layer_norm — can you confirm you also set conv_context_size: causal and causal_downsampling: true? If only the norm changed, the encoder isn't actually streaming-correct yet.

3. Lower the learning rate and use a longer warmup.
The stock hybrid config (Noam, lr: 5.0, warmup_steps: 10000) is tuned for from-scratch training and will destroy the pretrained encoder in the first few hundred steps — which itself triggers blank collapse. For this kind of adaptation I'd drop the effective LR a lot, lengthen the warmup, and keep the CTC aux loss alive (aux_ctc.ctc_loss_weight: 0.3,
you can even bump it early) so it bootstraps the encoder alignment.

4. Reconsider excluding pre_encode.out.weight.
Excluding it randomly re-initializes the projection from the subsampling block into the conformer layers, which is a big perturbation right at the encoder input. causal_downsampling: true shouldn't change that layer's shape, so if you added it to exclude to silence a shape-mismatch error, that mismatch likely points to something structural (a different
feat_in/n_mels, d_model, or subsampling_factor than the pretrained model). I'd track down why it mismatched rather than excluding it.

[Firaskraiem1]

It would be helpful if you could post your full config (the model section + optim + the init_from_pretrained_model block as it actually ran), but in the meantime I'd suggest checking the following first:

1. Try loading the decoder/joint as well, not encoder-only. stt_ar_fastconformer_hybrid_large_pc_v1.0 is already a hybrid RNNT-CTC model with the Arabic tokenizer, so if you're keeping the same tokenizer there's no need to start the decoder from scratch. A cold decoder + a perturbed encoder is the most common way to fall into blank collapse. Try:

init_from_pretrained_model:
model0:
name: nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0
include:
- encoder
- decoder
- joint
- ctc_decoder # the aux CTC head
(If you did intentionally change the tokenizer/vocab, then a cold decoder is expected — but in that case make sure joint.num_classes, decoder.vocab_size, and aux_ctc.decoder.num_classes all match the new vocab.)

2. Make sure the full streaming block is set, not just att_context_size. Flipping only the attention context leaves the conv front-end in a regime it was never trained for (it'll leak future context across chunk boundaries). For a cache-aware streaming FastConformer all of these need to change together:

Param Offline (the ar model) Streaming
att_context_size [-1, -1] [70, 13]
att_context_style regular chunked_limited
conv_context_size null causal
conv_norm_type batch_norm layer_norm
causal_downsampling false true
Your exclude: conv.batch_norm suggests you already switched to layer_norm — can you confirm you also set conv_context_size: causal and causal_downsampling: true? If only the norm changed, the encoder isn't actually streaming-correct yet.

3. Lower the learning rate and use a longer warmup. The stock hybrid config (Noam, lr: 5.0, warmup_steps: 10000) is tuned for from-scratch training and will destroy the pretrained encoder in the first few hundred steps — which itself triggers blank collapse. For this kind of adaptation I'd drop the effective LR a lot, lengthen the warmup, and keep the CTC aux loss alive (aux_ctc.ctc_loss_weight: 0.3, you can even bump it early) so it bootstraps the encoder alignment.

4. Reconsider excluding pre_encode.out.weight. Excluding it randomly re-initializes the projection from the subsampling block into the conformer layers, which is a big perturbation right at the encoder input. causal_downsampling: true shouldn't change that layer's shape, so if you added it to exclude to silence a shape-mismatch error, that mismatch likely points to something structural (a different feat_in/n_mels, d_model, or subsampling_factor than the pretrained model). I'd track down why it mismatched rather than excluding it.

@nithinraok
Hello , sorry I forgot to mention that I'm doing two things simultaneously ,
I'm fine-tuning nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0 for Tunisian Arabic dialect ASR with 25h of Tunisian arabic speech (with frequent French/English code-switching, not covered by the base MSA model), while simultaneously converting the model from offline to streaming inference.
The base model's tokenizer was trained on MSA-only data and does not support code-switching with French/English Latin script.That's why i'm trying to train new tokenizer of the new dataset with 1024 vocab size same as the original model .

Environment:
GPU: Kaggle T4/P100 (16GB)

Config used:

name: "FastConformer-Hybrid-Streaming-Tunisian-Finetune"
init_from_pretrained_model:
  model0:
    name: nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0
    include:
      - encoder
    exclude:
      - conv.batch_norm
      - pre_encode.out.weight
model:
  sample_rate: 16000
  compute_eval_loss: false
  log_prediction: true
  skip_nan_grad: true
  rnnt_reduction: mean_volume

  model_defaults:
    enc_hidden: ${model.encoder.d_model}
    pred_hidden: 640
    joint_hidden: 640

  train_ds:
    manifest_filepath: /kaggle/input/datasets/x/tedxtn-dataset/TEDxTN_dataset/manifests_kaggle/train_manifest.jsonl
    sample_rate: ${model.sample_rate}
    batch_size: 8
    shuffle: true
    num_workers: 4
    pin_memory: true
    max_duration: 30.0
    min_duration: 0.5
    is_tarred: false
    tarred_audio_filepaths: null
    shuffle_n: 2048
    bucketing_strategy: "fully_randomized"
    bucketing_batch_size: null

  validation_ds:
    manifest_filepath: /kaggle/input/datasets/x/tedxtn-dataset/TEDxTN_dataset/manifests_kaggle/dev_manifest.jsonl
    sample_rate: ${model.sample_rate}
    batch_size: 8
    shuffle: false
    use_start_end_token: false
    num_workers: 4
    pin_memory: true

  test_ds:
    manifest_filepath: /kaggle/input/datasets/x/tedxtn-dataset/TEDxTN_dataset/manifests_kaggle/test_manifest.jsonl
    sample_rate: ${model.sample_rate}
    batch_size: 8
    shuffle: false
    use_start_end_token: false
    num_workers: 4
    pin_memory: true

  tokenizer:
    dir: /kaggle/input/datasets/x/tokenizer-1024-v2
    type: bpe

  preprocessor:
    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    sample_rate: ${model.sample_rate}
    normalize: "per_feature"
    window_size: 0.025
    window_stride: 0.01
    window: "hann"
    features: 80
    n_fft: 512
    frame_splicing: 1
    dither: 0.00001
    pad_to: 0

  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2
    time_masks: 10
    freq_width: 27
    time_width: 0.05

  encoder:
    _target_: nemo.collections.asr.modules.ConformerEncoder
    feat_in: ${model.preprocessor.features}
    feat_out: -1
    n_layers: 17
    d_model: 512
    use_bias: true
    subsampling: dw_striding
    subsampling_factor: 8
    subsampling_conv_channels: 256
    causal_downsampling: true
    ff_expansion_factor: 4
    self_attention_model: rel_pos
    n_heads: 8
    att_context_size: [70, 13]
    att_context_style: chunked_limited
    att_context_probs: null
    xscaling: true
    pos_emb_max_len: 5000
    conv_kernel_size: 9
    conv_norm_type: 'layer_norm'
    conv_context_size: causal
    dropout: 0.1
    dropout_pre_encoder: 0.1
    dropout_emb: 0.0
    dropout_att: 0.1
    stochastic_depth_drop_prob: 0.0
    stochastic_depth_mode: linear
    stochastic_depth_start_layer: 1

  decoder:
    _target_: nemo.collections.asr.modules.RNNTDecoder
    normalization_mode: null
    random_state_sampling: false
    blank_as_pad: true
    prednet:
      pred_hidden: ${model.model_defaults.pred_hidden}
      pred_rnn_layers: 1
      t_max: null
      dropout: 0.2

  joint:
    _target_: nemo.collections.asr.modules.RNNTJoint
    log_softmax: null
    preserve_memory: false
    fuse_loss_wer: true
    fused_batch_size: 4
    jointnet:
      joint_hidden: ${model.model_defaults.joint_hidden}
      activation: "relu"
      dropout: 0.2

  decoding:
    strategy: "greedy_batch"
    greedy:
      max_symbols: 10
    beam:
      beam_size: 2
      return_best_hypothesis: false
      score_norm: true
      tsd_max_sym_exp: 50
      alsd_max_target_len: 2.0

  aux_ctc:
    ctc_loss_weight: 0.3
    use_cer: false
    ctc_reduction: 'mean_batch'
    decoder:
      _target_: nemo.collections.asr.modules.ConvASRDecoder
      feat_in: null
      num_classes: -1
      vocabulary: []
    decoding:
      strategy: "greedy"

  interctc:
    loss_weights: []
    apply_at_layers: []

  loss:
    loss_name: "default"
    warprnnt_numba_kwargs:
      fastemit_lambda: 0.0
      clamp: -1.0

  optim:
    name: adamw
    lr: 1e-3
    betas: [0.9, 0.98]
    weight_decay: 1e-3
    sched:
      name: CosineAnnealing
      warmup_steps: 500
      warmup_ratio: null
      min_lr: 5e-6

trainer:
  devices: -1
  num_nodes: 1
  max_epochs: 30
  max_steps: -1
  val_check_interval: 0.5
  accelerator: auto
  strategy:
    _target_: lightning.pytorch.strategies.DDPStrategy
    gradient_as_bucket_view: true
    find_unused_parameters: true 
  accumulate_grad_batches: 4
  gradient_clip_val: 1
  precision: bf16
  log_every_n_steps: 100
  enable_progress_bar: true
  num_sanity_val_steps: 0
  check_val_every_n_epoch: 1
  sync_batchnorm: true
  enable_checkpointing: false
  logger: false
  benchmark: false

exp_manager:
  exp_dir: /kaggle/working/experiments_streaming
  name: ${name}
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  checkpoint_callback_params:
    monitor: "val_wer"
    mode: "min"
    save_top_k: 3
    always_save_nemo: true
  resume_if_exists: false
  resume_ignore_no_checkpoint: true
  create_wandb_logger: true
  wandb_logger_kwargs:
    name: streaming-tedxtnfinetune-${now:%Y-%m-%d_%H-%M-%S}-${trainer.max_epochs}epochs-LR${model.optim.lr}-MinLR${model.optim.sched.min_lr}-Warmup${model.optim.sched.warmup_steps}steps
    project: Streaming-arabic-asr

With this script :

import glob
import os
import shutil

import lightning.pytorch as pl
from omegaconf import OmegaConf, open_dict

from nemo.collections.asr.models import ASRModel, EncDecHybridRNNTCTCBPEModel
from nemo.core.config import hydra_runner
from nemo.utils import logging
from nemo.utils.exp_manager import exp_manager
from nemo.utils.trainer_utils import resolve_trainer_cfg


@hydra_runner(
    config_path="../conf/conformer/hybrid_transducer_ctc/", config_name="conformer_hybrid_transducer_ctc_bpe"
)
def main(cfg):

    logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')

    trainer = pl.Trainer(**resolve_trainer_cfg(cfg.trainer))
    
    exp_manager(trainer, cfg.get("exp_manager", None))
    asr_model = EncDecHybridRNNTCTCBPEModel(cfg=cfg.model, trainer=trainer)

    

    # Initialize the weights of the model from another model, if provided via config
    asr_model.maybe_init_from_pretrained_checkpoint(cfg)

    trainer.fit(asr_model)

    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
        if asr_model.prepare_test(trainer):
            trainer.test(asr_model)


if __name__ == '__main__':
    main()  # noqa pylint: disable=no-value-for-parameter

Observations from Wandb after 44 with this config :