grad is not finite

[stormand]

When fine-tuning ZipVoice with my custom Cantonese dataset, I encounter the following error: Grad Scale is small: 1.0339757e-25.
Additionally, after enabling inf-check, I get warnings indicating that all grad are not finite. What could be causing this issue?

[stormand]

Here is my finetune script:

#!/bin/bash

# This script is an example of fine-tuning ZipVoice on your custom datasets.

# Add project root to PYTHONPATH
export PYTHONPATH=../../:$PYTHONPATH

# Set bash to 'debug' mode, it will exit on:
# -e 'error', -u 'undefined variable', -o ... 'error in pipeline', -x 'print commands',
set -e
set -u
set -o pipefail

stage=5
stop_stage=5

# Number of jobs for data preparation
nj=20

# Whether the language of training data is one of Chinese and English
is_zh_en=0

# Language identifier, used when language is not Chinese or English
# see https://github.com/rhasspy/espeak-ng/blob/master/docs/languages.md
# Example of French: lang=fr
lang=yue

if [ $is_zh_en -eq 1 ]; then
      tokenizer=emilia
else
      tokenizer=espeak
      [ "$lang" = "default" ] && { echo "Error: lang is not set!" >&2; exit 1; }
fi

# You can set `max_len` according to statistics from the command 
# `lhotse cut describe data/fbank/custom_cuts_train.jsonl.gz`.
# Set `max_len` to 99% duration.

# Maximum length (seconds) of the training utterance, will filter out longer utterances
max_len=20

# Download directory for pre-trained models
download_dir=download/

# We suppose you have two TSV files: "data/raw/custom_train.tsv" and 
# "data/raw/custom_dev.tsv", where "custom" is your dataset name, 
# "train"/"dev" are used for training and validation respectively.

# Each line of the TSV files should be in one of the following formats:
# (1) `{uniq_id}\t{text}\t{wav_path}` if the text corresponds to the full wav,
# (2) `{uniq_id}\t{text}\t{wav_path}\t{start_time}\t{end_time}` if text corresponds
#     to part of the wav. The start_time and end_time specify the start and end
#     times of the text within the wav, which should be in seconds.
# > Note: {uniq_id} must be unique for each line.
for subset in train dev;do
      file_path=data/raw/custom_${subset}.tsv
      [ -f "$file_path" ] || { echo "Error: expect $file_path !" >&2; exit 1; }
done

### Prepare the training data (1 - 4)

if [ ${stage} -le 1 ] && [ ${stop_stage} -ge 1 ]; then
      echo "Stage 1: Prepare manifests for custom dataset from tsv files"

      for subset in train dev;do
            python3 -m zipvoice.bin.prepare_dataset \
                  --tsv-path data/raw/custom_${subset}.tsv \
                  --prefix custom-finetune \
                  --subset raw_${subset} \
                  --num-jobs ${nj} \
                  --output-dir data/manifests
      done
      # The output manifest files are "data/manifests/custom-finetune_cuts_raw_train.jsonl.gz".
      # and "data/manifests/custom-finetune_cuts_raw_dev.jsonl.gz".
fi


if [ ${stage} -le 2 ] && [ ${stop_stage} -ge 2 ]; then
      echo "Stage 2: Add tokens to manifests"
      # For "emilia" and "espeak" tokenizers, it's better to prepare the tokens 
      # before training. Otherwise, the on-the-fly tokenization can significantly
      # slow down the training.
      for subset in train dev;do
            python3 -m zipvoice.bin.prepare_tokens \
                  --input-file data/manifests/custom-finetune_cuts_raw_${subset}.jsonl.gz \
                  --output-file data/manifests/custom-finetune_cuts_${subset}.jsonl.gz \
                  --tokenizer ${tokenizer} \
                  --lang ${lang}
      done
      # The output manifest files are "data/manifests/custom-finetune_cuts_train.jsonl.gz".
      # and "data/manifests/custom-finetune_cuts_dev.jsonl.gz".
fi

if [ $stage -le 3 ] && [ $stop_stage -ge 3 ]; then
      echo "Stage 3: Compute Fbank for custom dataset"
      # You can skip this step and use `--on-the-fly-feats 1` in training stage
      for subset in train dev; do
            python3 -m zipvoice.bin.compute_fbank \
                  --source-dir data/manifests \
                  --dest-dir data/fbank \
                  --dataset custom-finetune \
                  --subset ${subset} \
                  --num-jobs ${nj}
      done
fi

if [ ${stage} -le 4 ] && [ ${stop_stage} -ge 4 ]; then
      echo "Stage 4: Download pre-trained model, tokens file, and model config"
      # Uncomment this line to use HF mirror
      # export HF_ENDPOINT=https://hf-mirror.com
      hf_repo=k2-fsa/ZipVoice
      mkdir -p ${download_dir}
      for file in model.pt tokens.txt model.json; do
            huggingface-cli download \
                  --local-dir ${download_dir} \
                  ${hf_repo} \
                  zipvoice/${file}
      done
fi

### Training ZipVoice (5 - 6)

if [ ${stage} -le 5 ] && [ ${stop_stage} -ge 5 ]; then
      echo "Stage 5: Fine-tune the ZipVoice model"

      [ -z "$max_len" ] && { echo "Error: max_len is not set!" >&2; exit 1; }

      python3 -m zipvoice.bin.train_zipvoice \
            --world-size 1 \
            --use-fp16 1 \
            --finetune 1 \
            --base-lr 0.0001 \
            --num-iters 10000 \
            --save-every-n 1000 \
            --max-duration 100 \
            --max-len ${max_len} \
            --model-config ${download_dir}/zipvoice/model.json \
            --checkpoint ${download_dir}/zipvoice/model.pt \
            --tokenizer ${tokenizer} \
            --lang ${lang} \
            --token-file ${download_dir}/zipvoice/tokens.txt \
            --dataset custom \
            --train-manifest data/fbank/custom-finetune_cuts_train.jsonl.gz \
            --dev-manifest data/fbank/custom-finetune_cuts_dev.jsonl.gz \
            --exp-dir exp/zipvoice_finetune \
			--inf-check true

fi

[stormand]

here is the diagnostics output:

module=<top-level>.grad, type=ZipVoice, dim=0, size=1, value [2.2e+04], norm=2.2e+04, mean=2.15e+04, rms=2.15e+04
module=<top-level>.grad, type=ZipVoice, dim=0, size=1, abs [2.2e+04], mean=2.15e+04, rms=2.15e+04
module=<top-level>.grad, type=ZipVoice, dim=0, size=1, max [6.6e+04], mean=6.55e+04, rms=6.55e+04
module=<top-level>.grad, type=ZipVoice, dim=0, size=1, min [2e+03], mean=2.05e+03, rms=2.05e+03
module=<top-level>.output, type=ZipVoice, dim=0, size=1, value [0.16], norm=0.16, mean=0.158, rms=0.158
module=<top-level>.output, type=ZipVoice, dim=0, size=1, abs [0.16], mean=0.158, rms=0.158
module=<top-level>.output, type=ZipVoice, dim=0, size=1, max [0.24], mean=0.24, rms=0.24
module=<top-level>.output, type=ZipVoice, dim=0, size=1, min [0.084], mean=0.0836, rms=0.0836
module=embed.grad, type=Embedding, dim=0, size=4..12, abs percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=0, size=4..12, max percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=0, size=4..12, min percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=0, size=4..12, positive percentiles: [0 0 0 0 0 0 0 0 0 0 0], mean=0, rms=0
module=embed.grad, type=Embedding, dim=0, size=4..12, value percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=0, size=4..12, rms percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=0, size=11, rms-sort percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=0, size=4..12, stddev percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, abs percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, max percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, min percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, positive percentiles: [0 0 0 0 0 0 0 0 0 0 0], mean=0, rms=0
module=embed.grad, type=Embedding, dim=1, size=181..801, value percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, rms percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=11, rms-sort percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, abs percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, max percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, min percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, positive percentiles: [0 0 0 0 0 0 0 0 0 0 0], mean=0, rms=0
module=embed.grad, type=Embedding, dim=1, size=181..801, value percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, rms percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=11, rms-sort percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=1, size=181..801, stddev percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=2, size=192, abs percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=2, size=192, max percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=2, size=192, min percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=2, size=192, positive percentiles: [0 0 0 0 0 0 0 0 0 0 0], mean=0, rms=0
module=embed.grad, type=Embedding, dim=2, size=192, value percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=2, size=192, rms percentiles: [nan nan nan nan nan nan nan nan nan nan nan], norm=nan, mean=nan, rms=nan
module=embed.grad, type=Embedding, dim=2, size=11, rms-sort percentiles: [nan nan nan nan nan nan nan nan nan nan nan], mean=nan, rms=nan