args.norm_size in compute_bias.py

[Mogul5306]

What is this parameter (args.norm_size ) used for? In line #73, after adding up the three contribution values, why is this norm_size added.

[avaxiao]

Hello, the reason for normalizing the gradients is to determine their relative magnitudes. If we normalize them with softmax for each individual example, the ratios among the three features become similar. Therefore, it is necessary to define a norm_size for this purpose. However, it's worth exploring other normalization methods, as some of them might not require setting a norm_size parameter.