Reactant array values are changing

[benjwweber]

I am investigating some gradients and noticed that

julia> gradients == gradients
false

and when I look at them I get different values from time to time

julia> gradients.head_ibot.chain.dense_1.bias
2048-element ConcreteIFRTArray{Float32,1} with "mhlo.sharding = {replicated}":
 59177.938
     0.0
    -5.183805f-34
     0.0
     ⋮
     0.0
 30460.812
     0.0

julia> gradients.head_ibot.chain.dense_1.bias
2048-element ConcreteIFRTArray{Float32,1} with "mhlo.sharding = {replicated}":
  59177.938
      0.0
 -45811.688
      0.0
      ⋮
      0.0
      0.0
      0.0

julia> gradients.head_ibot.chain.dense_1.bias
2048-element ConcreteIFRTArray{Float32,1} with "mhlo.sharding = {replicated}":
 -2.6467664f-30
  0.0
 -1.1724075f16
  0.0
  ⋮
  0.0
  0.0
  0.0

julia> gradients.head_ibot.chain.dense_1.bias
2048-element ConcreteIFRTArray{Float32,1} with "mhlo.sharding = {replicated}":
   1.0809131f31
   0.0
  -1.1724075f16
   0.0
   ⋮
 NaN
   1.54f-43
   0.0

I am trying to work out an MWE

[wsmoses]

yeah I think we need more informatiobn about this to help