Hello! In line 74 in layers.py you have implemented bias as follows:
out = self.act_fn(torch.matmul(self.inp, self.weights))
if self.use_bias:
out = out + self.bias
Typically, one would have the bias inside the activation function, i.e. f(wx+b) as opposed to f(wx) + b (or, alternatively, wf(x)+b, but that requires additional changes). Is there any particular reason it was done this way? Did you experiment with the standard method?
Hello! In line 74 in layers.py you have implemented bias as follows:
Typically, one would have the bias inside the activation function, i.e. f(wx+b) as opposed to f(wx) + b (or, alternatively, wf(x)+b, but that requires additional changes). Is there any particular reason it was done this way? Did you experiment with the standard method?