Implement a backpropagation feature for each module in the transformer architechture.
Use the Backpropagation Interface featuring the backPropagate method to mark and define back-propagatable classes containing weights.
The Transformer classes like DecoderTransformer will pass a delta-gradient-matrix through the network. Each class implementing Backpropagation needs to calculate a delta matrix for the weights and the input. Also the backPropagate method should trigger a private updateWeights method using the weight delta gradient matrix and the learingrate to recalculate weights. To calculate module intern deltas just partial derive the modules outputFunction using the chainrule and the inserted delta with respect to the weights .
Implement a backpropagation feature for each module in the transformer architechture.
Use the
BackpropagationInterface featuring thebackPropagatemethod to mark and define back-propagatable classes containing weights.The Transformer classes like
DecoderTransformerwill pass a delta-gradient-matrix through the network. Each class implementingBackpropagationneeds to calculate a delta matrix for theweightsand the input. Also thebackPropagatemethod should trigger a privateupdateWeightsmethod using the weight delta gradient matrix and the learingrate to recalculateweights. To calculate module intern deltas just partial derive the modules outputFunction using the chainrule and the inserted delta with respect to theweights.