Skip to content

Backpropagation implementation for Transformer #7

Description

@damoebe

Implement a backpropagation feature for each module in the transformer architechture.

Use the Backpropagation Interface featuring the backPropagate method to mark and define back-propagatable classes containing weights.

The Transformer classes like DecoderTransformer will pass a delta-gradient-matrix through the network. Each class implementing Backpropagation needs to calculate a delta matrix for the weights and the input. Also the backPropagate method should trigger a private updateWeights method using the weight delta gradient matrix and the learingrate to recalculate weights. To calculate module intern deltas just partial derive the modules outputFunction using the chainrule and the inserted delta with respect to the weights .

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions