MuP for Mamba #74

Open

opened

on May 30, 2024

Hello and thank you for sharing the repo. I'd like to know if muP would work out of the box with Mamba model or I would have to rescale some constants like in transformers attention_scores?

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests