Skip to content

Layer Types Configuration Intuition #15

@SpoSer23

Description

@SpoSer23

Hi!
I was inquiring about the layer_types config that you used for tinyllama_lckv.json, What was the intuition for this config based on the num_attention_heads and num_key_value_heads and num_hidden_layers for the layer_types and forward_passes and backward_passes ?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions