cerebras.modelzoo.common.utils.model.attention_utils.FixedSparseAttentionBuilder#

class cerebras.modelzoo.common.utils.model.attention_utils.FixedSparseAttentionBuilder(num_heads, max_sequence_length, num_different_head_attn_configs=1, local_attn_ctx=4, global_attn_ctx=1, attention_type='unidirectional', horizontal_global_attention=False)[source]#

Bases: cerebras.modelzoo.common.utils.model.attention_utils.SparseAttentionBuilder

Methods

build_mask

get_pytorch_mask

set_global_attention

set_local_attention

trim_to_autoregressive_format

previous

cerebras.modelzoo.common.utils.model.attention_utils

next

cerebras.modelzoo.common.utils.model.attention_utils.SparseAttentionBuilder