Zyphra AI launches Zyphra Hierarchical Attention (ZHA): A path toward true secondary attention
Zyphra AI releases the sparse attention method Zyphra Hierarchical Attention (ZHA), using a tree-based indexer to make retrieval itself sparse, requiring only about 4 billion tokens to distill into a frozen pre-trained model.
Return to original text
A short introduction is retained here; the full text from the source website is not reposted. For specific wording and subsequent updates, please refer to the original article.
Open AIHOT original