Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
  • Loading branch information
Hanyuezhuohua committed Dec 9, 2024
1 parent 762d9af commit 67584ef
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion index.html
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ <h2 class="title is-3">Introduction</h2>
<p>
We introduce <u>S</u>tructured <u>S</u>parse
<u>F</u>ine-<u>T</u>uning (<b>S<sup>2</sup>FT</b>), the first PEFT method for LLMs that achieves high
quality, efficient training, and scalable serving simutaneously.
quality, efficient training, and scalable serving simultaneously.
S<sup>2</sup>FT accomplishes this by <b>“selecting sparsely and computing densely”</b>. It selects a few
heads and channels in the MHA and FFN modules for each Transformer block,
respectively. Next, it co-permutes weight matrices on both sides of the coupled
Expand Down

0 comments on commit 67584ef

Please sign in to comment.