exllamaとは

exllamaはローダーのライブラリであり、Llamaベースの大規模言語モデルの推論が早くなる模様

リンク

Colab
github

準備

Google Colabを開き、メニューから「ランタイム→ランタイムのタイプを変更」でランタイムを「GPU」に変更します。

環境構築

インストール手順です。

!git clone https://github.com/oobabooga/text-generation-webui
%cd text-generation-webui
!pip install -r requirements.txt

!pip install safetensors sentencepiece ninja
!mkdir /content/text-generation-webui/repositories
%cd repositories
!git clone https://github.com/turboderp/exllama

推論

(1)モデルのダウンロード

%cd /content/text-generation-webui
!python download-model.py TheBloke/vicuna-13b-v1.3-GPTQ

(2)webuiの起動

!python3 server.py --model TheBloke_vicuna-13b-v1.3-GPTQ --chat --share --auto-devices --loader exllama

chat画面

速度

Output generated in 6.40 seconds (8.28 tokens/s, 53 tokens, context 51, seed 2028799410)
Output generated in 3.31 seconds (19.31 tokens/s, 64 tokens, context 141, seed 350957030)
Output generated in 9.41 seconds (21.25 tokens/s, 200 tokens, context 141, seed 1325683099)
Output generated in 1.77 seconds (20.29 tokens/s, 36 tokens, context 341, seed 356656391)
Output generated in 4.57 seconds (18.61 tokens/s, 85 tokens, context 403, seed 1443216874)

体感としてもめちゃくちゃ早い。

Advanced Application

一応vicuna13B-v1.3の検証も兼ねてリオネルメッシみたいに上手くなりたいので聞いてみました
output

Sure! Here are some tips on how to play soccer like Lionel Messi:

1. Practice dribbling: One of Messi's signature skills is his incredible ball control and dribbling ability. To work on this skill, practice dribbling with both feet in small spaces such as cones or around obstacles.
2. Improve your vision: Messi has an amazing ability to see the field and find open teammates. Work on improving your own field vision by practicing passing and moving without the ball.
3. Develop your shooting technique: Messi is known for his powerful shots from long range. Work on developing your own shot by practicing different techniques such as curling the ball into the top corner or hitting it low past the goalkeeper.
4. Train your agility and quickness: Messi is also known for his speed and agility on the pitch. Incorporate exercises that focus on improving your quickness and agility, such as ladder drills and cone drills.
5. Play with confidence: Messi plays with a lot of confidence and belief in himself. Try to emulate this mindset when you step onto the field by visualizing yourself scoring goals and making great passes.
6. Study the game: Watch videos of Messi playing and take note of his movements, decisions, and actions on the field. This can help you understand his style of play and learn from him.

いや本当にchatGPTww

最後に

今回は爆速レスポンス体験としてのexllama + GPTQを試してみました。本当に早くなる、、今回はtext-generation-webui上でしか試せませんでしたが、普通に推論として動く方法も模索していきたい。

今後ともLLM, Diffusion model, Image Analysis, 3Dに関連する試した記事を投稿していく予定なのでよろしくお願いします。