Skip to content

Commit

Permalink
Update README
Browse files Browse the repository at this point in the history
  • Loading branch information
duzx16 committed Jul 26, 2023
1 parent bd7e7c4 commit 882956c
Showing 1 changed file with 15 additions and 12 deletions.
27 changes: 15 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,12 @@ ChatGLM2-6B 开源模型旨在与开源社区一起推动大模型技术发展
### MMLU

| Model | Average | STEM | Social Sciences | Humanities | Others |
| ----- | ----- | ---- | ----- | ----- | ----- |
| ----- |------| ---- |------|-------| ----- |
| ChatGLM-6B | 40.63 | 33.89 | 44.84 | 39.02 | 45.71 |
| ChatGLM2-6B (base) | 47.86 | 41.20 | 54.44 | 43.66 | 54.46 |
| ChatGLM2-6B | 45.46 | 40.06 | 51.61 | 41.23 | 51.24 |
| ChatGLM2-12B (base) | 56.18 | 48.18 | 65.13 | 52.58 | 60.93 |
| ChatGLM2-12B | 52.13 | 47.00 | 61.00 | 46.10 | 56.05 |

> Chat 模型使用 zero-shot CoT (Chain-of-Thought) 的方法测试,Base 模型使用 few-shot answer-only 的方法测试
Expand All @@ -67,12 +68,13 @@ ChatGLM2-6B 开源模型旨在与开源社区一起推动大模型技术发展
### GSM8K

| Model | Accuracy | Accuracy (Chinese)* |
|---------------------|----------| ----- |
| ChatGLM-6B | 4.82 | 5.85 |
| ChatGLM2-6B (base) | 32.37 | 28.95 |
| ChatGLM2-6B | 28.05 | 20.45 |
| Model | Accuracy | Accuracy (Chinese)* |
|--------------|----------| - |
| ChatGLM-6B | 4.82 | 5.85 |
| ChatGLM2-6B (base) | 32.37 | 28.95 |
| ChatGLM2-6B | 28.05 | 20.45 |
| ChatGLM2-12B (base) | 40.94 | 42.71 |
| ChatGLM2-12B | 38.13 | 23.43 |

> 所有模型均使用 few-shot CoT 的方法测试,CoT prompt 来自 http://arxiv.org/abs/2201.11903
>
Expand All @@ -81,12 +83,13 @@ ChatGLM2-6B 开源模型旨在与开源社区一起推动大模型技术发展

### BBH

| Model | Accuracy |
|---------------------|----------|
| ChatGLM-6B | 18.73 |
| ChatGLM2-6B (base) | 33.68 |
| ChatGLM2-6B | 30.00 |
| ChatGLM2-12B (base) | 36.02 |
| Model | Accuracy |
|--------------|-------|
| ChatGLM-6B | 18.73 |
| ChatGLM2-6B (base) | 33.68 |
| ChatGLM2-6B | 30.00 |
| ChatGLM2-12B (base) | 36.02 |
| ChatGLM2-12B | 39.98 |

> 所有模型均使用 few-shot CoT 的方法测试,CoT prompt 来自 https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main/cot-prompts
Expand Down

0 comments on commit 882956c

Please sign in to comment.