-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into docs/fix-error-link
- Loading branch information
Showing
67 changed files
with
844 additions
and
493 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| ## Requirements for running Jan App in GPU mode on Windows and Linux | ||
| - You must have an NVIDIA driver that supports CUDA 11.4 or higher. Refer [here](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver). | ||
| To check if the NVIDIA driver is installed, open PowerShell or Terminal and enter the following command: | ||
| ```bash | ||
| nvidia-smi | ||
| ``` | ||
| If you see a result similar to the following, you have successfully installed the NVIDIA driver: | ||
| ```bash | ||
| +-----------------------------------------------------------------------------+ | ||
| | NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 | | ||
| |-------------------------------+----------------------+----------------------+ | ||
| | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | ||
| | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | ||
| | | | MIG M. | | ||
| |===============================+======================+======================| | ||
| | 0 NVIDIA GeForce ... Off | 00000000:01:00.0 On | N/A | | ||
| | 0% 51C P8 10W / 170W | 364MiB / 7982MiB | 0% Default | | ||
| | | | N/A | | ||
| +-------------------------------+----------------------+----------------------+ | ||
| ``` | ||
|
|
||
| - You must have CUDA 11.4 or higher. | ||
| To check if CUDA is installed, open PowerShell or Terminal and enter the following command: | ||
| ```bash | ||
| nvcc --version | ||
| ``` | ||
| If you see a result similar to the following, you have successfully installed CUDA: | ||
| ```bash | ||
| nvcc: NVIDIA (R) Cuda compiler driver | ||
| Cuda compilation tools, release 11.4, V11.4.100 | ||
| Build cuda_11.4.r11.4/compiler.30033411_0 | ||
| ``` | ||
|
|
||
| - Specifically for Linux, you will need to have a CUDA compatible driver, refer [here](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver), and you must add the `.so` libraries of CUDA and the CUDA compatible driver to the `LD_LIBRARY_PATH` environment variable, refer [here](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions). | ||
|
|
||
| ## How to switch mode CPU/GPU Jan app | ||
|
|
||
| By default, Jan app will run in CPU mode. When starting Jan app, the program will automatically check if your computer meets the requirements to run in GPU mode. If it does, we will automatically enable GPU mode and pick the GPU has highest VGRAM for you (feature allowing users to select one or more GPU devices for use - currently in planning). You can check whether you are using CPU mode or GPU mode in the settings/advance section of Jan app. (see image below).  | ||
|
|
||
| If you have GPU mode but it is not enabled by default, the following possibilities may exist, you can follow the next steps to fix the error: | ||
|
|
||
| 1. You have not installed the NVIDIA driver, refer to the NVIDIA driver that supports CUDA 11.4 [here](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver). | ||
|
|
||
| 2. You have not installed the CUDA toolkit or your CUDA toolkit is not compatible with the NVIDIA driver, refer to CUDA compatibility [here](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver). | ||
|
|
||
| 3. You have not installed a CUDA compatible driver, refer [here](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver), and you must add the `.so` libraries of CUDA and the CUDA compatible driver to the `LD_LIBRARY_PATH` environment variable, refer [here](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions). For Windows, add the `.dll` libraries of CUDA and the CUDA compatible driver to the `PATH` environment variable. Usually, when installing CUDA on Windows, this environment variable is automatically added, but if you do not see it, you can add it manually by referring [here](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html#environment-setup). | ||
|
|
||
| ## To check the current GPU-related settings that Jan app has detected, you can go to the Settings/Advanced section as shown in the image below. | ||
|  | ||
|  | ||
|  | ||
|  | ||
|
|
||
| When you have an issue with GPU mode, share the `settings.json` with us will help us to solve the problem faster. | ||
|
|
||
| ## Tested on | ||
|
|
||
| - Windows 11 Pro 64-bit, NVIDIA GeForce RTX 4070ti GPU, CUDA 12.2, NVIDIA driver 531.18 | ||
| - Ubuntu 22.04 LTS, NVIDIA GeForce RTX 4070ti GPU, CUDA 12.2, NVIDIA driver 545 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| import { FileManagerRoute } from '../../../api' | ||
| import { HttpServer } from '../../index' | ||
|
|
||
| export const fsRouter = async (app: HttpServer) => { | ||
| app.post(`/app/${FileManagerRoute.syncFile}`, async (request: any, reply: any) => {}) | ||
|
|
||
| app.post(`/app/${FileManagerRoute.getUserSpace}`, async (request: any, reply: any) => {}) | ||
|
|
||
| app.post(`/app/${FileManagerRoute.getResourcePath}`, async (request: any, reply: any) => {}) | ||
|
|
||
| app.post(`/app/${FileManagerRoute.fileStat}`, async (request: any, reply: any) => {}) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| export type FileStat = { | ||
| isDirectory: boolean | ||
| size: number | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| --- | ||
| title: Hardware Requirements | ||
| description: Jan is a ChatGPT-alternative that runs on your own computer, with a local API server. | ||
| keywords: | ||
| [ | ||
| Jan AI, | ||
| Jan, | ||
| ChatGPT alternative, | ||
| local AI, | ||
| private AI, | ||
| conversational AI, | ||
| no-subscription fee, | ||
| large language model, | ||
| ] | ||
| --- | ||
|
|
||
| Jan is designed to be lightweight and able to run Large Language Models (LLMs) out-of-the-box. | ||
|
|
||
| The current download size is less than 150 MB and has a disk space of ~300 MB. | ||
|
|
||
| To ensure optimal performance, please see the following system requirements: | ||
|
|
||
| ## Disk Space | ||
|
|
||
| - Minimum requirement | ||
| - At least 5 GB of free disk space is required to accommodate the download, storage, and management of open-source LLM models. | ||
| - Recommended | ||
| - For an optimal experience and to run most available open-source LLM models on Jan, it is recommended to have 10 GB of free disk space. | ||
|
|
||
| ## RAM and GPU VRAM | ||
|
|
||
| The amount of RAM on your system plays a crucial role in determining the size and complexity of LLM models you can effectively run. Jan can be utilized on traditional computers where RAM is a key resource. For enhanced performance, Jan also supports GPU acceleration, utilizing the VRAM of your graphics card. | ||
|
|
||
| ## Best Models for your V/RAM | ||
|
|
||
| The RAM and GPU VRAM requirements are dependent on the size and complexity of the LLM models you intend to run. The following are some general guidelines to help you determine the amount of RAM or VRAM you need to run LLM models on Jan | ||
|
|
||
| - `8 GB of RAM`: Suitable for running smaller models like 3B models or quantized 7B models | ||
| - `16 GB of RAM (recommended)`: This is considered the "minimum usable models" threshold, particularly for 7B models (e.g Mistral 7B, etc) | ||
| - `Beyond 16GB of RAM`: Required for handling larger and more sophisticated model, such as 70B models. | ||
|
|
||
| ## Architecture | ||
|
|
||
| Jan is designed to run on muptiple architectures, versatility and widespread usability. The supported architectures include: | ||
|
|
||
| ### CPU Support | ||
|
|
||
| - `x86`: Jan is well-suited for systems with x86 architecture, which is commonly found in traditional desktops and laptops. It ensures smooth performance on a variety of devices using x86 processors. | ||
| - `ARM`: Jan is optimized to run efficiently on ARM-based systems, extending compatibility to a broad range of devices using ARM processors. | ||
|
|
||
| ### GPU Support | ||
|
|
||
| - `NVIDIA` | ||
| - `AMD` | ||
| - `ARM64 Mac` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.