-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* docs: create QA_Script * docs: finalize QA Script template
- Loading branch information
Showing
1 changed file
with
210 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,210 @@ | ||
| # [Release Version] QA Script | ||
|
|
||
| **Release Version:** | ||
|
|
||
| **Operating System:** | ||
|
|
||
| --- | ||
|
|
||
| ## A. Installation, Update, and Uninstallation | ||
|
|
||
| ### 1. Users install app | ||
|
|
||
| - [ ] :key: Test for clear user installation instructions. | ||
| - [ ] :key: Verify that the installation path is correct for each OS. | ||
| - [ ] Check that the installation package is not corrupted and passes all security checks. | ||
| - [ ] Validate that the app is correctly installed in the default or user-specified directory. | ||
| - [ ] Ensure that all necessary dependencies are installed along with the app. | ||
| - [ ] :key: :rocket: Confirm that the app launches successfully after installation. | ||
|
|
||
| ### 2. Users update app | ||
|
|
||
| - [ ] :key: Test that the updated version includes the new features or fixes outlined in the update notes. | ||
| - [ ] :key: Validate that the update does not corrupt user data or settings. | ||
| - [ ] :key: Confirm that the app restarts or prompts the user to restart after an update. | ||
|
|
||
| ### 3. Users uninstall app | ||
|
|
||
| - [ ] :key: Check that the uninstallation process removes all components of the app from the system. | ||
| - [ ] Clean the Jan root directory and open the app to check if it creates all the necessary folders, especially models and extensions. | ||
| - [ ] When updating the app, check if the `/models` directory has any JSON files that change according to the update. | ||
| - [ ] Verify if updating the app also updates extensions correctly (test functionality changes; support notifications for necessary tests with each version related to extensions update). | ||
|
|
||
| ### 4. Users close app | ||
|
|
||
| - [ ] :key: Ensure that after closing the app, all models are unloaded. | ||
|
|
||
| ## B. Overview | ||
|
|
||
| ### 1. Users use shortcut keys | ||
|
|
||
| - [ ] :key: Test each shortcut key to confirm it works as described (My models, navigating, opening, closing, etc.). | ||
|
|
||
| ### 2. Users check the memory usage and CPU usage | ||
|
|
||
| - [ ] :key: Ensure that the interface presents the correct numbers for memory and CPU usage. | ||
|
|
||
| ### 3. Users check the `active model` | ||
|
|
||
| - [ ] :key: Verify that the app correctly displays the state of the loading model (e.g., loading, ready, error). | ||
| - [ ] :key: Confirm that the app allows users to switch between models if multiple are available. | ||
| - [ ] Check that the app provides feedback or instructions if the model fails to load. | ||
|
|
||
| ## C. Thread | ||
|
|
||
| ### 1. Users can chat with Jan, the default assistant | ||
|
|
||
| - [ ] Verify that the input box for messages is present and functional. | ||
| - [ ] :key: Check if typing a message and hitting `Send` results in the message appearing in the chat window. | ||
| - [ ] :key: Confirm that Jan, the default assistant, replies to user inputs. | ||
| - [ ] :key: Ensure that the conversation thread is maintained without any loss of data upon sending multiple messages. | ||
| - [ ] Test for the ability to send different types of messages (e.g., text, emojis, code blocks). | ||
| - [ ] :key: Validate the scroll functionality in the chat window for lengthy conversations. | ||
| - [ ] Check if the user can renew responses multiple times. | ||
| - [ ] Check if the user can copy the response. | ||
| - [ ] Check if the user can delete responses. | ||
| - [ ] :warning: Test if the user deletes the message midway, then the assistant stops that response. | ||
| - [ ] :key: Check the `clear message` button works. | ||
| - [ ] :key: Check the `delete entire chat` works. | ||
| - [ ] :warning: Check if deleting all the chat retains the system prompt. | ||
| - [ ] Check the output format of the AI (code blocks, JSON, markdown, ...). | ||
| - [ ] :key: Validate that there is appropriate error handling and messaging if the assistant fails to respond. | ||
| - [ ] Test assistant's ability to maintain context over multiple exchanges. | ||
| - [ ] :key: Check the `create new chat` button works correctly | ||
| - [ ] Confirm that by changing `models` mid-thread the app can still handle it. | ||
| - [ ] Check that by changing `instructions` mid-thread the app can still handle it. | ||
| - [ ] Check the `regenerate` button renews the response. | ||
| - [ ] Check the `Instructions` update correctly after the user updates it midway. | ||
|
|
||
| ### 2. Users can customize chat settings like model parameters via both the GUI & thread.json | ||
|
|
||
| - [ ] :key: Confirm that the chat settings options are accessible via the GUI. | ||
| - [ ] Test the functionality to adjust model parameters (e.g., Temperature, Top K, Top P) from the GUI and verify they are reflected in the chat behavior. | ||
| - [ ] :key: Ensure that changes can be saved and persisted between sessions. | ||
| - [ ] Validate that users can access and modify the thread.json file. | ||
| - [ ] :key: Check that changes made in thread.json are correctly applied to the chat session upon reload or restart. | ||
| - [ ] Verify if there is a revert option to go back to previous settings after changes are made. | ||
| - [ ] Test for user feedback or confirmation after saving changes to settings. | ||
| - [ ] Check the maximum and minimum limits of the adjustable parameters and how they affect the assistant's responses. | ||
| - [ ] :key: Validate user permissions for those who can change settings and persist them. | ||
| - [ ] :key: Ensure that users switch between threads with different models, the app can handle it. | ||
|
|
||
| ### 3. Users can click on a history thread | ||
|
|
||
| - [ ] Test the ability to click on any thread in the history panel. | ||
| - [ ] :key: Verify that clicking a thread brings up the past conversation in the main chat window. | ||
| - [ ] :key: Ensure that the selected thread is highlighted or otherwise indicated in the history panel. | ||
| - [ ] Confirm that the chat window displays the entire conversation from the selected history thread without any missing messages. | ||
| - [ ] :key: Check the performance and accuracy of the history feature when dealing with a large number of threads. | ||
| - [ ] Validate that historical threads reflect the exact state of the chat at that time, including settings. | ||
| - [ ] :key: :warning: Test the search functionality within the history panel for quick navigation. | ||
| - [ ] :key: Verify the ability to delete or clean old threads. | ||
| - [ ] :key: Confirm that changing the title of the thread updates correctly. | ||
|
|
||
| ### 4. Users can config instructions for the assistant. | ||
|
|
||
| - [ ] Ensure there is a clear interface to input or change instructions for the assistant. | ||
| - [ ] Test if the instructions set by the user are being followed by the assistant in subsequent conversations. | ||
| - [ ] :key: Validate that changes to instructions are updated in real time and do not require a restart of the application or session. | ||
| - [ ] :key: Confirm that the assistant's behavior changes in accordance with the new instructions provided. | ||
| - [ ] :key: Check for the ability to reset instructions to default or clear them completely. | ||
| - [ ] :key: Test the feature that allows users to save custom sets of instructions for different scenarios. | ||
| - [ ] Validate that instructions can be saved with descriptive names for easy retrieval. | ||
| - [ ] :key: Check if the assistant can handle conflicting instructions and how it resolves them. | ||
| - [ ] Ensure that instruction configurations are documented for user reference. | ||
|
|
||
| ## D. Hub | ||
|
|
||
| ### 1. Users can discover recommended models (Jan ships with a few preconfigured model.json files) | ||
|
|
||
| - [ ] :key: Verify that recommended models are displayed prominently on the main page. | ||
| - [ ] :key: Ensure that each model's recommendations are consistent with the user’s activity and preferences. | ||
| - [ ] Test the functionality of any filters that refine model recommendations. | ||
|
|
||
| ### 2. Users can download models suitable for their devices, e.g. compatible with their RAM | ||
|
|
||
| - [ ] Display the best model for their RAM at the top. | ||
| - [ ] :key: Ensure that models are labeled with RAM requirements and compatibility. | ||
| - [ ] :key: Validate that the download function is disabled for models that exceed the user's system capabilities. | ||
| - [ ] Test that the platform provides alternative recommendations for models not suitable due to RAM limitations. | ||
| - [ ] :key: Check the download model functionality and validate if the cancel download feature works correctly. | ||
|
|
||
| ### 3. Users can download models via a HuggingFace URL (coming soon) | ||
|
|
||
| - [ ] :key: Have the warning/status when the user enters the URL. | ||
| - [ ] :key: Check the progress bar reflects the right process. | ||
| - [ ] Validate the error handling for invalid or inaccessible URLs. | ||
|
|
||
| ### 4. Users can add a new model to the Hub | ||
|
|
||
| - [ ] :key: Have clear instructions so users can do their own. | ||
| - [ ] :key: Ensure the new model updates after restarting the app. | ||
| - [ ] Ensure it raises clear errors for users to fix the problem while adding a new model. | ||
|
|
||
| ### 5. Users can use the model as they want | ||
|
|
||
| - [ ] :key: Check `start` button response exactly what it does. | ||
| - [ ] :key: Check `stop` button response exactly what it does. | ||
| - [ ] :key: Check `delete` button response exactly what it does. | ||
| - [ ] Check if starting another model stops the other model entirely. | ||
| - [ ] Check the `Explore models` navigate correctly to the model panel. | ||
| - [ ] :key: Check when deleting a model it will delete all the files on the user's computer. | ||
| - [ ] The recommended tags should present right for the user's hardware. | ||
| - [ ] Assess that the descriptions of models are accurate and informative. | ||
|
|
||
| ## E. System Monitor | ||
|
|
||
| ### 1. Users can see disk and RAM utilization | ||
|
|
||
| - [ ] :key: Verify that the RAM and VRAM utilization graphs display accurate information. | ||
| - [ ] :key: Check that the CPU usage is accurately reported in real time. | ||
| - [ ] :key: Validate that the utilization percentages reflect the actual usage compared to the system's total available resources. | ||
| - [ ] :key: Ensure that the system monitors updates dynamically as the models run and stop. | ||
|
|
||
| ### 2. Users can start and stop models based on system health | ||
|
|
||
| - [ ] :key: Test the 'Start' action for a model to ensure it initiates and the system resource usage reflects this change. | ||
| - [ ] :key: Verify the 'Stop' action for a model to confirm it ceases operation and frees up the system resources accordingly. | ||
| - [ ] :key: Check the functionality that allows starting a model based on available system resources. | ||
| - [ ] :key: Validate that the system prevents starting a new model if it exceeds safe resource utilization thresholds. | ||
| - [ ] Ensure that the system provides warnings or recommendations when resource utilization is high before starting new models. | ||
| - [ ] Test the ease of accessing model settings from the system monitor for resource management. | ||
| - [ ] Confirm that any changes in model status (start/stop) are logged or reported to the user for transparency. | ||
|
|
||
| ## F. Settings | ||
|
|
||
| ### 1. Users can set color themes and dark/ light modes | ||
|
|
||
| - [ ] Verify that the theme setting is easily accessible in the `Appearance` tab. | ||
| - [ ] :key: Check that the theme change is reflected immediately upon selection. | ||
| - [ ] :key: Test the `Light`, `Dark`, and `System` theme settings to ensure they are functioning as expected. | ||
| - [ ] Confirm that the application saves the theme preference and persists it across sessions. | ||
| - [ ] Validate that all elements of the UI are compatible with the theme changes and maintain legibility and contrast. | ||
|
|
||
| ### 2. Users change the extensions | ||
|
|
||
| - [ ] Confirm that the `Extensions` tab lists all available plugins. | ||
| - [ ] :key: Test the toggle switch for each plugin to ensure it enables or disables the plugin correctly. | ||
| - [ ] Verify that plugin changes take effect without needing to restart the application unless specified. | ||
| - [ ] :key: Check that the plugin's status (`Installed the latest version`) updates accurately after any changes. | ||
| - [ ] Validate the `Manual Installation` process by selecting and installing a plugin file. | ||
| - [ ] Test for proper error handling and user feedback when a plugin installation fails. | ||
|
|
||
| ### 3. Users change the advanced settings | ||
|
|
||
| - [ ] :key: Test the `Experimental Mode` toggle to confirm it enables or disables experimental features as intended. | ||
| - [ ] :key: Check the functionality of `Open App Directory` to ensure it opens the correct folder in the system file explorer. | ||
| - [ ] Validate that changes in advanced settings are applied immediately or provide appropriate instructions if a restart is needed. | ||
| - [ ] Test the application's stability when experimental features are enabled. | ||
|
|
||
| ### 4. Users can add custom plugins via manual installation [TBU] | ||
|
|
||
| - [ ] Verify that the `Manual Installation` option is clearly visible and accessible in the `Extensions` section. | ||
| - [ ] Test the functionality of the `Select` button within the `Manual Installation` area. | ||
| - [ ] :warning: Check that the file picker dialog allows for the correct plugin file types (e.g., .tgz). | ||
| - [ ] :key: Validate that the selected plugin file installs correctly and the plugin becomes functional. | ||
| - [ ] Ensure that there is a progress indicator or confirmation message once the installation is complete. | ||
| - [ ] Confirm that if the installation is interrupted or fails, the user is given a clear error message. | ||
| - [ ] :key: Test that the application prevents the installation of incompatible or corrupt plugin files. | ||
| - [ ] :key: Check that the user can uninstall or disable custom plugins as easily as pre-installed ones. | ||
| - [ ] Verify that the application's performance remains stable after the installation of custom plugins. |