This AI Starter Kit is a demonstration of how to use Sambanova's models as coding assistants leveraging the Continue extension for VSCode and JetBrains. The Kit includes:
-
A configurable SambaNova Cloud - Continue connector. The connector generates answers from a SambaNova cloud hosted model.
-
A configurable SambaStudio - Continue connector. The connector generates answers from a deployed LLM.
-
An installation, setup, and usage guide.
For this starter kit, you will need access to an SambaNova Cloud account or a SambaStudio Environment.
You can use a LLM of your choice, either from SambaNova Cloud or from SambaStudio, but is highly recommended to use Meta-Llama-3-8B-Instruct
The next step sets you up to use one of the models available from SambaNova. It depends on whether you're a SambaNova customer who uses SambaStudio or you want to use the publicly available SambaNova Cloud.
-
Create a SambaNova Cloud account at SambaNova Cloud.
-
Get your SambaNova Cloud API key
To perform this setup, you must be a SambaNova customer with a SambaStudio account.
-
Log in to SambaStudio and get your API authorization key. The steps for getting this key are described here.
-
Select the LLM to use (e,g. CoE1.1 with Llama 3 8B instruct) and deploy an endpoint for inference. See the SambaStudio endpoint documentation.
you will need to install the Continue VSCode or JetBrains Extension. this can be done directly from the extensions/plugins section of you IDE searching Continue
After installing Continue you will need to do the basic setup
- First you will be prompted to select a model to use you can skip this step and close the *Continue window
After the basic installation it is needed to set the custom SambaNovaCloud or the SambaStudio Continue connectors
First you should modify the Continue config.ts file
-
Open the
config.tsfile in~/.continue/folder and replace the contents with the contents of the config.ts kit provided file. -
If you are using SambaNova Cloud: replace the sambanova_api_key variable with your previously generated SambaNova Cloud api key
For example, for an api key
123456ab-cdef-0123-4567-890abcdefupdate the first section in the config.ts file as://SambaNova Cloud usage const sambanova_api_key = "123456ab-cdef-0123-4567-890abcdef"
-
If you are using SambaStudio: replace the sambastudio_base_url, sambastudio_project_id, sambastudio_endpoint_id and sambastudio_api_key variables with your SambaStudio endpoint keys:
For example, for an endpoint with the URL
https://api-stage.sambanova.net/api/predict/generic/stream/12345678-9abc-def0-1234-56789abcdef0/456789ab-cdef-0123-4567-89abcdef012with api key123456ab-cdef-0123-4567-890abcdefupdate the first section in the config.ts file as://SambaStudio usage const sambastudio_base_url = "https://api-stage.sambanova.net"; const sambastudio_project_id = "12345678-9abc-def0-1234-56789abcdef0"; const sambastudio_endpoint_id = "456789ab-cdef-0123-4567-89abcdef012"; const sambastudio_api_key = "123456ab-cdef-0123-4567-890abcdef"
Then you should set the Continue config.json file. press ⌘+l this will open a new Continue session, Click in the gear ⚙ bottom right button, the the json file will open, replace the contents of this file with the contents of the config.json kit provided file.
You can interact with the LLM directly using ⌘+l command this will open a Continue session in a new extension window then ask anything to the to the model!
You can ask information of a selected snippet of code to the LLM directly selecting your text and then pressing ⌘+l, this will open a Continue session in a new extension window with the code snippet as context, then ask anything related with your code!
You can ask your LLM to modify your code, add functionalities, documentation etc, over a selected code snippet, for this first select the code snippet you want the model to modify, then press ⌘+i, this will open an input bar in the top of your IDE, then write your desired changes and press enter, this will generate the modified code for you and you can edit it, accept or reject the proposed changes.
You can ask the model to inspect your terminal error outputs to explain you the error and give you some suggestions, for this after getting an error in your terminal only press ⌘+shit+r, this will open a Continue session in a new extension window with the error explanation!
You can execute your custom commands/prompts selecting a code snippet and then pressing ⌘+l to open a new Continue session then write /<yourCommand> to generate, see how to create your custom commands here
See more about Continue extension usage here
The example template can be further customized based on the use case.
You can change the default SambaNova Cloud model or the default model used with SambaStudio CoE models modifying the model and expert selection in config.ts file:
-
If using SambaNova Cloud:
const sambanova_model = "llama3-405b";
-
If using SambaStudio CoE:
const sambastudio_use_coe = true; const sambastudio_coe_expert_name = "Meta-Llama-3-8B-Instruct";
Also you can modify model parameters modifying the body params of the SambaNova Cloud model and Sambastudio model in the SambastudioModel an SambaNovaCloudModel definitions of config.ts file.
You can add your custom commands adding them to the config.json file
A custom command should have the following structure
{
"name": "yourCommand",
"prompt": "{{{ input }}} \n\n custom prompt",
"description": "Description of your custom command"
}This kit work aims to show the integration of SambaNova Systems RDU acceleration models, The work here has been leveraged and adapted from the Continue documentation. The original docs can be found in https://docs.continue.dev/