model_notebooks/sample-notebook-arcee-nova-on-sagemaker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Arcee-Nova on SageMaker through the Hugging Face hub\n",
    "\n",
    "This sample notebook shows you how to deploy [Arcee Nova](https://huggingface.co/arcee-ai/Arcee-Nova) using Amazon SageMaker. Arcee-Nova is an small language model built by [Arcee.ai](https://www.arcee.ai). Its performance approaches that of GPT-4 from May 2023, marking a significant milestone for open-source models. \n",
    "\n",
    "Nova is a merge of Qwen2-72B-Instruct with a custom model tuned on a generalist dataset mixture. Its maximum context size is 32K.\n",
    "\n",
    "## Use cases\n",
    "Arcee-Nova excels across a wide range of language tasks, demonstrating particular strength in:\n",
    "* **Reasoning**: Solving complex problems and drawing logical conclusions.\n",
    "* **Creative Writing**: Generating engaging and original content across various genres.\n",
    "* **Coding**: Assisting with programming tasks, from code generation to debugging.\n",
    "* **General Language Understanding**: Comprehending and generating human-like text in diverse contexts.\n",
    "\n",
    "Arcee-Nova can be applied to various business tasks:\n",
    "* **Customer Service**: Implement sophisticated chatbots and virtual assistants.\n",
    "* **Content Creation**: Generate high-quality written content for marketing and documentation.\n",
    "* **Software Development**: Accelerate coding processes and improve code quality.\n",
    "* **Data Analysis**: Enhance data interpretation and generate insightful reports.\n",
    "* **Research and Development**: Assist in literature reviews and hypothesis generation.\n",
    "* **Legal and Compliance**: Automate contract analysis and regulatory compliance checks.\n",
    "* **Education and Training**: Create adaptive learning systems and intelligent tutoring programs.\n",
    "\n",
    "## Pre-requisites\n",
    "1. Before running this notebook, please make sure you got this notebook from the model catalog on SageMaker AWS Management Console.\n",
    "1. **Note**: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.\n",
    "1. Ensure that IAM role used has **AmazonSageMakerFullAccess**.\n",
    "\n",
    "## Contents\n",
    "1. [Import dependencies](#1.-Import-dependencies)\n",
    "\n",
    "2. [Create an endpoint and perform real-time inference](#2.-Create-an-endpoint-and-perform-real-time-inference)\n",
    "    1. [Define the endpoint configuration](#A.-Define-the-endpoint-configuration)\n",
    "    2. [Create the endpoint](#B.-Create-the-endpoint)\n",
    "    3. [Define a test payload](#C.-Define-a-test-payload)\n",
    "    4. [Perform real-time inference](#D.-Perform-real-time-inference)\n",
    "    5. [Visualize output](#E.-Visualize-output)\n",
    "\n",
    "3. [Clean-up](#4.-Clean-up)\n",
    "    1. [Delete the model](#A.-Delete-the-model)\n",
    "    2. [Delete the endpoint](#B.-Delete-the-endpoint)\n",
    "\n",
    "\n",
    "## Usage instructions\n",
    "You can run this notebook one cell at a time (By using Shift+Enter for running a cell)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "pip install -qU boto3 sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import json\n",
    "import pprint\n",
    "\n",
    "import boto3\n",
    "import sagemaker\n",
    "from IPython.display import Markdown, display\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.huggingface import (HuggingFaceModel,\n",
    "                                   get_huggingface_llm_image_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "role = get_execution_role()\n",
    "sagemaker_session = sagemaker.Session()\n",
    "runtime_sm_client = boto3.client(\"runtime.sagemaker\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create an endpoint and perform real-time inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we're deploying Arcee-Nova on a SageMaker real-time endpoint hosted on a GPU instance. If you need general information on real-time inference with Amazon SageMaker, please refer to the SageMaker [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html).\n",
    "\n",
    "The endpoint runs a Hugging Face [Deep Learning Container](https://huggingface.co/docs/sagemaker/index), powered by the Hugging Face [Text Generation Inference](https://huggingface.co/docs/text-generation-inference/index) Server (TGI). TGI enables high-performance text generation for the most popular open-source language models. \n",
    "\n",
    "For flexibility, you can pick from two sample configurations, depending your use case and the instances types available to you. Please make sure to run just one of the configuration cells below.\n",
    "\n",
    "#### G5 endpoint\n",
    "\n",
    "The first configuration focuses on cost efficiency. It uses a [g5.12xlarge](https://aws.amazon.com/ec2/instance-types/g5/) instance. This instance has four NVDIA A10G GPUs, with a total of 96 GB of GPU RAM. Nova has 72 billion 16-bit parameters, which cannot fit in 96GB. Thus, we apply 4-bit quantization to the model, and shrink it by a factor of 4! \n",
    "\n",
    "For context size, we use the default value defined by the TGI inference server, i.e. 4K.\n",
    "\n",
    "#### P4 endpoint\n",
    "\n",
    "The second configuration focuses on performance. It uses a [p4d.24xlarge](https://aws.amazon.com/ec2/instance-types/p4/) instance. This instance has eight NVDIA A100 GPUs, with a total of 320 GB of GPU RAM. This is more than enough to load the full precision model, and we can also use a much larger context size: 32K.\n",
    "\n",
    "#### OpenAI compatibility\n",
    "\n",
    "For both configurations, we enable the [OpenAI Messages API](https://huggingface.co/docs/text-generation-inference/messages_api) available in TGI. This will alllow you to invoke the endpoint in the same way you would invoke an OpenAI model. Likewise, the output format will be identical to the OpenAI models. If that's not desirable, you can simply comment out the line setting `MESSAGES_API_ENABLED` to `true`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Define the endpoint configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"arcee-ai/Arcee-Nova\"\n",
    "endpoint_name_prefix = \"Arcee-Nova\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# g5 endpoint\n",
    "\n",
    "real_time_inference_instance_type = \"ml.g5.12xlarge\"\n",
    "\n",
    "model_environment = {\n",
    "    \"HF_MODEL_ID\": model_id,\n",
    "    \"SM_NUM_GPUS\": \"4\",\n",
    "    \"HF_MODEL_QUANTIZE\": \"bitsandbytes-nf4\",\n",
    "    \"MESSAGES_API_ENABLED\": \"true\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# p4 endpoint\n",
    "\n",
    "real_time_inference_instance_type = \"ml.p4d.24xlarge\"\n",
    "\n",
    "model_environment = {\n",
    "    \"HF_MODEL_ID\": model_id,\n",
    "    \"SM_NUM_GPUS\": \"8\",\n",
    "    \"MAX_INPUT_TOKENS\": \"16384\",\n",
    "    \"MAX_TOTAL_TOKENS\": \"32768\",\n",
    "    \"MESSAGES_API_ENABLED\": \"true\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Create the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a deployable model from the Hugging Face Hub model.\n",
    "model = HuggingFaceModel(\n",
    "    role=role,\n",
    "    image_uri=get_huggingface_llm_image_uri(\"huggingface\", version=\"2.2.0\"),\n",
    "    env=model_environment,\n",
    ")\n",
    "\n",
    "# create a unique endpoint name\n",
    "timestamp = \"{:%Y-%m-%d-%H-%M-%S}\".format(datetime.datetime.now())\n",
    "endpoint_name = f\"{endpoint_name_prefix}-{timestamp}\"\n",
    "print(f\"Deploying endpoint {endpoint_name}\")\n",
    "\n",
    "# deploy the model\n",
    "response = model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=real_time_inference_instance_type,\n",
    "    endpoint_name=endpoint_name,\n",
    "    model_data_download_timeout=3600,\n",
    "    container_startup_health_check_timeout=900,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the endpoint is in service, you will be able to perform real-time inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### C. Define a test payload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_sample_input = {\n",
    "    \"model\": \"tgi\",\n",
    "    \"messages\": [\n",
    "        {\"role\": \"system\", \"content\": \"You are a friendly and helpful AI assistant.\"},\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Suggest 5 names for a new neighborhood pet food store. Names should be short, fun, easy to remember, and respectful of pets. \\\n",
    "        Explain why customers would like them.\",\n",
    "        },\n",
    "    ],\n",
    "    \"max_tokens\": 1024,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### D. Perform real-time inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = runtime_sm_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType=\"application/json\",\n",
    "    Body=json.dumps(model_sample_input),\n",
    ")\n",
    "\n",
    "output = json.loads(response[\"Body\"].read().decode(\"utf8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### E. Visualize output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print the raw JSON output in OpenAI format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pprint.pprint(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also print the generated output with Markdown formatting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(Markdown(output[\"choices\"][0][\"message\"][\"content\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some more examples. Please feel free to tweak them and add your own!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = \"\"\"Please write a friendly marketing pitch for a new SaaS AI platform called Arcee Cloud.\n",
    "We will send this pitch by email to business and technical decision-makers, so make it sound exciting yet professional.\n",
    "The contact email is sales@arcee.ai. Feel free to use emojis as appropriate.\n",
    "Arcee Cloud makes it simple for enterprise users to tailor open-source small language models to their own domain knowledge,\n",
    "in order to build high-quality, cost-effective and secure AI solutions.\"\"\"\n",
    "\n",
    "model_sample_input = {\n",
    "    \"model\": \"tgi\",\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"You are a friendly and helpful Marketing Manager working at Arcee.ai.\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": prompt},\n",
    "    ],\n",
    "    \"max_tokens\": 1024,\n",
    "}\n",
    "\n",
    "response = runtime_sm_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType=\"application/json\",\n",
    "    Body=json.dumps(model_sample_input),\n",
    ")\n",
    "\n",
    "output = json.loads(response[\"Body\"].read().decode(\"utf8\"))\n",
    "display(Markdown(output[\"choices\"][0][\"message\"][\"content\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_sample_input = {\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"As a friendly technical assistant engineer, answer the question in detail.\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": \"Why are transformers better models than LSTM?\"},\n",
    "    ],\n",
    "    \"max_tokens\": 1024,\n",
    "}\n",
    "\n",
    "response = runtime_sm_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType=\"application/json\",\n",
    "    Body=json.dumps(model_sample_input),\n",
    ")\n",
    "\n",
    "output = json.loads(response[\"Body\"].read().decode(\"utf8\"))\n",
    "display(Markdown(output[\"choices\"][0][\"message\"][\"content\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_sample_input = {\n",
    "    \"model\": \"tgi\",\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"You are Darlene, a friendly and helpful salesperson \\\n",
    "        working at Crystal River Classic Bikes, a classic motorcycle dealership in central Florida.\",\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Using English, write a personalized customer email to get \\\n",
    "        them to sign up for a test ride on the new 2025 motorcycles that are visible at the dealership. \\\n",
    "        Tone should be warm and personal, make sure to weave in the customer information below. \\\n",
    "        Wyatt, your chief mechanic and road captain, has just won the 2024 State Award for Best Mechanic. \\\n",
    "        \\\n",
    "        Customer information:\\\n",
    "        - name: Julien \\\n",
    "        - last visit: 6 months ago for bike service \\\n",
    "        - Owns 2 bikes, a 2002 sporty bike and a 2007 cruiser \\\n",
    "        \",\n",
    "        },\n",
    "    ],\n",
    "    \"max_tokens\": 1024,\n",
    "}\n",
    "\n",
    "response = runtime_sm_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType=\"application/json\",\n",
    "    Body=json.dumps(model_sample_input),\n",
    ")\n",
    "\n",
    "output = json.loads(response[\"Body\"].read().decode(\"utf8\"))\n",
    "display(Markdown(output[\"choices\"][0][\"message\"][\"content\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have successfully performed a real-time inference, you do not need the endpoint any more. You can terminate the endpoint to avoid being charged."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Clean-up\n",
    "\n",
    "Please don't forget to run the cells below to delete all resources and avoid unecessary charges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Delete the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.sagemaker_session.delete_endpoint(endpoint_name)\n",
    "model.sagemaker_session.delete_endpoint_config(endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Delete the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.delete_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thank you for trying out Arcee-Nova on SageMaker. We have only scratched the surface of what you can do with this model.\n",
    "\n",
    "We'd be happy to hear from you, learn more about your use case, and help you build your next AI-driven solution. Please reach out to julien@arcee.ai."
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}