model_notebooks/sample-notebook-arcee-scribe-on-sagemaker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Arcee-Scribe on SageMaker through the Hugging Face Hub\n",
    "\n",
    "This sample notebook shows you how to deploy [Arcee Scribe](https://huggingface.co/arcee-ai/Arcee-Scribe) using Amazon SageMaker. Arcee-Scribe is a versatile chat model built by [Arcee.ai](https://www.arcee.ai). It exhibits strong reasoning abilities on the Big-Bench Hard (BBH) dataset, and we discovered it to be particularly adept at creative writing tasks. \n",
    "\n",
    "The model is a merge of InternLM-2.5-chat 7B with a custom InternLM finetune, incorporating both general and writing-specific data. Its maximum context size is 32K. \n",
    "\n",
    "## Use cases\n",
    "Arcee-Scribe excels across a wide range of language tasks, demonstrating particular strength in:\n",
    "* **Reasoning**: Solving complex problems and drawing logical conclusions.\n",
    "* **Creative Writing**: Generating engaging and original content across various genres.\n",
    "* **General Language Understanding**: Comprehending and generating human-like text in diverse contexts.\n",
    "\n",
    "Arcee-Nova can be applied to various business tasks:\n",
    "* **Customer Service**: Implement sophisticated chatbots and virtual assistants.\n",
    "* **Content Creation**: Generate high-quality written content for marketing and documentation.\n",
    "* **Data Analysis**: Enhance data interpretation and generate insightful reports.\n",
    "* **Research and Development**: Assist in literature reviews and hypothesis generation.\n",
    "* **Legal and Compliance**: Automate contract analysis and regulatory compliance checks.\n",
    "* **Education and Training**: Create adaptive learning systems and intelligent tutoring programs.\n",
    "\n",
    "## Pre-requisites\n",
    "1. Before running this notebook, please make sure you got this notebook from the model catalog on SageMaker AWS Management Console.\n",
    "1. **Note**: This notebook contains elements which render correctly in Jupyter interface. Open this notebook from an Amazon SageMaker Notebook Instance or Amazon SageMaker Studio.\n",
    "1. Ensure that IAM role used has **AmazonSageMakerFullAccess**.\n",
    "\n",
    "## Contents\n",
    "1. [Import dependencies](#1.-Import-dependencies)\n",
    "\n",
    "2. [Create an endpoint and perform real-time inference](#2.-Create-an-endpoint-and-perform-real-time-inference)\n",
    "    1. [Define the endpoint configuration](#A.-Define-the-endpoint-configuration)\n",
    "    2. [Create the endpoint](#B.-Create-the-endpoint)\n",
    "    3. [Define a test payload](#C.-Define-a-test-payload)\n",
    "    4. [Perform real-time inference](#D.-Perform-real-time-inference)\n",
    "    5. [Visualize output](#E.-Visualize-output)\n",
    "\n",
    "3. [Clean-up](#4.-Clean-up)\n",
    "    1. [Delete the model](#A.-Delete-the-model)\n",
    "    2. [Delete the endpoint](#B.-Delete-the-endpoint)\n",
    "    \n",
    "\n",
    "## Usage instructions\n",
    "You can run this notebook one cell at a time (By using Shift+Enter for running a cell)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh\n",
    "pip install -qU boto3 sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import json\n",
    "import pprint\n",
    "\n",
    "import boto3\n",
    "import sagemaker\n",
    "from IPython.display import Markdown, display\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.djl_inference.model import DJLModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "role = get_execution_role()\n",
    "sagemaker_session = sagemaker.Session()\n",
    "runtime_sm_client = boto3.client(\"runtime.sagemaker\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create an endpoint and perform real-time inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we're deploying Arcee-Scribe on a SageMaker real-time endpoint hosted on a GPU instance. If you need general information on real-time inference with Amazon SageMaker, please refer to the SageMaker [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html).\n",
    "\n",
    "The endpoint runs a DJLServing [Deep Learning Container](https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-models-frameworks-djl-serving.html), powered by vLLM. vLLM enables high-performance text generation for the most popular open-source language models. \n",
    "\n",
    "For flexibility, you can pick from two sample configurations, depending your use case and the instances types available to you. Please make sure to run just one of the configuration cells below.\n",
    "\n",
    "#### G5.2xlarge endpoint\n",
    "\n",
    "The first configuration focuses on cost efficiency. It uses a [g5.2xlarge](https://aws.amazon.com/ec2/instance-types/g5/) instance. This instance has a single NVDIA A10G GPU, with a total of 24 GB of GPU RAM. Scribe is small enough to fit and no quantization is required.\n",
    "\n",
    "For context size, we use 4K.\n",
    "\n",
    "#### G5.48xlarge endpoint\n",
    "\n",
    "The second configuration focuses on performance. It uses a [g5.48xlarge](https://aws.amazon.com/ec2/instance-types/g5/) instance. This instance has eight NVDIA A10G GPUs, with a total of 192 GB of GPU RAM. We can use the full context size: 32K.\n",
    "\n",
    "#### OpenAI compatibility\n",
    "\n",
    "For both configurations, the [OpenAI Messages API](https://huggingface.co/docs/text-generation-inference/messages_api) available in vLLM. This will allow you to optionally invoke the endpoint in the same way you would invoke an OpenAI model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Define the endpoint configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = \"arcee-ai/Arcee-Scribe\"\n",
    "endpoint_name_prefix = \"Arcee-Scribe\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# g5.2xlarge endpoint\n",
    "\n",
    "real_time_inference_instance_type = \"ml.g5.2xlarge\"\n",
    "\n",
    "model_environment = {\n",
    "    \"OPTION_DTYPE\": \"bf16\",\n",
    "    \"OPTION_MAX_MODEL_LEN\": \"4096\",\n",
    "    \"OPTION_TRUST_REMOTE_CODE\": \"true\",\n",
    "    \"OPTION_ROLLING_BATCH\": \"vllm\",\n",
    "    \"TENSOR_PARALLEL_DEGREE\": \"max\",\n",
    "    \"OPTION_MAX_ROLLING_BATCH_SIZE\": \"16\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# g5.48xlarge endpoint\n",
    "\n",
    "real_time_inference_instance_type = \"ml.g5.48xlarge\"\n",
    "\n",
    "model_environment = {\n",
    "    \"OPTION_DTYPE\": \"bf16\",\n",
    "    \"OPTION_MAX_MODEL_LEN\": \"32768\",\n",
    "    \"OPTION_TRUST_REMOTE_CODE\": \"true\",\n",
    "    \"OPTION_ROLLING_BATCH\": \"vllm\",\n",
    "    \"TENSOR_PARALLEL_DEGREE\": \"max\",\n",
    "    \"OPTION_MAX_ROLLING_BATCH_SIZE\": \"64\",\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Create the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a deployable model from the model package.\n",
    "model = DJLModel(model_id=model_id, role=role, env=model_environment)\n",
    "\n",
    "# create a unique endpoint name\n",
    "timestamp = \"{:%Y-%m-%d-%H-%M-%S}\".format(datetime.datetime.now())\n",
    "endpoint_name = f\"{endpoint_name_prefix}-{timestamp}\"\n",
    "print(f\"Deploying endpoint {endpoint_name}\")\n",
    "\n",
    "# deploy the model\n",
    "response = model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=real_time_inference_instance_type,\n",
    "    endpoint_name=endpoint_name,\n",
    "    model_data_download_timeout=3600,\n",
    "    container_startup_health_check_timeout=900,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the endpoint is in service, you will be able to perform real-time inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### C. Define a test payload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_sample_input = {\n",
    "    \"messages\": [\n",
    "        {\"role\": \"system\", \"content\": \"You are a friendly and helpful AI assistant.\"},\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Suggest 5 names for a new neighborhood pet food store. Names should be short, fun, easy to remember, and respectful of pets. \\\n",
    "        Explain why customers would like them.\",\n",
    "        },\n",
    "    ],\n",
    "    \"max_tokens\": 1024,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### D. Perform real-time inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = runtime_sm_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType=\"application/json\",\n",
    "    Body=json.dumps(model_sample_input),\n",
    ")\n",
    "\n",
    "output = json.loads(response[\"Body\"].read().decode(\"utf8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### E. Visualize output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print the raw JSON output in OpenAI format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pprint.pprint(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also print the generated output with Markdown formatting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(Markdown(output[\"choices\"][0][\"message\"][\"content\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some more examples. Please feel free to tweak them and add your own!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = \"\"\"Please write a friendly marketing pitch for a new SaaS AI platform called Arcee Cloud.\n",
    "We will send this pitch by email to business and technical decision-makers, so make it sound exciting yet professional.\n",
    "The contact email is sales@arcee.ai. Feel free to use emojis as appropriate.\n",
    "Arcee Cloud makes it simple for enterprise users to tailor open-source small language models to their own domain knowledge,\n",
    "in order to build high-quality, cost-effective and secure AI solutions.\"\"\"\n",
    "\n",
    "model_sample_input = {\n",
    "    \"messages\": [\n",
    "        {\"role\": \"system\", \"content\": \"You are a friendly and helpful AI assistant.\"},\n",
    "        {\"role\": \"user\", \"content\": prompt},\n",
    "    ],\n",
    "    \"max_tokens\": 1024,\n",
    "}\n",
    "\n",
    "response = runtime_sm_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType=\"application/json\",\n",
    "    Body=json.dumps(model_sample_input),\n",
    ")\n",
    "\n",
    "output = json.loads(response[\"Body\"].read().decode(\"utf8\"))\n",
    "display(Markdown(output[\"choices\"][0][\"message\"][\"content\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_sample_input = {\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"As a friendly technical assistant engineer, answer the question in detail.\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": \"Why are transformers better models than LSTM?\"},\n",
    "    ],\n",
    "    \"max_tokens\": 1024,\n",
    "}\n",
    "\n",
    "response = runtime_sm_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType=\"application/json\",\n",
    "    Body=json.dumps(model_sample_input),\n",
    ")\n",
    "\n",
    "output = json.loads(response[\"Body\"].read().decode(\"utf8\"))\n",
    "display(Markdown(output[\"choices\"][0][\"message\"][\"content\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_sample_input = {\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"You are Darlene, a friendly and helpful salesperson \\\n",
    "        working at Crystal River Classic Bikes, a classic motorcycle dealership in central Florida.\",\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"Using English, write a personalized customer email to get \\\n",
    "        them to sign up for a test ride on the new 2025 motorcycles that are visible at the dealership. \\\n",
    "        Tone should be warm and personal, make sure to weave in the customer information below. \\\n",
    "        Wyatt, your chief mechanic and road captain, has just won the 2024 State Award for Best Mechanic. \\\n",
    "        \\\n",
    "        Customer information:\\\n",
    "        - name: Julien \\\n",
    "        - last visit: 6 months ago for bike service \\\n",
    "        - Owns 2 bikes, a 2002 sporty bike and a 2007 cruiser \\\n",
    "        \",\n",
    "        },\n",
    "    ],\n",
    "    \"max_tokens\": 1024,\n",
    "}\n",
    "\n",
    "response = runtime_sm_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType=\"application/json\",\n",
    "    Body=json.dumps(model_sample_input),\n",
    ")\n",
    "\n",
    "output = json.loads(response[\"Body\"].read().decode(\"utf8\"))\n",
    "display(Markdown(output[\"choices\"][0][\"message\"][\"content\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have successfully performed a real-time inference, you do not need the endpoint any more. You can terminate the endpoint to avoid being charged."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Clean-up\n",
    "\n",
    "Please don't forget to run the cells below to delete all resources and avoid unecessary charges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Delete the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.sagemaker_session.delete_endpoint(endpoint_name)\n",
    "model.sagemaker_session.delete_endpoint_config(endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Delete the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.delete_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thank you for trying out Arcee-Scribe on SageMaker. We have only scratched the surface of what you can do with this model.\n",
    "\n",
    "We'd be happy to hear from you, learn more about your use case, and help you build your next AI-driven solution. Please reach out to julien@arcee.ai."
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}