Cerebras-gpt-13B.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, GPT2Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_model(model_name):    \n",
    "    model = AutoModelForCausalLM.from_pretrained(\n",
    "        model_name,\n",
    "        torch_dtype=torch.float16,\n",
    "        device_map=\"auto\",\n",
    "        max_memory={\n",
    "            0: \"28GiB\",\n",
    "            \"cpu\": \"110GiB\", },\n",
    "        low_cpu_mem_usage=True,\n",
    "    ).eval()\n",
    "    \n",
    "    tokenizer = GPT2Tokenizer.from_pretrained(\n",
    "        model_name, )\n",
    "\n",
    "    return model, tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"cerebras/Cerebras-GPT-13B\"\n",
    "model, tokenizer = load_model(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is capital of USA?\n",
      "The capital of the United States of America (USA) is Washington, D.C."
     ]
    }
   ],
   "source": [
    "inputs = tokenizer(\n",
    "    \"\"\"What is capital of USA?\n",
    "    \"\"\",\n",
    "    truncation=True,\n",
    "    return_tensors=\"pt\",\n",
    ").to(\"cuda\")\n",
    "\n",
    "with torch.inference_mode():\n",
    "    completion = model.generate(\n",
    "        **inputs,\n",
    "        use_cache=True,\n",
    "        do_sample=True,\n",
    "        temperature=0.5,\n",
    "        no_repeat_ngram_size=1,\n",
    "        repetition_penalty=1.0,\n",
    "        top_p=0.92,\n",
    "        top_k=0,\n",
    "        max_new_tokens=128, )\n",
    "\n",
    "output = tokenizer.decode(completion.squeeze())\n",
    "print(output)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The temperature parameter in the generate method of Hugging Face's transformers library** - is used to control the degree of randomness and creativity in the generated text. It determines the range of probabilities that are considered when selecting the next word in the generated text.\n",
    "\n",
    "A lower temperature will result in the model generating more conservative and predictable text, while a higher temperature will lead to more creative and diverse output, but with a higher likelihood of generating nonsensical or irrelevant text.\n",
    "\n",
    "The temperature parameter is usually set to a value between 0 and 1, with 1 being the default value. A temperature of 0 would result in the model always generating the same output, while a very high temperature (e.g., 10) would result in the model generating extremely diverse and unpredictable output.\n",
    "\n",
    "---------------\n",
    "\n",
    "The **use_cache parameter in the generate method of Hugging Face's transformers** library controls whether or not the model's internal cache should be used during text generation.\n",
    "\n",
    "When use_cache is set to True, the model will use the cached values of the previous sequence generated to speed up the generation of subsequent sequences. This can be particularly useful when generating long sequences, as it can significantly reduce the amount of time required to generate each subsequent token.\n",
    "\n",
    "On the other hand, when use_cache is set to False, the model will not use its internal cache, and will instead generate each token in the sequence from scratch. This can be useful in situations where you want to ensure that the generated sequence is completely independent of any previously generated sequence.\n",
    "\n",
    "The default value of use_cache is True, meaning that the model will use its internal cache by default during text generation. However, depending on your use case, you may want to experiment with setting use_cache to False to see how it affects the quality and speed of the generated text."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}