[E-12]Project_Transformer_Chatbot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "exotic-lodge",
   "metadata": {},
   "source": [
    "#  프로젝트: 한국어 데이터로 챗봇 만들기\n",
    "\n",
    "#### 배경\n",
    "- transformer 챗봇 모델을 만들어서 영어 데이터로 학습 후 예측 문장을 만들어봤다.\n",
    "- 다른 언어인 한국어로 transformer 챗봇 모델을 만들어보자."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "local-technique",
   "metadata": {},
   "source": [
    "## Step 1. 데이터 수집하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "harmful-savings",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import os\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dietary-whole",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12시 땡!</td>\n",
       "      <td>하루가 또 가네요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1지망 학교 떨어졌어</td>\n",
       "      <td>위로해 드립니다.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3박4일 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3박4일 정도 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PPL 심하네</td>\n",
       "      <td>눈살이 찌푸려지죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Q            A  label\n",
       "0           12시 땡!   하루가 또 가네요.      0\n",
       "1      1지망 학교 떨어졌어    위로해 드립니다.      0\n",
       "2     3박4일 놀러가고 싶다  여행은 언제나 좋죠.      0\n",
       "3  3박4일 정도 놀러가고 싶다  여행은 언제나 좋죠.      0\n",
       "4          PPL 심하네   눈살이 찌푸려지죠.      0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 데이터 가져오기\n",
    "path = os.getenv(\"HOME\") + '/aiffel/transformer_chatbot/data/ChatbotData .csv'\n",
    "df = pd.read_csv(path)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "crucial-organization",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    5290\n",
       "1    3570\n",
       "2    2963\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# label은 무엇을 뜻하는 것일까?\n",
    "df['label'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "running-notion",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12시 땡!</td>\n",
       "      <td>하루가 또 가네요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1지망 학교 떨어졌어</td>\n",
       "      <td>위로해 드립니다.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3박4일 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3박4일 정도 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PPL 심하네</td>\n",
       "      <td>눈살이 찌푸려지죠.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>SD카드 망가졌어</td>\n",
       "      <td>다시 새로 사는 게 마음 편해요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>SD카드 안돼</td>\n",
       "      <td>다시 새로 사는 게 마음 편해요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Q                   A  label\n",
       "0           12시 땡!          하루가 또 가네요.      0\n",
       "1      1지망 학교 떨어졌어           위로해 드립니다.      0\n",
       "2     3박4일 놀러가고 싶다         여행은 언제나 좋죠.      0\n",
       "3  3박4일 정도 놀러가고 싶다         여행은 언제나 좋죠.      0\n",
       "4          PPL 심하네          눈살이 찌푸려지죠.      0\n",
       "5        SD카드 망가졌어  다시 새로 사는 게 마음 편해요.      0\n",
       "6          SD카드 안돼  다시 새로 사는 게 마음 편해요.      0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5290</th>\n",
       "      <td>1000일 만난 여자친구와 이별</td>\n",
       "      <td>더 오래 만날 사람 만날 거예요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5291</th>\n",
       "      <td>10년 연애. 헤어졌습니다.</td>\n",
       "      <td>더 공허함이 크시겠네요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5292</th>\n",
       "      <td>10년 연애사 되돌아보니 다 부질없네</td>\n",
       "      <td>더 좋은 사람 만나실 거예요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5293</th>\n",
       "      <td>10년 연예의끝</td>\n",
       "      <td>더 마음이 허하겠어요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5294</th>\n",
       "      <td>10년만나다 헤어지네</td>\n",
       "      <td>충분히 슬퍼하고 충분히 아파하다가 이겨내세요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5295</th>\n",
       "      <td>10년이라는 시간이 참 무색하다 싶네</td>\n",
       "      <td>함께 걸은 길을 혼자 돌아가고 계시겠네요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5296</th>\n",
       "      <td>10월의 마지막밤 . 더 보고싶네 그사람</td>\n",
       "      <td>이별은 보고 싶은 마음을 참는 과정이죠.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           Q                          A  label\n",
       "5290       1000일 만난 여자친구와 이별         더 오래 만날 사람 만날 거예요.      1\n",
       "5291         10년 연애. 헤어졌습니다.              더 공허함이 크시겠네요.      1\n",
       "5292    10년 연애사 되돌아보니 다 부질없네           더 좋은 사람 만나실 거예요.      1\n",
       "5293                10년 연예의끝               더 마음이 허하겠어요.      1\n",
       "5294             10년만나다 헤어지네  충분히 슬퍼하고 충분히 아파하다가 이겨내세요.      1\n",
       "5295    10년이라는 시간이 참 무색하다 싶네    함께 걸은 길을 혼자 돌아가고 계시겠네요.      1\n",
       "5296  10월의 마지막밤 . 더 보고싶네 그사람     이별은 보고 싶은 마음을 참는 과정이죠.      1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8860</th>\n",
       "      <td>짝사랑만큼 고통스러운 건 없겠지.</td>\n",
       "      <td>짝사랑 만큼 감정소모가 큰 건 없을 거예요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8861</th>\n",
       "      <td>1년 넘게 만났는데 지금도 불타올라</td>\n",
       "      <td>정열적인 사랑을 하고 있나봐요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8862</th>\n",
       "      <td>1년 동거 중인데 계속 좋아</td>\n",
       "      <td>서로 깊게 알게되면서 더 좋아졌나봅니다.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8863</th>\n",
       "      <td>1년 동거하고 결혼했어</td>\n",
       "      <td>축하합니다!</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8864</th>\n",
       "      <td>1년 만났는데도 그 사람에 대해 잘 모르겠어</td>\n",
       "      <td>더 만나보세요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8865</th>\n",
       "      <td>1년반 만났는데 결혼하자고 해도 될까?</td>\n",
       "      <td>기다리고 있을지도 몰라요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8866</th>\n",
       "      <td>1년째 동거중이야</td>\n",
       "      <td>서로 알아가는 단계인가봐요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             Q                         A  label\n",
       "8860        짝사랑만큼 고통스러운 건 없겠지.  짝사랑 만큼 감정소모가 큰 건 없을 거예요.      2\n",
       "8861       1년 넘게 만났는데 지금도 불타올라         정열적인 사랑을 하고 있나봐요.      2\n",
       "8862           1년 동거 중인데 계속 좋아    서로 깊게 알게되면서 더 좋아졌나봅니다.      2\n",
       "8863              1년 동거하고 결혼했어                    축하합니다!      2\n",
       "8864  1년 만났는데도 그 사람에 대해 잘 모르겠어                  더 만나보세요.      2\n",
       "8865     1년반 만났는데 결혼하자고 해도 될까?            기다리고 있을지도 몰라요.      2\n",
       "8866                 1년째 동거중이야           서로 알아가는 단계인가봐요.      2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# label의 예를 보자\n",
    "from IPython.display import display\n",
    "\n",
    "display(df[df['label'] == 0][:7])\n",
    "display(df[df['label'] == 1][:7])\n",
    "display(df[df['label'] == 2][:7])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "resident-banner",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Q        0\n",
       "A        0\n",
       "label    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 결측치 확인\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "tutorial-confirmation",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 11823 entries, 0 to 11822\n",
      "Data columns (total 3 columns):\n",
      " #   Column  Non-Null Count  Dtype \n",
      "---  ------  --------------  ----- \n",
      " 0   Q       11823 non-null  object\n",
      " 1   A       11823 non-null  object\n",
      " 2   label   11823 non-null  int64 \n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 277.2+ KB\n"
     ]
    }
   ],
   "source": [
    "# 데이터 전반적 개요 확인\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "necessary-albuquerque",
   "metadata": {},
   "source": [
    "> #### note:\n",
    "> - `label` 컬럼은 답변에 대한 카테고리로서 아래와 같다고 한다.(그런데 완전 fit한 것 같지는 않다.)\n",
    ">   - 0 : 일상다반사\n",
    ">   - 1 : 이별(부정)\n",
    ">   - 2 : 사랑(긍정)\n",
    "> - 데이터는 '사랑보다 아름다운 실연'이라는 다음 카페의 이야기를 참고하여 만든 인공 데이터라고 한다.\n",
    "> - 그렇기에 주로 사랑, 그리고 그에 관한 위로에 관한 답변이 주를 이루는 것 같은데,\n",
    "> - 하나 눈여겨 볼 점은 데이터가 **shuffle된 것 같지 않다.**(그러나 `tf.data.Dataset` 자료형으로 하고 shuffle을 하면 될 것 같다.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "palestinian-management",
   "metadata": {},
   "source": [
    "## Step 2. 데이터 전처리하기\n",
    "- punctuation은 띄어쓰기를 더해주고\n",
    "- 한국어에 맞게 전처리되어야 할 부분은 없는지 고민해보자"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "raised-expense",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "q_lengths min : 1\n",
      "q_lengths max : 15\n",
      "q_lengths mean : 3.587414361837097\n",
      "=============================\n",
      "a_lengths min : 1\n",
      "a_lengths max : 21\n",
      "a_lengths mean : 3.6936479742874058\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzsAAAKuCAYAAACG8wJrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABBdUlEQVR4nO39fbRldX0n+L8/VhFJxBaQaiQ8pOiEsUFEtKvRtE4anxCJP8rMyuQH050Qu/pXsQdNnMlMGkN3MDHMMr9O4kST1iZCixm71DHa1FISJYa0w1pRKQjyVNqUiqEID6XgAzoaIZ/54+5iLuW99XjuOffu+3qtddbd+7u/++zPOXXu2fW++3u+p7o7AAAAY/OkWRcAAACwFIQdAABglIQdAABglIQdAABglIQdAABglIQdAABglPYZdqrq8Kr6TFV9tqruqKpfH9rfXVVfqqpbhtuZQ3tV1duqakdV3VpVz5t3XxdV1V3D7aIle1QAAMCqt3Y/+nw3yUu6+5GqOizJDVX1J8O2/7W7P7hH/1cmOWW4PT/JO5I8v6qOTnJZkg1JOslNVbW1ux+exAMBAACYb59hp+e+dfSRYfWw4ba3byLdmOQ9w36fqqojq+q4JGcnua67H0qSqrouyblJtix2R8ccc0yvX79+Px4GAEvppptu+kp3r5t1HcuRcxXA7C12ntqfKzupqjVJbkryY0n+oLs/XVX/KsnlVfVrST6R5JLu/m6S45PcM2/3nUPbYu17Hmtzks1JctJJJ2Xbtm37UyIAS6iqvjzrGpar9evXO1cBzNhi56n9mqCgux/r7jOTnJDkrKo6Pckbk/zDJP84ydFJ/vUkCu3uK7p7Q3dvWLfOHxEBAICDc0CzsXX315Jcn+Tc7r6v53w3yX9MctbQ7d4kJ87b7YShbbF2AACAiduf2djWVdWRw/IPJnl5ks8Nn8NJVVWSVye5fdhla5KfG2Zle0GSr3f3fUk+luScqjqqqo5Kcs7QBgAAMHH785md45JcPXxu50lJPtDdH6mqP6+qdUkqyS1JXjv0vzbJeUl2JPl2ktckSXc/VFVvTnLj0O83dk9WAAAAMGn7MxvbrUmeu0D7Sxbp30kuXmTbVUmuOsAaAQAADtgBfWYHAABgpRB2AACAURJ2AACAURJ2AACAURJ2AACAURJ2AACAURJ2AACAURJ2AACAUdrnl4oyPesv+ejE7/Put/zkxO8TgJXvUM85zi/ASuDKDgAAMErCDgAAMErCDgAAMErCDgAAMErCDgAAMErCDgAAMErCDgArWlUdXlWfqarPVtUdVfXrQ/vJVfXpqtpRVe+vqh8Y2p88rO8Ytq+fd19vHNo/X1WvmNFDAmBChB0AVrrvJnlJdz8nyZlJzq2qFyT5rSRv7e4fS/Jwkk1D/01JHh7a3zr0S1WdluSCJM9Kcm6Sf19Va6b5QACYLGEHgBWt5zwyrB423DrJS5J8cGi/Osmrh+WNw3qG7S+tqhra39fd3+3uLyXZkeSspX8EACwVYQeAFa+q1lTVLUkeTHJdki8k+Vp3Pzp02Znk+GH5+CT3JMmw/etJnj6/fYF99jze5qraVlXbdu3aNeFHA8CkCDsArHjd/Vh3n5nkhMxdjfmHS3y8K7p7Q3dvWLdu3VIeCoBDIOwAMBrd/bUk1yf58SRHVtXaYdMJSe4dlu9NcmKSDNufluSr89sX2AeAFUjYAWBFq6p1VXXksPyDSV6eZHvmQs9PD90uSnLNsLx1WM+w/c+7u4f2C4bZ2k5OckqSz0zlQQCwJNbuuwsALGvHJbl6mDntSUk+0N0fqao7k7yvqn4zyV8luXLof2WSP6qqHUkeytwMbOnuO6rqA0nuTPJokou7+7EpPxYAJkjYAWBF6+5bkzx3gfYvZoHZ1Lr7O0n++0Xu6/Ikl0+6RgBmwzA2AABglIQdAABglIQdAABglIQdAABglIQdAABglIQdAABglIQdAABglIQdAABglIQdAABglPYZdqrq8Kr6TFV9tqruqKpfH9pPrqpPV9WOqnp/Vf3A0P7kYX3HsH39vPt649D++ap6xZI9KgAAYNXbnys7303yku5+TpIzk5xbVS9I8ltJ3trdP5bk4SSbhv6bkjw8tL916JeqOi3JBUmeleTcJP++qtZM8LEAAAA8bp9hp+c8MqweNtw6yUuSfHBovzrJq4fljcN6hu0vraoa2t/X3d/t7i8l2ZHkrEk8CAAAgD3t12d2qmpNVd2S5MEk1yX5QpKvdfejQ5edSY4flo9Pck+SDNu/nuTp89sX2Gf+sTZX1baq2rZr164DfkAAAADJfoad7n6su89MckLmrsb8w6UqqLuv6O4N3b1h3bp1S3UYAABg5A5oNrbu/lqS65P8eJIjq2rtsOmEJPcOy/cmOTFJhu1PS/LV+e0L7AMAADBR+zMb27qqOnJY/sEkL0+yPXOh56eHbhcluWZY3jqsZ9j+593dQ/sFw2xtJyc5JclnJvQ4AAAAnmDtvrvkuCRXDzOnPSnJB7r7I1V1Z5L3VdVvJvmrJFcO/a9M8kdVtSPJQ5mbgS3dfUdVfSDJnUkeTXJxdz822YcDAAAwZ59hp7tvTfLcBdq/mAVmU+vu7yT57xe5r8uTXH7gZQIAAByYA/rMDgAAwEoh7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7AAAAKMk7ACwolXViVV1fVXdWVV3VNUvDe1vqqp7q+qW4XbevH3eWFU7qurzVfWKee3nDm07quqSWTweACZn7awLAIBD9GiSX+7um6vqqUluqqrrhm1v7e7fnt+5qk5LckGSZyX54SR/VlX/zbD5D5K8PMnOJDdW1dbuvnMqjwKAiRN2AFjRuvu+JPcNy9+squ1Jjt/LLhuTvK+7v5vkS1W1I8lZw7Yd3f3FJKmq9w19hR2AFcowNgBGo6rWJ3lukk8PTa+rqlur6qqqOmpoOz7JPfN22zm0Lda+0HE2V9W2qtq2a9euST4EACZI2AFgFKrqiCR/nOQN3f2NJO9I8qNJzszclZ/fmdSxuvuK7t7Q3RvWrVs3qbsFYMIMYwNgxauqwzIXdN7b3R9Kku5+YN72P0zykWH13iQnztv9hKEte2kHYAUSdgBY0aqqklyZZHt3/+689uOGz/MkyU8luX1Y3prkP1XV72ZugoJTknwmSSU5papOzlzIuSDJ/zCdR3Fw1l/y0VmXALCsCTsArHQvTPKzSW6rqluGtl9NcmFVnZmkk9yd5BeSpLvvqKoPZG7igUeTXNzdjyVJVb0uyceSrElyVXffMb2HAcCkCTsArGjdfUPmrsrs6dq97HN5kssXaL92b/sBsLKYoAAAABglYQcAABglYQcAABglYQcAABglYQcAABgls7EBAAfsUL7j5+63/OQEKwFYnCs7AADAKO0z7FTViVV1fVXdWVV3VNUvDe1vqqp7q+qW4XbevH3eWFU7qurzVfWKee3nDm07quqSpXlIAAAA+zeM7dEkv9zdN1fVU5PcVFXXDdve2t2/Pb9zVZ2W5IIkz0ryw0n+rKr+m2HzHyR5eZKdSW6sqq3dfeckHggAAMB8+ww73X1fkvuG5W9W1fYkx+9ll41J3tfd303yparakeSsYduO7v5iklTV+4a+ws4SOpQx1XtjvDUAAMvdAU1QUFXrkzw3yaeTvDDJ66rq55Jsy9zVn4czF4Q+NW+3nfl/w9E9e7Q/f4FjbE6yOUlOOumkAylvapYqQAAAAJOz3xMUVNURSf44yRu6+xtJ3pHkR5OcmbkrP78ziYK6+4ru3tDdG9atWzeJuwQAAFah/bqyU1WHZS7ovLe7P5Qk3f3AvO1/mOQjw+q9SU6ct/sJQ1v20g4AADBR+zMbWyW5Msn27v7dee3Hzev2U0luH5a3Jrmgqp5cVScnOSXJZ5LcmOSUqjq5qn4gc5MYbJ3MwwAAAHii/bmy88IkP5vktqq6ZWj71SQXVtWZSTrJ3Ul+IUm6+46q+kDmJh54NMnF3f1YklTV65J8LMmaJFd19x0TeyQAAADz7M9sbDckqQU2XbuXfS5PcvkC7dfubT8AAIBJ2e8JCgAAAFYSYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQeAFa2qTqyq66vqzqq6o6p+aWg/uqquq6q7hp9HDe1VVW+rqh1VdWtVPW/efV009L+rqi6a1WMCYDKEHQBWukeT/HJ3n5bkBUkurqrTklyS5BPdfUqSTwzrSfLKJKcMt81J3pHMhaMklyV5fpKzkly2OyABsDIJOwCsaN19X3ffPCx/M8n2JMcn2Zjk6qHb1UlePSxvTPKenvOpJEdW1XFJXpHkuu5+qLsfTnJdknOn90gAmDRhB4DRqKr1SZ6b5NNJju3u+4ZN9yc5dlg+Psk983bbObQt1r7QcTZX1baq2rZr167JPQAAJkrYAWAUquqIJH+c5A3d/Y3527q7k/SkjtXdV3T3hu7esG7dukndLQATJuwAsOJV1WGZCzrv7e4PDc0PDMPTMvx8cGi/N8mJ83Y/YWhbrB2AFUrYAWBFq6pKcmWS7d39u/M2bU2ye0a1i5JcM6/954ZZ2V6Q5OvDcLePJTmnqo4aJiY4Z2gDYIVaO+sCAOAQvTDJzya5rapuGdp+NclbknygqjYl+XKSnxm2XZvkvCQ7knw7yWuSpLsfqqo3J7lx6Pcb3f3QVB4BAEtC2AFgRevuG5LUIptfukD/TnLxIvd1VZKrJlcdALNkGBsAADBKwg4AADBKwg4AADBK+ww7VXViVV1fVXdW1R1V9UtD+9FVdV1V3TX8PGpor6p6W1XtqKpbq+p58+7roqH/XVV10WLHBAAAOFT7c2Xn0SS/3N2nJXlBkour6rQklyT5RHefkuQTw3qSvDLJKcNtc5J3JHPhKMllSZ6f5Kwkl+0OSAAAAJO2z7DT3fd1983D8jeTbE9yfJKNSa4eul2d5NXD8sYk7+k5n0py5PBlbq9Icl13P9TdDye5Lsm5k3wwAAAAux3QZ3aqan2S5yb5dJJjhy9hS5L7kxw7LB+f5J55u+0c2hZr3/MYm6tqW1Vt27Vr14GUBwAA8Lj9DjtVdUSSP07yhu7+xvxtw3cW9CQK6u4runtDd29Yt27dJO4SAABYhfYr7FTVYZkLOu/t7g8NzQ8Mw9My/HxwaL83yYnzdj9haFusHQAAYOL2Zza2SnJlku3d/bvzNm1NsntGtYuSXDOv/eeGWdlekOTrw3C3jyU5p6qOGiYmOGdoAwAAmLi1+9HnhUl+NsltVXXL0ParSd6S5ANVtSnJl5P8zLDt2iTnJdmR5NtJXpMk3f1QVb05yY1Dv9/o7ocm8SAAAAD2tM+w0903JKlFNr90gf6d5OJF7uuqJFcdSIEAAAAH44BmYwMAAFgphB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AAGCUhB0AVrSquqqqHqyq2+e1vamq7q2qW4bbefO2vbGqdlTV56vqFfPazx3adlTVJdN+HABMnrADwEr37iTnLtD+1u4+c7hdmyRVdVqSC5I8a9jn31fVmqpak+QPkrwyyWlJLhz6ArCCrZ11AQBwKLr7k1W1fj+7b0zyvu7+bpIvVdWOJGcN23Z09xeTpKreN/S9c9L1AjA9ruwAMFavq6pbh2FuRw1txye5Z16fnUPbYu0LqqrNVbWtqrbt2rVr0nUDMCHCDgBj9I4kP5rkzCT3JfmdSd55d1/R3Ru6e8O6desmedcATJBhbACMTnc/sHu5qv4wyUeG1XuTnDiv6wlDW/bSDsAK5coOAKNTVcfNW/2pJLtnatua5IKqenJVnZzklCSfSXJjklOq6uSq+oHMTWKwdZo1AzB5ruwAsKJV1ZYkZyc5pqp2JrksydlVdWaSTnJ3kl9Iku6+o6o+kLmJBx5NcnF3Pzbcz+uSfCzJmiRXdfcd030kAEyasAPAitbdFy7QfOVe+l+e5PIF2q9Ncu0ESwNgxvYZdqrqqiSvSvJgd58+tL0pyf8vye4paH513ncYvDHJpiSPJfnF7v7Y0H5ukt/L3F/M3tXdb5nsQwEAVoL1l3z0kPa/+y0/OaFKgLHbn8/svDu+rA0AAFhh9nllx5e1AQAAK9GhzMa2JF/W5ovaAACASTjYsLNkX9bmi9oAAIBJOKjZ2HxZGwAAsNwd1JUdX9YGAAAsd/sz9bQva+P7HOq0oQsxlSgAAJO0P7Ox+bI2AABgxTmU2dgAAACWLWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHgBWvqq6qqger6vZ5bUdX1XVVddfw86ihvarqbVW1o6purarnzdvnoqH/XVV10SweCwCTI+wAMAbvTnLuHm2XJPlEd5+S5BPDepK8Mskpw21zknckc+EoyWVJnp/krCSX7Q5IAKxMwg4AK153fzLJQ3s0b0xy9bB8dZJXz2t/T8/5VJIjq+q4JK9Icl13P9TdDye5Lt8foABYQYQdAMbq2O6+b1i+P8mxw/LxSe6Z12/n0LZY+/epqs1Vta2qtu3atWuyVQMwMcIOAKPX3Z2kJ3h/V3T3hu7esG7dukndLQATJuwAMFYPDMPTMvx8cGi/N8mJ8/qdMLQt1g7ACrXPsGOGGwBWqK1Jdp9vLkpyzbz2nxvOWS9I8vVhuNvHkpxTVUcN57VzhjYAVqj9ubLz7pjhBoBlrKq2JPnLJM+sqp1VtSnJW5K8vKruSvKyYT1Jrk3yxSQ7kvxhkv8xSbr7oSRvTnLjcPuNoQ2AFWrtvjp09yerav0ezRuTnD0sX53kL5L868yb4SbJp6pq9ww3Z2eY4SZJqmr3DDdbDv0hALDadfeFi2x66QJ9O8nFi9zPVUmummBpAMzQwX5mxww3AADAsnbIExSY4QYAAFiODjbsmOEGAABY1g427JjhBgAAWNb2OUHBMMPN2UmOqaqdmZtV7S1JPjDMdvPlJD8zdL82yXmZm+Hm20lek8zNcFNVu2e4ScxwAwAALLH9mY3NDDcAAMCKc8gTFAAAACxHwg4AADBKwg4AADBKwg4AADBKwg4AADBK+5yNDQBgOVl/yUcPet+73/KTE6wEWO5c2QEAAEZJ2AEAAEZJ2AEAAEZJ2AEAAEZJ2AEAAEZJ2AEAAEbJ1NMAMEOHMo0yAHvnyg4AADBKwg4AADBKhrGxbCzVUA7flg0AsDq5sgMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIzS2lkXsNTWX/LRWZcAAADMgCs7AADAKAk7AADAKAk7AADAKAk7AADAKAk7AIxWVd1dVbdV1S1VtW1oO7qqrququ4afRw3tVVVvq6odVXVrVT1vttUDcKiEHQDG7sXdfWZ3bxjWL0nyie4+JcknhvUkeWWSU4bb5iTvmHqlAEyUsAPAarMxydXD8tVJXj2v/T0951NJjqyq42ZQHwATckhhx/AAAJa5TvLxqrqpqjYPbcd2933D8v1Jjh2Wj09yz7x9dw5t36eqNlfVtqratmvXrqWoG4AJmMSVHcMDAFiuXtTdz8vcOejiqvqJ+Ru7uzMXiA5Id1/R3Ru6e8O6desmVCoAk7YUw9gMDwBgWejue4efDyb5cJKzkjyw+/wz/Hxw6H5vkhPn7X7C0AbACnWoYWfiwwMMDQBgEqrqKVX11N3LSc5JcnuSrUkuGrpdlOSaYXlrkp8bhl2/IMnX553PAFiB1h7i/i/q7nur6u8nua6qPjd/Y3d3VR3Q8IDuviLJFUmyYcOGAx5aAACDY5N8uKqSufPdf+ruP62qG5N8oKo2Jflykp8Z+l+b5LwkO5J8O8lrpl8yAJN0SGFn/vCAqnrC8IDuvs/wAABmpbu/mOQ5C7R/NclLF2jvJBdPoTQApuSgh7EZHgAAACxnh3Jlx/AAAGBFWX/JRw9637vf8pMTrASYhoMOO4YHAAAAy9lSTD0NAAAwc8IOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSof0paKwEhzKNKOLMf0oAMDy58oOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSmtnXQAAwEqw/pKPHtL+d7/lJydUCbC/XNkBAABGSdgBAABGSdgBAABGSdgBAABGyQQFcBAO9UOqi/HhVQCAyXFlBwAAGCVhBwAAGCVhBwAAGCVhBwAAGCVhBwAAGCWzsQEATMGhzORptk44OK7sAAAAo+TKDiwjS/H9Pf4aCLDyuSoEB8eVHQAAYJSEHQAAYJSmPoytqs5N8ntJ1iR5V3e/Zdo1AMBinKcYm0MdIm0YHCvZVMNOVa1J8gdJXp5kZ5Ibq2prd985zTpgNVmKzwElTn6Mk/MUfD+fF2Ilm/aVnbOS7OjuLyZJVb0vycYkTiKwwixViFoKTrYcAOcpmKBZniu895NMP+wcn+Seees7kzx/foeq2pxk87D6SFV9fkq1HYxjknxl1kUcBHVPl7qn6/vqrt+aUSUHZrk/3z8y6wKmZJ/nqeT7zlXfrarbp1DbwVrur63lXl+y/GtU3wIO8L3fc3holkN9C56nlt3U0919RZIrZl3H/qiqbd29YdZ1HCh1T5e6p0vdTMP8c9Vy/7dT36Fb7jWq79At9xrVd/CmPRvbvUlOnLd+wtAGAMuB8xTAiEw77NyY5JSqOrmqfiDJBUm2TrkGAFiM8xTAiEx1GFt3P1pVr0vyscxN6XlVd98xzRombEUMt1uAuqdL3dOlbg7aQZ6nlvu/nfoO3XKvUX2HbrnXqL6DVN096xoAAAAmbtrD2AAAAKZC2AEAAEZJ2DlAVXViVV1fVXdW1R1V9UuzrulAVNWaqvqrqvrIrGvZX1V1ZFV9sKo+V1Xbq+rHZ13T/qiq/2l4jdxeVVuq6vBZ17SYqrqqqh6c/10hVXV0VV1XVXcNP4+aZY0LWaTufze8Vm6tqg9X1ZEzLHFBC9U9b9svV1VX1TGzqI2FVdW5VfX5qtpRVZcssP3JVfX+Yfunq2r9lOvb57mpqs6uqq9X1S3D7demXOPdVXXbcOxtC2yvqnrb8BzeWlXPm2Jtz5z3vNxSVd+oqjfs0Wfqz9+hvDdX1UVDn7uq6qIp1rdf78H7ej0scY1vqqp75/1bnrfIvnv9vV/C+t4/r7a7q+qWRfZd8udwsfeW5fQ63KfudjuAW5LjkjxvWH5qkv+a5LRZ13UA9f/PSf5Tko/MupYDqPnqJP9yWP6BJEfOuqb9qPn4JF9K8oPD+geS/Pys69pLvT+R5HlJbp/X9v9PcsmwfEmS35p1nftZ9zlJ1g7Lv7VS6h7aT8zcB+O/nOSYWdfp9vi/y5okX0jyD4b3oM/u+b6f5H9M8s5h+YIk759yjfs8NyU5e5bv/Unu3tvrOsl5Sf4kSSV5QZJPz/Df+/4kPzLr5+9g35uTHJ3ki8PPo4blo6ZU3369B+/r9bDENb4pyf+yH6+Dvf7eL1V9e2z/nSS/NqvncLH3luX0OtzXzZWdA9Td93X3zcPyN5Nsz9x/bJe9qjohyU8medesa9lfVfW0zL0RXJkk3f233f21mRa1/9Ym+cGqWpvkh5L8zYzrWVR3fzLJQ3s0b8xc0Mzw89XTrGl/LFR3d3+8ux8dVj+Vue9JWVYWeb6T5K1JfiWJmWOWl7OS7OjuL3b33yZ5X+Z+P+ab//vywSQvraqaVoEr+dw0z8Yk7+k5n0pyZFUdN4M6XprkC9395Rkc+wkO4b35FUmu6+6HuvvhJNclOXca9S239+C9vN/uy/783h+yvdU3vIf8TJItkz7u/trLe8uyeR3ui7BzCIZhCs9N8ukZl7K//vfM/Ufq72Zcx4E4OcmuJP+x5obfvauqnjLrovalu+9N8ttJ/jrJfUm+3t0fn21VB+zY7r5vWL4/ybGzLOYg/YvM/aV42auqjUnu7e7PzroWvs/xSe6Zt74z3x8kHu8z/Efv60mePpXq9rCPc9OPV9Vnq+pPqupZ060sneTjVXVTVW1eYPv+PM/TcEEW/8/lLJ+/3fbnvXm5PJd7ew/e1+thqb1uGGp31SJDsJbDc/jfJnmgu+9aZPtUn8M93ltWzOtQ2DlIVXVEkj9O8obu/sas69mXqnpVkge7+6ZZ13KA1mbu8u47uvu5Sb6Vuculy9rwxrkxc2Hth5M8par++WyrOng9dz16RV1tqKpLkzya5L2zrmVfquqHkvxqkql+hoLx2ce56ebMDc16TpK3J/nPUy7vRd39vCSvTHJxVf3ElI+/TzX3RbLnJ/k/F9g86+fv+yzn9+b9eA+e5evhHUl+NMmZmfuD5O9M8dgH4sLs/arO1J7Dvb23LOfXYSLsHJSqOixz/+Dv7e4Pzbqe/fTCJOdX1d2ZuxT7kqr6P2Zb0n7ZmWRnd+/+C+UHMxd+lruXJflSd+/q7u8l+VCSfzLjmg7UA7uHkAw/H5xxPfutqn4+yauS/LPhTXi5+9HMBePPDr+jJyS5uaqeMdOq2O3ezH2earcThrYF+wxDV5+W5KtTqW6wr3NTd3+jux8Zlq9NclhNcSKM4Yp3uvvBJB/O3DCh+fbneV5qr0xyc3c/sOeGWT9/8+zPe/NMn8v9eQ/ej9fDkunuB7r7se7+uyR/uMixZ/0crk3y3yV5/2J9pvUcLvLesuxfh7sJOwdoGD95ZZLt3f27s65nf3X3G7v7hO5en7lL9H/e3cv+SkN335/knqp65tD00iR3zrCk/fXXSV5QVT80vGZemrlxrivJ1iS7Z065KMk1M6xlv1XVuZkbrnl+d3971vXsj+6+rbv/fnevH35Hd2buA6H3z7g05tyY5JSqOnn4y/8Fmfv9mG/+78tPZ+49dmpBe3/OTVX1jN2fI6qqszL3f4CpBLKqekpVPXX3cuY+xL7nbIRbk/xczXlB5ob/3pfpWvQv6bN8/vawP+/NH0tyTlUdNYw0OGdoW3L78x68n6+Hpaxx/mfBfmqRY+/P7/1SelmSz3X3zoU2Tus53Mt7y7J+HT7BpGc8GPstyYsyd6nu1iS3DLfzZl3XAT6Gs7OyZmM7M8m24Tn/z5nBTB4HWfevJ/lc5t58/ijJk2dd015q3ZK5S/nfy9x/tDdl7vMGn0hyV5I/S3L0rOvcz7p3ZG6M8O7fz3fOus79qXuP7XfHbGzL6pa5mcL+a+ZmZ7p0aPuNzP2HLkkOz9zQpx1JPpPkH0y5vgXPTUlem+S1Q5/XJbkjc7NKfSrJP5liff9gOO5nhxp2P4fz66skfzA8x7cl2TDl5/ApmQsvT5vXNtPn70Dem5NsSPKuefv+i+H1uCPJa6ZY34LvwZkb0n3t3l4PU6zxj4bX2K2Z+0/7cXvWOKx/3+/9NOob2t+9+7U3r+/Un8O9vLcsm9fhvm41FAIAADAqhrEBAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjtHbWBezNMccc0+vXr591GQCr3k033fSV7l436zqWI+cqgNlb7Dy1rMPO+vXrs23btlmXAbDqVdWXZ13DcuVcBTB7i52nDGMDAABGSdgBAABGSdgBAABGSdgBAABGSdgBAABGSdgBAABGSdgBAABGSdgBAABGSdgBAABGSdiBJbJly5acfvrpWbNmTU4//fRs2bJl1iUBwOOcp1gN1s66ABijLVu25NJLL82VV16ZF73oRbnhhhuyadOmJMmFF1444+oAWO2cp1gtqrtnXcOiNmzY0Nu2bZt1GXDATj/99Lz97W/Pi1/84sfbrr/++rz+9a/P7bffPsPK4OBU1U3dvWHWdSxHzlWsRM5TjM1i5ylhB5bAmjVr8p3vfCeHHXbY423f+973cvjhh+exxx6bYWVwcISdxTlXsRI5TzE2i52nfGYHlsCpp56aG2644QltN9xwQ0499dQZVQQA/y/nKVYLYQeWwKWXXppNmzbl+uuvz/e+971cf/312bRpUy699NJZlwYAzlOsGiYogCWw+8Odr3/967N9+/aceuqpufzyy33oE4BlwXmK1cJndgDYJ5/ZWZxzFcDs+cwOAACwqgg7AADAKAk7AADAKAk7AADAKAk7AADAKAk7AIxGVZ1YVddX1Z1VdUdV/dLQfnRVXVdVdw0/j1pk/4uGPndV1UXTrR6ma8uWLTn99NOzZs2anH766dmyZcusS4KJE3YAGJNHk/xyd5+W5AVJLq6q05JckuQT3X1Kkk8M609QVUcnuSzJ85OcleSyxUIRrHRbtmzJpZdemre//e35zne+k7e//e259NJLBR5GR9gBYDS6+77uvnlY/maS7UmOT7IxydVDt6uTvHqB3V+R5Lrufqi7H05yXZJzl7xomIHLL788V155ZV784hfnsMMOy4tf/OJceeWVufzyy2ddGkyUsAPAKFXV+iTPTfLpJMd2933DpvuTHLvALscnuWfe+s6hbaH73lxV26pq265duyZXNEzJ9u3b86IXvegJbS960Yuyffv2GVUES0PYAWB0quqIJH+c5A3d/Y3527q7k/Sh3H93X9HdG7p7w7p16w7lrmAmTj311Nxwww1PaLvhhhty6qmnzqgiWBrCDgCjUlWHZS7ovLe7PzQ0P1BVxw3bj0vy4AK73pvkxHnrJwxtMDqXXnppNm3alOuvvz7f+973cv3112fTpk259NJLZ10aTNTapbjTqroqyauSPNjdp++x7ZeT/HaSdd39laU4PgCrU1VVkiuTbO/u3523aWuSi5K8Zfh5zQK7fyzJ/zZvUoJzkrxxCcuFmbnwwguTJK9//euzffv2nHrqqbn88ssfb4exWJKwk+TdSX4/yXvmN1bViZk7efz1Eh0XgNXthUl+NsltVXXL0ParmQs5H6iqTUm+nORnkqSqNiR5bXf/y+5+qKrenOTGYb/f6O6Hplo9TNGFF14o3DB6SxJ2uvuTwwdD9/TWJL+Shf+iBgCHpLtvSFKLbH7pAv23JfmX89avSnLV0lQHwLRN7TM7VbUxyb3d/dl99DPDDQAAcMimEnaq6ocyN4zg1/bV1ww3AADAJEzrys6PJjk5yWer6u7MzXBzc1U9Y0rHBwAAVpmlmqDgCbr7tiR/f/f6EHg2mI0NAABYKktyZaeqtiT5yyTPrKqdw+w3AAAAU7NUs7HtdR7D7l6/FMcFAADYbWqzsQEAAEyTsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIySsAMAAIzS2lkXAACTUlVXJXlVkge7+/Sh7f1Jnjl0OTLJ17r7zAX2vTvJN5M8luTR7t4whZIBWELCDgBj8u4kv5/kPbsbuvv/u3u5qn4nydf3sv+Lu/srS1YdAFMl7AAwGt39yapav9C2qqokP5PkJVMtCoCZ8ZkdAFaL/zbJA9191yLbO8nHq+qmqtq8tzuqqs1Vta2qtu3atWvihQIwGcIOAKvFhUm27GX7i7r7eUlemeTiqvqJxTp29xXdvaG7N6xbt27SdQIwIcIOAKNXVWuT/HdJ3r9Yn+6+d/j5YJIPJzlrOtUBsFSEHQBWg5cl+Vx371xoY1U9paqeuns5yTlJbp9ifQAsAWEHgNGoqi1J/jLJM6tqZ1VtGjZdkD2GsFXVD1fVtcPqsUluqKrPJvlMko92959Oq24AlobZ2AAYje6+cJH2n1+g7W+SnDcsfzHJc5a0OACmzpUdAABglJYk7FTVVVX1YFXdPq/t31XV56rq1qr6cFUduRTHBgAASJbuys67k5y7R9t1SU7v7jOS/Nckb1yiY8OycMQRR6SqHr8dccQRsy4JAGBVWZKw092fTPLQHm0f7+5Hh9VPJTlhKY4Ny8ERRxyRb33rW1m/fn127NiR9evX51vf+pbAAwAwRbOaoOBfZC/fdQAr3e6g86UvfSlJ8qUvfSknn3xy7r777tkWBgCwikx9goKqujTJo0neu8j2zVW1raq27dq1a7rFwQT92Z/92V7XAQBYWlMNO1X180leleSfdXcv1Ke7r+juDd29Yd26ddMsDybqZS972V7XAQBYWlMLO1V1bpJfSXJ+d397WseFWXjKU56Su+++OyeffHK+8IUvPD6E7SlPecqsSwMAWDWW5DM7wzdYn53kmKrameSyzM2+9uQk11VVknyqu1+7FMeHWXvkkUdyxBFH5O67786P/diPJZkLQI888siMKwMAWD2WJOws8g3WVy7FsWC5EmwAAGZr6hMUAAAATIOwAwCwCp1xxhlP+PLrM844Y9YlwcQJOwAAq8wZZ5yR2267Leeff3527dqV888/P7fddpvAw+gIOwAAq8zuoHPNNdfkmGOOyTXXXPN44IExEXYAAFahK6+8cq/rMAbCDgDAKrRp06a9rsMYCDsAAKvMs5/97GzdujUbN27MV77ylWzcuDFbt27Ns5/97FmXBhO1JN+zAwDA8nXrrbfmjDPOyNatW7Nu3bokcwHo1ltvnXFlMFnCDgDAKiTYsBoYxgYAAIySsAMAAIySsANLZM2aNU/4Zuo1a9bMuiQAgFVF2IElsGbNmvzd3/1djjjiiNx000054ogj8nd/93cCDwDAFJmgAJbA7qDzzW9+M0nyzW9+M0996lPzyCOPzLgyAIDVw5UdWCL/5b/8l72uAwCwtIQdWCL/9J/+072uA5NXVVdV1YNVdfu8tjdV1b1VdctwO2+Rfc+tqs9X1Y6qumR6VcNsnHHGGU/4bOkZZ5wx65Jg4oQdWAJPetKT8sgjj+SpT31qbr755seHsD3pSX7lYIm9O8m5C7S/tbvPHG7X7rmxqtYk+YMkr0xyWpILq+q0Ja0UZuiMM87IbbfdlvPPPz+7du3K+eefn9tuu03gYXT8zwuWwGOPPfZ44PlH/+gfPR50HnvssVmXBqPW3Z9M8tBB7HpWkh3d/cXu/tsk70uycaLFwTKyO+hcc801OeaYY3LNNdc8HnhgTIQdWCKPPfZYuvvxm6ADM/W6qrp1GOZ21ALbj09yz7z1nUPbgqpqc1Vtq6ptu3btmnStMBVXXnnlXtdhDIQdAMbuHUl+NMmZSe5L8juHeofdfUV3b+juDevWrTvUu4OZ2LRp017XYQyEHQBGrbsf6O7Huvvvkvxh5oas7eneJCfOWz9haINRevazn52tW7dm48aN+cpXvpKNGzdm69atefaznz3r0mCifM8OAKNWVcd1933D6k8luX2BbjcmOaWqTs5cyLkgyf8wpRJh6m699dacccYZ2bp1a3ZfnXz2s5+dW2+9dcaVwWQJOwCMRlVtSXJ2kmOqameSy5KcXVVnJukkdyf5haHvDyd5V3ef192PVtXrknwsyZokV3X3HdN/BDA9gg2rgbADwGh094ULNC/4qevu/psk581bvzbJ901LDcDK5TM7AADAKAk7AADAKAk7AADAKAk7AADAKAk7AACr0NOf/vRU1eO3pz/96bMuCSZO2AEAWGWe/vSn56GHHsqznvWsfPnLX86znvWsPPTQQwIPo7MkYaeqrqqqB6vq9nltR1fVdVV11/DzqKU4NiwX8/9atvsGAMvB7qBz++2356STTsrtt9/+eOCBMVmqKzvvTnLuHm2XJPlEd5+S5BPDOozS/GDztre9bcF2AJila6+9dq/rMAZLEna6+5NJ9vzTwMYkVw/LVyd59VIcG5aT7s7rX//6dPesSwGAJzjvvPP2ug5jMM3P7Bzb3fcNy/cnOXahTlW1uaq2VdW2Xbt2Ta86mLD5V3QWWgeAWTn66KNzxx135PTTT89f//Vf5/TTT88dd9yRo48+etalwUTVUv3FuarWJ/lId58+rH+tu4+ct/3h7t7r53Y2bNjQ27ZtW5L6YCntHq42//droTZYKarqpu7eMOs6liPnKlaq3ZMU7Hb00Ufnq1/96gwrgoO32Hlqmld2Hqiq44Zijkvy4BSPDTNRVXn729/uszoALDtf/epX092P3wQdxmiaYWdrkouG5YuSXDPFY8NUzb9684u/+IsLtgMAsLSWaurpLUn+Mskzq2pnVW1K8pYkL6+qu5K8bFiH0Zr/17LdNwAApmftUtxpd1+4yKaXLsXxAAAA9jTNYWwAAABTI+wAAACjJOwAAACjJOwAAKxChx9+eKrq8dvhhx8+65Jg4oQdAIBV5vDDD893v/vdHHvssdm+fXuOPfbYfPe73xV4GJ0lmY0NAIDla3fQuf/++5Mk999/f57xjGfkgQcemHFlMFmu7AAArEJ/8Rd/sdd1GANhBwBgFTr77LP3ug5jIOwAAKwyT37yk/PAAw/kGc94Rj73uc89PoTtyU9+8qxLg4nymR0ARqOqrkryqiQPdvfpQ9u/S/L/SfK3Sb6Q5DXd/bUF9r07yTeTPJbk0e7eMKWyYeq+853v5PDDD88DDzyQU089NclcAPrOd74z48pgslzZgSUyfzrP3Tdgyb07ybl7tF2X5PTuPiPJf03yxr3s/+LuPlPQYTX4zne+k+5+/CboMEbCDiyB+cHmjDPOWLAdmLzu/mSSh/Zo+3h3PzqsfirJCVMvDICZMIwNllB3P74s6MCy8C+SvH+RbZ3k41XVSf5Dd1+x2J1U1eYkm5PkpJNOmniRAEyGKzuwROZf0VloHZiuqro0yaNJ3rtIlxd19/OSvDLJxVX1E4vdV3df0d0bunvDunXrlqBaACZB2IElcuutt+51HZieqvr5zE1c8M96/iXXebr73uHng0k+nOSsqRUIwJIQdmAJVVWe85znGMIGM1RV5yb5lSTnd/e3F+nzlKp66u7lJOckuX16VQKwFIQdWALz/3A8/4rOIn9QBiakqrYk+cskz6yqnVW1KcnvJ3lqkuuq6paqeufQ94er6tph12OT3FBVn03ymSQf7e4/ncFDgKkxayirgQkKYIkINjB93X3hAs1XLtL3b5KcNyx/MclzlrA0WFbmB5t3vvOdee1rX/t4u/MXY+LKDgDAKtXd+YVf+AUBh9ESdgAAVqF3vvOde12HMRB2AABWod1D1xZbhzEQdgAAVqmqyn/4D//B5ASMlrADALDKzP+MzvwrOj67w9iYjQ0AYBUSbFgNXNkBAABGSdgBAABGSdgBAABGSdgBAABGSdgBAABGaephp6r+p6q6o6pur6otVXX4tGsAAADGb6php6qOT/KLSTZ09+lJ1iS5YJo1wLRU1ffdAGC5cJ5iNZjFMLa1SX6wqtYm+aEkfzODGmBJLXbCcCIBYDmYfz5685vfvGA7jMFUw05335vkt5P8dZL7kny9uz8+zRpgmrr78RsALDfdnX/zb/6N8xSjNe1hbEcl2Zjk5CQ/nOQpVfXP9+izuaq2VdW2Xbt2TbM8AIBVY/4VnYXWYQymPYztZUm+1N27uvt7ST6U5J/M79DdV3T3hu7esG7duimXBwCwOvzbf/tv97oOYzDtsPPXSV5QVT9Uc4NCX5pk+5RrgKnxoU8AlrOqym/+5m86TzFa0/7MzqeTfDDJzUluG45/xTRrgGlYbOyzMdEALAfzz0fzr+g4TzE2a6d9wO6+LMll0z4uTJsTBgDLmfMUq8Espp4GAABYcsIOAAAwSsIOAAAwSsIOAAAwSsIOAAAwSsIOAKNSVVdV1YNVdfu8tqOr6rqqumv4edQi+1409Lmrqi6aXtUwffO/C853wjFWwg4AY/PuJOfu0XZJkk909ylJPjGsP0FVHZ25r0Z4fpKzkly2WCiClW5+sPnH//gfL9gOYzD179kBgKXU3Z+sqvV7NG9McvawfHWSv0jyr/fo84ok13X3Q0lSVddlLjRtWapaYdbmf9eOoMMYubIDwGpwbHffNyzfn+TYBfocn+Seees7h7bvU1Wbq2pbVW3btWvXZCuFKZl/RWehdRgDYQeAVaXn/pR9SF8d391XdPeG7t6wbt26CVUG03XjjTfudR3GQNgBYDV4oKqOS5Lh54ML9Lk3yYnz1k8Y2mC0qipnnXWWIWyMlrADwGqwNcnu2dUuSnLNAn0+luScqjpqmJjgnKENRmf+Z3XmX9GZ3w5jIOwAMCpVtSXJXyZ5ZlXtrKpNSd6S5OVVdVeSlw3rqaoNVfWuJBkmJnhzkhuH22/snqwAxqi7v+8GY2M2NgBGpbsvXGTTSxfouy3Jv5y3flWSq5aoNACmTNiBAzDNMc3+wgYAcGiEHTgABxNAqkpwAQCYAZ/ZAQAARknYAQAARknYAQAARknYAQAARknYAQAARknYAQAARsnU0wAAI+H74OCJhB0AgJHwfXDwRIaxAQAAoyTsAAAAoyTsAAAAoyTsAAAAoyTsAAAAoyTsAAAAozT1sFNVR1bVB6vqc1W1vap+fNo1AAAA4zeL79n5vSR/2t0/XVU/kOSHZlADAAAwclMNO1X1tCQ/keTnk6S7/zbJ306zBgAAYHWY9jC2k5PsSvIfq+qvqupdVfWU+R2qanNVbauqbbt27ZpyeQAAwFhMO+ysTfK8JO/o7ucm+VaSS+Z36O4runtDd29Yt27dlMsDAADGYtphZ2eSnd396WH9g5kLPwAAABM11bDT3fcnuaeqnjk0vTTJndOsAQAAWB1mMRvb65O8d5iJ7YtJXjODGgAAgJGbetjp7luSbJj2cQEAgNVl6l8qCgDTVlXPrKpb5t2+UVVv2KPP2VX19Xl9fm1G5QIwIbMYxgYAU9Xdn09yZpJU1Zok9yb58AJd/6/uftUUSwNgCbmyA8Bq89IkX+juL8+6EACWlrADwGpzQZIti2z78ar6bFX9SVU9a7E78AXYACuDsAPAqjHMBHp+kv9zgc03J/mR7n5Okrcn+c+L3Y8vwAZYGYQdAFaTVya5ubsf2HNDd3+jux8Zlq9NclhVHTPtAgGYHGEHgNXkwiwyhK2qnlFVNSyflblz5FenWBsAE2Y2NgBWhap6SpKXJ/mFeW2vTZLufmeSn07yr6rq0ST/d5ILurtnUSsAkyHsALAqdPe3kjx9j7Z3zlv+/SS/P+26AFg6hrEBAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjJOwAAACjNJOwU1Vrquqvquojszg+AAAwfrO6svNLSbbP6NgAAMAqMPWwU1UnJPnJJO+a9rEBAIDVY+0Mjvm/J/mVJE9daGNVbU6yOUlOOumk6VXFqnL00Ufn4YcfntrxqmoqxznqqKPy0EMPTeVYsNJU1d1JvpnksSSPdveGPbZXkt9Lcl6Sbyf5+e6+edp1AjA5Uw07VfWqJA92901VdfZCfbr7iiRXJMmGDRt6etWxmjz88MPpHt/La1qhClawF3f3VxbZ9sokpwy35yd5x/ATgBVq2sPYXpjk/OGva+9L8pKq+j+mXAMALGRjkvf0nE8lObKqjpt1UQAcvKmGne5+Y3ef0N3rk1yQ5M+7+59PswYAVq1O8vGqumkYMr2n45PcM29959D2fapqc1Vtq6ptu3btWoJSWe2OPvroVNVUbkmmdqyjjz56xs8sq80sPrMDALPwou6+t6r+fpLrqupz3f3Jg7kjQ65ZaoZbw2TM7EtFu/svuvtVszo+AKtLd987/HwwyYeTnLVHl3uTnDhv/YShDYAVamZhBwCmpaqeUlVP3b2c5Jwkt+/RbWuSn6s5L0jy9e6+b8qlAjBBhrEBsBocm+TDwxCatUn+U3f/aVW9Nkm6+51Jrs3ctNM7Mjf19GtmVCsAEyLsADB63f3FJM9ZoP2d85Y7ycXTrAuApWUYGwAAMErCDgAAMErCDgAAMErCDgAAMErCDgAAMErCDgAAMErCDgAAMErCDgAAMErCDgAAMErCDgAAMEprZ10AzEJf9veSNz1t1mVMXF/292ZdAgDAsiHssCrVr38j3T3rMiauqtJvmnUVAADLg2FsAADAKAk7AADAKAk7AADAKAk7AADAKAk7AADAKAk7AADAKAk7AADAKAk7AADAKAk7AADAKK2ddQEAADxRX/b3kjc9bdZlTFxf9vdmXQKrjLADALDM1K9/I9096zImrqrSb5p1FawmhrEBAACjJOwAAACjJOwAMHpVdWJVXV9Vd1bVHVX1Swv0Obuqvl5Vtwy3X5tFrQBMzlQ/s1NVJyZ5T5Jjk3SSK7r796ZZAwCr0qNJfrm7b66qpya5qaqu6+479+j3f3X3q2ZQHwBLYNpXdnafbE5L8oIkF1fVaVOuAYBVprvv6+6bh+VvJtme5PjZVgXAUptq2HGyAWDWqmp9kucm+fQCm3+8qj5bVX9SVc/ay31srqptVbVt165dS1UqAIdoZlNPL3ayqarNSTYnyUknnTT9wlg1qmrWJUzcUUcdNesSYFmrqiOS/HGSN3T3N/bYfHOSH+nuR6rqvCT/OckpC91Pd1+R5Iok2bBhw/jmBwYYiZmEnb2dbJxAmIZpfndBVY3yuxJgpamqwzJ37nlvd39oz+3zz0fdfW1V/fuqOqa7vzLNOgGYnKnPxravkw0ATFrNXcq9Msn27v7dRfo8Y+iXqjorc+fIr06vSgAmbdqzse3zZAMAS+CFSX42yW1VdcvQ9qtJTkqS7n5nkp9O8q+q6tEk/3eSC9plWYAVbdrD2BY82XT3tVOuA4BVpLtvSLLXD+p19+8n+f3pVATANEw17OzPyQYAAGASZjYbGwAAizNrKBw6YQcAYJkxayhMxtRnYwMAAJgGYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABglYQcAABiltbMuAFaSqpraft19UMcCYPVynoInEnbgAHhjB2A5c56CJzKMDQAAGCVhBwAAGCVhBwAAGCVhBwAAGCVhBwAAGCVhBwAAGCVhBwAAGCVhBwAAGCVhBwAAGCVhB4BVoarOrarPV9WOqrpkge1Prqr3D9s/XVXrZ1AmABMk7AAwelW1JskfJHllktOSXFhVp+3RbVOSh7v7x5K8NclvTbdKACZN2AFgNTgryY7u/mJ3/22S9yXZuEefjUmuHpY/mOSlVVVTrBGACVs76wL25qabbvpKVX151nXAITomyVdmXQQcoh+ZdQGH6Pgk98xb35nk+Yv16e5Hq+rrSZ6eBX5/q2pzks3D6iNV9fmJVwzT4zzFGCx4nlrWYae71826BjhUVbWtuzfMug5gcrr7iiRXzLoOmATnKcbMMDYAVoN7k5w4b/2EoW3BPlW1NsnTknx1KtUBsCSEHQBWgxuTnFJVJ1fVDyS5IMnWPfpsTXLRsPzTSf68u3uKNQIwYct6GBuMhKEuMGPDZ3Bel+RjSdYkuaq776iq30iyrbu3JrkyyR9V1Y4kD2UuEMFq4DzFaJU/WgEAAGNkGBsAADBKwg4AADBKwg4skaq6qqoerKrbZ10LAOzJeYrVQNiBpfPuJOfOuggAWMS74zzFyAk7sES6+5OZm9EJAJYd5ylWA2EHAAAYJWEHAAAYJWEHAAAYJWEHAAAYJWEHlkhVbUnyl0meWVU7q2rTrGsCgN2cp1gNqrtnXQMAAMDEubIDAACMkrADAACMkrADAACMkrADAACMkrADAACMkrADAACMkrADAACM0v8D0kyM1F1skagAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1008x864 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 질문과 답변의 길이 확인\n",
    "q_lengths = [len(sentence.split()) for sentence in df['Q']]\n",
    "a_lengths = [len(sentence.split()) for sentence in df['A']]\n",
    "\n",
    "print(f'q_lengths min : {min(q_lengths)}')\n",
    "print(f'q_lengths max : {max(q_lengths)}')\n",
    "print(f'q_lengths mean : {np.mean(q_lengths)}')\n",
    "print('=============================')\n",
    "print(f'a_lengths min : {min(a_lengths)}')\n",
    "print(f'a_lengths max : {max(a_lengths)}')\n",
    "print(f'a_lengths mean : {np.mean(a_lengths)}')\n",
    "\n",
    "# 그래프로 분포 확인\n",
    "plt.figure(figsize=(14, 12))\n",
    "plt.subplot(2, 2, 1)\n",
    "plt.hist(q_lengths, bins=15)\n",
    "plt.subplot(2, 2, 2)\n",
    "plt.hist(a_lengths, bins=21)\n",
    "plt.subplot(2, 2, 3)\n",
    "plt.boxplot(q_lengths)\n",
    "plt.subplot(2, 2, 4)\n",
    "plt.boxplot(a_lengths)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "contained-label",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>개당황</td>\n",
       "      <td>다음부터는 더 많이 아세요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>개졸려</td>\n",
       "      <td>낮잠을 잠깐 자도 괜찮아요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>개좋아</td>\n",
       "      <td>저도 좋아해주세요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>건방져</td>\n",
       "      <td>기분이 나쁘셨나봐요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>겁난다</td>\n",
       "      <td>용기 내보세요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Q                A  label\n",
       "89   개당황  다음부터는 더 많이 아세요.      0\n",
       "93   개졸려  낮잠을 잠깐 자도 괜찮아요.      0\n",
       "94   개좋아       저도 좋아해주세요.      0\n",
       "118  건방져      기분이 나쁘셨나봐요.      0\n",
       "123  겁난다         용기 내보세요.      0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017</th>\n",
       "      <td>배에서 꾸룩꾸룩 소리 남</td>\n",
       "      <td>꼬르륵</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018</th>\n",
       "      <td>배에서 소리 남</td>\n",
       "      <td>꼬르륵</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2366</th>\n",
       "      <td>새로운 마음으로</td>\n",
       "      <td>시작!</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2367</th>\n",
       "      <td>새로운 마음으로 시작</td>\n",
       "      <td>시작!</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3652</th>\n",
       "      <td>이 세상에서 누가 제일 예뻐</td>\n",
       "      <td>저요!</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Q    A  label\n",
       "2017    배에서 꾸룩꾸룩 소리 남  꼬르륵      0\n",
       "2018         배에서 소리 남  꼬르륵      0\n",
       "2366         새로운 마음으로  시작!      0\n",
       "2367      새로운 마음으로 시작  시작!      0\n",
       "3652  이 세상에서 누가 제일 예뻐  저요!      0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 평균 길이(3개)를 갖는 문장들을 살짝 살펴보자\n",
    "display(df[df['Q'].apply(lambda x : len(x)) == 3][:5])\n",
    "display(df[df['A'].apply(lambda x : len(x)) == 3][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "enhanced-organ",
   "metadata": {},
   "source": [
    "> #### note:\n",
    "> - 원본 데이터의 전체 문장 쌍의 갯수는 11823개이고,\n",
    "> - Q와 A 둘 다 약 3개 정도의 평균 문장 내 단어 개수를 보여준다.(굉장히 짧은 것들이 많다...)\n",
    "> - 그리고 **중복된 답변**이 많은 것 같다.\n",
    "> ##### 데이터도 살짝 적고 학습될 문장 자체도 살짝 적은 것 같다.(뭔가 있어도 최대한 데이터를 살리는 쪽으로 가야겠다.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accurate-pride",
   "metadata": {},
   "source": [
    "#### # punctuation 전처리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "described-possession",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================== \n",
      " Q 문장\n",
      ", 개수 : 39\n",
      "' 개수 : 2\n",
      ". 개수 : 2148\n",
      "? 개수 : 1705\n",
      "! 개수 : 88\n",
      "\" 개수 : 0\n",
      "# 개수 : 0\n",
      "% 개수 : 2\n",
      ") 개수 : 0\n",
      "\n",
      "========================== \n",
      " A 문장\n",
      ", 개수 : 36\n",
      "' 개수 : 2\n",
      ". 개수 : 11400\n",
      "? 개수 : 94\n",
      "! 개수 : 223\n",
      "\" 개수 : 0\n",
      "# 개수 : 0\n",
      "% 개수 : 0\n",
      ") 개수 : 0\n"
     ]
    }
   ],
   "source": [
    "# punctuation이 별로 없는 것 같은데 있는지 확인해보자.\n",
    "punctuations = [\",\", \"'\", \".\", \"?\", \"!\", '\"', \"#\", \"%\", \")\"]\n",
    "\n",
    "print('========================== \\n Q 문장')\n",
    "for punc in punctuations:\n",
    "    print(f\"{punc} 개수 : {df['Q'].apply(lambda x: punc in x).sum()}\")\n",
    "\n",
    "print('')\n",
    "print('========================== \\n A 문장')\n",
    "for punc in punctuations:\n",
    "    print(f\"{punc} 개수 : {df['A'].apply(lambda x: punc in x).sum()}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "favorite-tuner",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7318</th>\n",
       "      <td>윤종신 '좋니' 노래 가사가 엄청 와닿네</td>\n",
       "      <td>노래 가사가 공감이 많이 되네요.</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9827</th>\n",
       "      <td>사귀는 사이인데 '야'라고 해. 기분 나쁜데 어떡해?</td>\n",
       "      <td>그건 좀 아닌 거 같다고 바로 말하세요.</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  Q                       A  label\n",
       "7318         윤종신 '좋니' 노래 가사가 엄청 와닿네      노래 가사가 공감이 많이 되네요.      1\n",
       "9827  사귀는 사이인데 '야'라고 해. 기분 나쁜데 어떡해?  그건 좀 아닌 거 같다고 바로 말하세요.      2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2059</th>\n",
       "      <td>베터리 15%야</td>\n",
       "      <td>미리 충전하세요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2060</th>\n",
       "      <td>베터리 겨우 15%야</td>\n",
       "      <td>미리 충전하세요.</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Q          A  label\n",
       "2059     베터리 15%야  미리 충전하세요.      0\n",
       "2060  베터리 겨우 15%야  미리 충전하세요.      0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# '는 왜 있는 걸까?\n",
    "display(df[df['Q'].apply(lambda x: \"'\" in x)])\n",
    "\n",
    "# %는 왜 있는 걸까?\n",
    "display(df[df['Q'].apply(lambda x: \"%\" in x)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "median-arabic",
   "metadata": {},
   "outputs": [],
   "source": [
    "# '와 %도 .과 마찬가지로 그냥 띄어쓰기를 해서 한 단어로만 인식하지 않게 해주면 되겠다.\n",
    "# 전처리 함수\n",
    "def preprocess_sentence(sentence):\n",
    "    sentence = re.sub(r\"([?.!,'%])\", r\" \\1 \", sentence) # punctuation과 단어 사이에 띄어쓰기를 넣어준다.\n",
    "    sentence = sentence.strip() # 뒤에 공백 없도록 해준다.\n",
    "    return sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "numerical-schema",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Q</th>\n",
       "      <th>A</th>\n",
       "      <th>label</th>\n",
       "      <th>pr_Q</th>\n",
       "      <th>pr_A</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12시 땡!</td>\n",
       "      <td>하루가 또 가네요.</td>\n",
       "      <td>0</td>\n",
       "      <td>12시 땡 !</td>\n",
       "      <td>하루가 또 가네요 .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1지망 학교 떨어졌어</td>\n",
       "      <td>위로해 드립니다.</td>\n",
       "      <td>0</td>\n",
       "      <td>1지망 학교 떨어졌어</td>\n",
       "      <td>위로해 드립니다 .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3박4일 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "      <td>3박4일 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠 .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3박4일 정도 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠.</td>\n",
       "      <td>0</td>\n",
       "      <td>3박4일 정도 놀러가고 싶다</td>\n",
       "      <td>여행은 언제나 좋죠 .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PPL 심하네</td>\n",
       "      <td>눈살이 찌푸려지죠.</td>\n",
       "      <td>0</td>\n",
       "      <td>PPL 심하네</td>\n",
       "      <td>눈살이 찌푸려지죠 .</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Q            A  label             pr_Q          pr_A\n",
       "0           12시 땡!   하루가 또 가네요.      0          12시 땡 !   하루가 또 가네요 .\n",
       "1      1지망 학교 떨어졌어    위로해 드립니다.      0      1지망 학교 떨어졌어    위로해 드립니다 .\n",
       "2     3박4일 놀러가고 싶다  여행은 언제나 좋죠.      0     3박4일 놀러가고 싶다  여행은 언제나 좋죠 .\n",
       "3  3박4일 정도 놀러가고 싶다  여행은 언제나 좋죠.      0  3박4일 정도 놀러가고 싶다  여행은 언제나 좋죠 .\n",
       "4          PPL 심하네   눈살이 찌푸려지죠.      0          PPL 심하네   눈살이 찌푸려지죠 ."
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 새로운 컬럼으로 전처리되 값을 추가해준다.\n",
    "df['pr_Q'] = df['Q'].apply(lambda x: preprocess_sentence(x))\n",
    "df['pr_A'] = df['A'].apply(lambda x: preprocess_sentence(x))\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "published-ceiling",
   "metadata": {},
   "source": [
    "## Step 3. SubwordTextEncoder 사용하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "satisfactory-spouse",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['12시 땡 !', '1지망 학교 떨어졌어', '3박4일 놀러가고 싶다', '3박4일 정도 놀러가고 싶다', 'PPL 심하네']\n",
      "========================\n",
      "['하루가 또 가네요 .', '위로해 드립니다 .', '여행은 언제나 좋죠 .', '여행은 언제나 좋죠 .', '눈살이 찌푸려지죠 .']\n"
     ]
    }
   ],
   "source": [
    "# tokenize할 때 Q, A 동시에 넣어줘야 하는데\n",
    "# 동시에 넣기 쉽게 리스트화 해주기\n",
    "q_list = list(df['pr_Q'])\n",
    "a_list = list(df['pr_A'])\n",
    "\n",
    "print(q_list[:5])\n",
    "print('========================')\n",
    "print(a_list[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "human-domestic",
   "metadata": {},
   "source": [
    "#### # 토큰화 진행"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cardiovascular-champagne",
   "metadata": {},
   "source": [
    "> #### note:\n",
    "> - 개인적으로 여태까지 진행했던 NLP 프로젝트 경험상 학습 시간에 비해 제대로 된 문장이 출력된 적이 별로 없었는데, `vocabulary`나 `임베딩 벡터`가 애초에 좋아야겠다라는 생각을 많이 했었다.\n",
    "> - 이번에는 토큰화하는 단어사전의 target_vocab_size 파라미터로 약 16000개로 넣어볼 예정이다(실습 때는 약 8000개)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "sufficient-medium",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "tokenizer = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(q_list + a_list, target_vocab_size=2**14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "stone-remainder",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "토큰화된 마지막 단어 번호 : 21862\n",
      "START_TOKEN의 번호 : [21863]\n",
      "END_TOKEN의 번호   : [21864]\n"
     ]
    }
   ],
   "source": [
    "# 시작 토큰과 종료 토큰에 고유한 정수 부여.\n",
    "START_TOKEN, END_TOKEN = [tokenizer.vocab_size], [tokenizer.vocab_size + 1]\n",
    "\n",
    "print(f'토큰화된 마지막 단어 번호 : {tokenizer.vocab_size-1}')\n",
    "print(f'START_TOKEN의 번호 : {START_TOKEN}')\n",
    "print(f'END_TOKEN의 번호   : {END_TOKEN}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "lucky-breath",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21865\n",
      "26\n"
     ]
    }
   ],
   "source": [
    "# 시작 토큰과 종료 토큰을 고려하여 +2를 하여 단어장의 크기 산정\n",
    "VOCAB_SIZE = tokenizer.vocab_size + 2\n",
    "\n",
    "# 문장의 최대 길이를 정하다.\n",
    "# 데이터가 굉장히 적다.\n",
    "# 그래서 다 살리는 것이 낫다.\n",
    "# 그래서 그냥 65으로 정했다.\n",
    "# 전처리된 후의 max 문장 길이를 뽑아보자.\n",
    "q_lengths_after = [len(sentence.split()) for sentence in df['pr_Q']]\n",
    "a_lengths_after = [len(sentence.split()) for sentence in df['pr_A']]\n",
    "\n",
    "# 둘 중에 가장 긴 것으로 정한다.\n",
    "MAXLEN = max(max(q_lengths_after), max(a_lengths_after))\n",
    "\n",
    "# 그리고 START_TOKEN과 ENDTOKEN이 들어갈 것이니 2를 더해준다.\n",
    "MAXLEN = MAXLEN +2\n",
    "print(VOCAB_SIZE)\n",
    "print(MAXLEN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "inner-technique",
   "metadata": {},
   "source": [
    "#### # 정수 인코딩 & 패딩하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "liked-immune",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "원문 : 1지망 학교 떨어졌어\n",
      "인코딩 : [21540, 580, 1066]\n",
      "디코딩 : 1지망 학교 떨어졌어\n"
     ]
    }
   ],
   "source": [
    "# 현재 tokenizer에는 질문, 답변의 단어들이 정수로 토큰화되어 있다.\n",
    "# 정수 인코딩 잘 되는지 확인\n",
    "print('원문 :', q_list[1])\n",
    "print('인코딩 :', tokenizer.encode(q_list[1]))\n",
    "print('디코딩 :', tokenizer.decode(tokenizer.encode(q_list[1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "contemporary-oxford",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 정수 인코딩, 최대 길이를 초과하는 샘플 제거, 패딩하는 함수 정의\n",
    "def tokenize_and_filter(inputs_1, inputs_2):\n",
    "    tokenized_inputs_1, tokenized_inputs_2 = [], []\n",
    "  \n",
    "    for (sentence1, sentence2) in zip(inputs_1, inputs_2):\n",
    "        # 정수 인코딩 과정에서 시작 토큰과 종료 토큰을 추가\n",
    "        # 인코딩 결과가 리스트임\n",
    "        # START_TOKEN, END_TOKEN도 리스트인 상태여서 '+'로 붙일 수 있음\n",
    "        sentence1 = START_TOKEN + tokenizer.encode(sentence1) + END_TOKEN\n",
    "        sentence2 = START_TOKEN + tokenizer.encode(sentence2) + END_TOKEN\n",
    "\n",
    "        # 최대 길이 이하인 경우에만 데이터셋으로 허용\n",
    "        if len(sentence1) <= MAXLEN and len(sentence2) <= MAXLEN:\n",
    "            tokenized_inputs_1.append(sentence1)\n",
    "            tokenized_inputs_2.append(sentence2)\n",
    "  \n",
    "    # 최대 길이에 맞춰 모든 데이터셋을 패딩\n",
    "    tokenized_inputs_1 = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_inputs_1, maxlen=MAXLEN, padding='post')\n",
    "    tokenized_inputs_2 = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenized_inputs_2, maxlen=MAXLEN, padding='post')\n",
    "  \n",
    "    return tokenized_inputs_1, tokenized_inputs_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spanish-moldova",
   "metadata": {},
   "source": [
    "> - MAXLEN에 맞춰서 제외된 데이터가 있을 테니 `tokenize_and_filter()` 진행 후 데이터 개수 재확인"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "later-hypothesis",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "단어장의 크기 : 21865\n",
      "필터링 전의 데이터 개수: 11823\n",
      "필터링 후의 질문 샘플 개수: 11823\n",
      "필터링 후의 답변 샘플 개수: 11823\n"
     ]
    }
   ],
   "source": [
    "questions, answers = tokenize_and_filter(q_list, a_list)\n",
    "\n",
    "print('단어장의 크기 :',(VOCAB_SIZE))\n",
    "print('필터링 전의 데이터 개수: {}'.format(len(df)))\n",
    "print('필터링 후의 질문 샘플 개수: {}'.format(len(questions)))\n",
    "print('필터링 후의 답변 샘플 개수: {}'.format(len(answers)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "lucky-mortality",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[21863 21576 18404    24 21864     0     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0]\n",
      "[21863  2038    44  3869     1 21864     0     0     0     0     0     0\n",
      "     0     0     0     0     0     0     0     0     0     0     0     0\n",
      "     0     0]\n"
     ]
    }
   ],
   "source": [
    "# 패딩됐는지 확인해보기\n",
    "print(questions[0])\n",
    "print(answers[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "changing-tiffany",
   "metadata": {},
   "source": [
    "#### # 데이터셋화( + 교사 강요를 위한 전처리)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "isolated-canadian",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "BATCH_SIZE = 64\n",
    "BUFFER_SIZE = len(questions)\n",
    "\n",
    "# 디코더는 이전의 target을 다음의 input으로 사용합니다.\n",
    "# 이에 따라 outputs에서는 START_TOKEN을 제거하겠습니다.\n",
    "dataset = tf.data.Dataset.from_tensor_slices((\n",
    "    {\n",
    "        'inputs': questions,\n",
    "        'dec_inputs': answers[:, :-1]\n",
    "    },\n",
    "    {\n",
    "        'outputs': answers[:, 1:]\n",
    "    },\n",
    "))\n",
    "\n",
    "dataset = dataset.cache()\n",
    "dataset = dataset.shuffle(BUFFER_SIZE)\n",
    "dataset = dataset.batch(BATCH_SIZE)\n",
    "dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "caroline-lexington",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<PrefetchDataset shapes: ({inputs: (None, 26), dec_inputs: (None, 25)}, {outputs: (None, 25)}), types: ({inputs: tf.int32, dec_inputs: tf.int32}, {outputs: tf.int32})>\n"
     ]
    }
   ],
   "source": [
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "applied-healing",
   "metadata": {},
   "source": [
    "> #### note:\n",
    "> - 각각의 shape을 통해 데이터가 잘 묶인 것을 확인할 수 있다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "going-church",
   "metadata": {},
   "source": [
    "## Step 4. 모델 구성하기"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rotary-hormone",
   "metadata": {},
   "source": [
    "#### # 모델 구현에 필요한  함수 정의\n",
    "- scaled_dot_product_attention\n",
    "- padding_mask\n",
    "- look_ahead_mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "demographic-hacker",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "# 스케일드 닷 프로덕트 어텐션 함수\n",
    "def scaled_dot_product_attention(query, key, value, mask):\n",
    "    # 어텐션 가중치는 Q와 K의 닷 프로덕트\n",
    "    matmul_qk = tf.matmul(query, key, transpose_b=True) #Q와 K가 모양이 같으니까 transpose_b를 통해 K 행렬을 전치시킴\n",
    "\n",
    "    # 가중치를 정규화\n",
    "    depth = tf.cast(tf.shape(key)[-1], tf.float32)\n",
    "    logits = matmul_qk / tf.math.sqrt(depth)\n",
    "\n",
    "    # 패딩에 마스크 추가\n",
    "    if mask is not None:\n",
    "        logits += (mask * -1e9)\n",
    "\n",
    "    # softmax적용\n",
    "    attention_weights = tf.nn.softmax(logits, axis=-1)\n",
    "\n",
    "    # 최종 어텐션은 가중치와 V의 닷 프로덕트\n",
    "    output = tf.matmul(attention_weights, value)\n",
    "    return output\n",
    "\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "informal-throat",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "# 패딩 마스킹을 구현한 함수\n",
    "def create_padding_mask(x):\n",
    "    mask = tf.cast(tf.math.equal(x, 0), tf.float32)\n",
    "    # (batch_size, 1, 1, sequence length)\n",
    "    return mask[:, tf.newaxis, tf.newaxis, :]\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "utility-flashing",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[[0. 0. 1. 0. 1.]]]\n",
      "\n",
      "\n",
      " [[[1. 1. 1. 0. 0.]]]], shape=(2, 1, 1, 5), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# 실험해보기\n",
    "print(create_padding_mask(tf.constant([[1, 2, 0, 3, 0], [0, 0, 0, 4, 5]])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "super-bulgarian",
   "metadata": {},
   "source": [
    "> #### note:\n",
    "> - 위와 같이 **0**으로 패딩된 곳은 1로 바뀌고, 아닌 **일반** 정수는 0으로 바뀌는, 즉 마스킹 해야할 부분을 1로 만들어주는 함수인 듯하다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "dominant-mystery",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "# 룩 어헤드 마스크 구현하기\n",
    "def create_look_ahead_mask(x):\n",
    "    seq_len = tf.shape(x)[1]\n",
    "    look_ahead_mask = 1 - tf.linalg.band_part(tf.ones((seq_len, seq_len)), -1, 0)\n",
    "    padding_mask = create_padding_mask(x)\n",
    "    return tf.maximum(look_ahead_mask, padding_mask)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "driven-rendering",
   "metadata": {},
   "source": [
    "> - 이게 어떤 일을 하는지 확인해보자!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "banner-triangle",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 4), dtype=int32, numpy=\n",
       "array([[ 0,  1,  2,  3],\n",
       "       [-1,  0,  1,  2],\n",
       "       [-2, -1,  0,  1],\n",
       "       [-3, -2, -1,  0]], dtype=int32)>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 실험\n",
    "test_tf = tf.constant([[ 0,  1,  2, 3],\n",
    " [-1,  0,  1, 2],\n",
    " [-2, -1,  0, 1],\n",
    " [-3, -2, -1, 0]],)\n",
    "\n",
    "test_tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "duplicate-soviet",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 4), dtype=int32, numpy=\n",
       "array([[ 0,  0,  0,  0],\n",
       "       [-1,  0,  0,  0],\n",
       "       [-2, -1,  0,  0],\n",
       "       [-3, -2, -1,  0]], dtype=int32)>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 실험\n",
    "tf.linalg.band_part(test_tf, -1, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "important-framing",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 4), dtype=int32, numpy=\n",
       "array([[1, 1, 1, 1],\n",
       "       [2, 1, 1, 1],\n",
       "       [3, 2, 1, 1],\n",
       "       [4, 3, 2, 1]], dtype=int32)>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 실험\n",
    "1 - tf.linalg.band_part(test_tf, -1, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "asian-layer",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 4), dtype=int32, numpy=\n",
       "array([[0, 1, 2, 3],\n",
       "       [0, 0, 1, 2],\n",
       "       [0, 0, 0, 1],\n",
       "       [0, 0, 0, 0]], dtype=int32)>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 실험\n",
    "tf.linalg.band_part(test_tf, 0, -3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "regulation-drive",
   "metadata": {},
   "source": [
    "> #### note:\n",
    "> - `tf.linalg.band_part()` 함수를 이해할 수 있는 쉬운 설명을 못 찾아서 정확히 이해는 안되지만,\n",
    "> - `num_lower`(두번째 인자)와 `num_upper`(세번째 인자) 사이의 값이 아닌 것들은 0으로 만들어 버린단다.\n",
    "> - 근데 이게 약간 upper right corner 쪽만 그렇게 하려는 것 같다.\n",
    "> - 그래서 이 경우엔 문장 내 아직 보면 안되는 단어들을 일단 0으로 만들어주고,\n",
    "> - 나머지 수는 음수로 만들어준다.\n",
    "> - 그리고 0으로 된 것들(패딩 포함)은 다시 1로 바꾸는 과정을 해서 결국에 **내가 보고 싶은 건 음수, 가려야 하는 건 1**로 만들어서\n",
    "> - 그 1인 것들이 마스킹 되게끔 하는 것 같다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "searching-square",
   "metadata": {},
   "source": [
    "#### # 모델 구현에 필요한 레이어(subclassing) 및 모델 생성 함수 만들기\n",
    "- Positional Encoding\n",
    "- Multi-Head Attention\n",
    "- 인코더 레이어 연결 함수\n",
    "- 인코더 층 쌓기 함수\n",
    "- 디코더 레이어 연결 함수\n",
    "- 디코더 층 쌓기 함수\n",
    "- ##### 하나로 합친 트랜스포머 모델 생성 함수"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "interior-phrase",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "# 포지셔널 인코딩 레이어\n",
    "class PositionalEncoding(tf.keras.layers.Layer):\n",
    "\n",
    "    def __init__(self, position, d_model):\n",
    "        super(PositionalEncoding, self).__init__()\n",
    "        self.pos_encoding = self.positional_encoding(position, d_model)\n",
    "\n",
    "    def get_angles(self, position, i, d_model):\n",
    "        angles = 1 / tf.pow(10000, (2 * (i // 2)) / tf.cast(d_model, tf.float32))\n",
    "        return position * angles\n",
    "\n",
    "    def positional_encoding(self, position, d_model):\n",
    "        # 각도 배열 생성\n",
    "        angle_rads = self.get_angles(\n",
    "            position=tf.range(position, dtype=tf.float32)[:, tf.newaxis],\n",
    "            i=tf.range(d_model, dtype=tf.float32)[tf.newaxis, :],\n",
    "            d_model=d_model)\n",
    "\n",
    "        # 배열의 짝수 인덱스에는 sin 함수 적용\n",
    "        sines = tf.math.sin(angle_rads[:, 0::2])\n",
    "        # 배열의 홀수 인덱스에는 cosine 함수 적용\n",
    "        cosines = tf.math.cos(angle_rads[:, 1::2])\n",
    "\n",
    "        # sin과 cosine이 교차되도록 재배열\n",
    "        pos_encoding = tf.stack([sines, cosines], axis=0)\n",
    "        pos_encoding = tf.transpose(pos_encoding,[1, 2, 0]) \n",
    "        pos_encoding = tf.reshape(pos_encoding, [position, d_model])\n",
    "\n",
    "        pos_encoding = pos_encoding[tf.newaxis, ...]\n",
    "        return tf.cast(pos_encoding, tf.float32)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        return inputs + self.pos_encoding[:, :tf.shape(inputs)[1], :]\n",
    "\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "premier-black",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "class MultiHeadAttention(tf.keras.layers.Layer):\n",
    "\n",
    "    def __init__(self, d_model, num_heads, name=\"multi_head_attention\"):\n",
    "        super(MultiHeadAttention, self).__init__(name=name)\n",
    "        self.num_heads = num_heads\n",
    "        self.d_model = d_model\n",
    "    \n",
    "        assert d_model % self.num_heads == 0\n",
    "    \n",
    "        self.depth = d_model // self.num_heads\n",
    "\n",
    "        self.query_dense = tf.keras.layers.Dense(units=d_model)\n",
    "        self.key_dense = tf.keras.layers.Dense(units=d_model)\n",
    "        self.value_dense = tf.keras.layers.Dense(units=d_model)\n",
    "\n",
    "        self.dense = tf.keras.layers.Dense(units=d_model)\n",
    "\n",
    "    def split_heads(self, inputs, batch_size):\n",
    "        inputs = tf.reshape(\n",
    "            inputs, shape=(batch_size, -1, self.num_heads, self.depth))\n",
    "        return tf.transpose(inputs, perm=[0, 2, 1, 3])\n",
    "\n",
    "    def call(self, inputs):\n",
    "        query, key, value, mask = inputs['query'], inputs['key'], inputs[\n",
    "            'value'], inputs['mask']\n",
    "        batch_size = tf.shape(query)[0]\n",
    "\n",
    "        # Q, K, V에 각각 Dense를 적용합니다\n",
    "        query = self.query_dense(query)\n",
    "        key = self.key_dense(key)\n",
    "        value = self.value_dense(value)\n",
    "\n",
    "        # 병렬 연산을 위한 머리를 여러 개 만듭니다\n",
    "        query = self.split_heads(query, batch_size)\n",
    "        key = self.split_heads(key, batch_size)\n",
    "        value = self.split_heads(value, batch_size)\n",
    "\n",
    "        # 스케일드 닷 프로덕트 어텐션 함수\n",
    "        scaled_attention = scaled_dot_product_attention(query, key, value, mask)\n",
    "\n",
    "        scaled_attention = tf.transpose(scaled_attention, perm=[0, 2, 1, 3])\n",
    "\n",
    "        # 어텐션 연산 후에 각 결과를 다시 연결(concatenate)합니다\n",
    "        concat_attention = tf.reshape(scaled_attention,\n",
    "                                  (batch_size, -1, self.d_model))\n",
    "\n",
    "        # 최종 결과에도 Dense를 한 번 더 적용합니다\n",
    "        outputs = self.dense(concat_attention)\n",
    "\n",
    "        return outputs\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "greater-alexander",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "# 인코더 하나의 레이어를 함수로 구현.\n",
    "# 이 하나의 레이어 안에는 두 개의 서브 레이어가 존재합니다.\n",
    "def encoder_layer(units, d_model, num_heads, dropout, name=\"encoder_layer\"):\n",
    "    inputs = tf.keras.Input(shape=(None, d_model), name=\"inputs\")\n",
    "\n",
    "    # 패딩 마스크 사용\n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name=\"padding_mask\")\n",
    "\n",
    "    # 첫 번째 서브 레이어 : 멀티 헤드 어텐션 수행 (셀프 어텐션)\n",
    "    attention = MultiHeadAttention(\n",
    "        d_model, num_heads, name=\"attention\")({\n",
    "            'query': inputs,\n",
    "            'key': inputs,\n",
    "            'value': inputs,\n",
    "           'mask': padding_mask\n",
    "        })\n",
    "    \n",
    "    # 어텐션의 결과는 Dropout과 Layer Normalization이라는 훈련을 돕는 테크닉을 수행\n",
    "    attention = tf.keras.layers.Dropout(rate=dropout)(attention)\n",
    "    attention = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(inputs + attention)\n",
    "\n",
    "    # 두 번째 서브 레이어 : 2개의 완전연결층\n",
    "    outputs = tf.keras.layers.Dense(units=units, activation='relu')(attention)\n",
    "    outputs = tf.keras.layers.Dense(units=d_model)(outputs)\n",
    "\n",
    "    # 완전연결층의 결과는 Dropout과 LayerNormalization이라는 훈련을 돕는 테크닉을 수행\n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(outputs)\n",
    "    outputs = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(attention + outputs)\n",
    "\n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, padding_mask], outputs=outputs, name=name)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "given-lounge",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "def encoder(vocab_size,\n",
    "            num_layers,\n",
    "            units,\n",
    "            d_model,\n",
    "            num_heads,\n",
    "            dropout,\n",
    "            name=\"encoder\"):\n",
    "    inputs = tf.keras.Input(shape=(None,), name=\"inputs\")\n",
    "\n",
    "    # 패딩 마스크 사용\n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name=\"padding_mask\")\n",
    "\n",
    "    # 임베딩 레이어\n",
    "    embeddings = tf.keras.layers.Embedding(vocab_size, d_model)(inputs)\n",
    "    embeddings *= tf.math.sqrt(tf.cast(d_model, tf.float32))\n",
    "\n",
    "    # 포지셔널 인코딩\n",
    "    embeddings = PositionalEncoding(vocab_size, d_model)(embeddings)\n",
    "\n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(embeddings)\n",
    "\n",
    "    # num_layers만큼 쌓아올린 인코더의 층.\n",
    "    for i in range(num_layers):\n",
    "        outputs = encoder_layer(\n",
    "            units=units,\n",
    "            d_model=d_model,\n",
    "            num_heads=num_heads,\n",
    "            dropout=dropout,\n",
    "            name=\"encoder_layer_{}\".format(i),\n",
    "        )([outputs, padding_mask])\n",
    "\n",
    "    return tf.keras.Model(\n",
    "          inputs=[inputs, padding_mask], outputs=outputs, name=name)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "governmental-petroleum",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "# 디코더 하나의 레이어를 함수로 구현.\n",
    "# 이 하나의 레이어 안에는 세 개의 서브 레이어가 존재합니다.\n",
    "def decoder_layer(units, d_model, num_heads, dropout, name=\"decoder_layer\"):\n",
    "    inputs = tf.keras.Input(shape=(None, d_model), name=\"inputs\")\n",
    "    enc_outputs = tf.keras.Input(shape=(None, d_model), name=\"encoder_outputs\")\n",
    "    look_ahead_mask = tf.keras.Input(\n",
    "        shape=(1, None, None), name=\"look_ahead_mask\")\n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name='padding_mask')\n",
    "\n",
    "    # 첫 번째 서브 레이어 : 멀티 헤드 어텐션 수행 (셀프 어텐션)\n",
    "    attention1 = MultiHeadAttention(\n",
    "        d_model, num_heads, name=\"attention_1\")(inputs={\n",
    "            'query': inputs,\n",
    "            'key': inputs,\n",
    "            'value': inputs,\n",
    "            'mask': look_ahead_mask\n",
    "        })\n",
    "\n",
    "    # 멀티 헤드 어텐션의 결과는 LayerNormalization이라는 훈련을 돕는 테크닉을 수행\n",
    "    attention1 = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(attention1 + inputs)\n",
    "\n",
    "    # 두 번째 서브 레이어 : 마스크드 멀티 헤드 어텐션 수행 (인코더-디코더 어텐션)\n",
    "    attention2 = MultiHeadAttention(\n",
    "        d_model, num_heads, name=\"attention_2\")(inputs={\n",
    "            'query': attention1,\n",
    "            'key': enc_outputs,\n",
    "            'value': enc_outputs,\n",
    "            'mask': padding_mask\n",
    "        })\n",
    "\n",
    "    # 마스크드 멀티 헤드 어텐션의 결과는\n",
    "    # Dropout과 LayerNormalization이라는 훈련을 돕는 테크닉을 수행\n",
    "    attention2 = tf.keras.layers.Dropout(rate=dropout)(attention2)\n",
    "    attention2 = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(attention2 + attention1)\n",
    "\n",
    "    # 세 번째 서브 레이어 : 2개의 완전연결층\n",
    "    outputs = tf.keras.layers.Dense(units=units, activation='relu')(attention2)\n",
    "    outputs = tf.keras.layers.Dense(units=d_model)(outputs)\n",
    "\n",
    "    # 완전연결층의 결과는 Dropout과 LayerNormalization 수행\n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(outputs)\n",
    "    outputs = tf.keras.layers.LayerNormalization(\n",
    "        epsilon=1e-6)(outputs + attention2)\n",
    "\n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, enc_outputs, look_ahead_mask, padding_mask],\n",
    "        outputs=outputs,\n",
    "        name=name)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "theoretical-address",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "def decoder(vocab_size,\n",
    "            num_layers,\n",
    "            units,\n",
    "            d_model,\n",
    "            num_heads,\n",
    "            dropout,\n",
    "            name='decoder'):\n",
    "    inputs = tf.keras.Input(shape=(None,), name='inputs')\n",
    "    enc_outputs = tf.keras.Input(shape=(None, d_model), name='encoder_outputs')\n",
    "    look_ahead_mask = tf.keras.Input(\n",
    "        shape=(1, None, None), name='look_ahead_mask')\n",
    "\n",
    "    # 패딩 마스크\n",
    "    padding_mask = tf.keras.Input(shape=(1, 1, None), name='padding_mask')\n",
    "  \n",
    "    # 임베딩 레이어\n",
    "    embeddings = tf.keras.layers.Embedding(vocab_size, d_model)(inputs)\n",
    "    embeddings *= tf.math.sqrt(tf.cast(d_model, tf.float32))\n",
    "\n",
    "    # 포지셔널 인코딩\n",
    "    embeddings = PositionalEncoding(vocab_size, d_model)(embeddings)\n",
    "\n",
    "    # Dropout이라는 훈련을 돕는 테크닉을 수행\n",
    "    outputs = tf.keras.layers.Dropout(rate=dropout)(embeddings)\n",
    "\n",
    "    for i in range(num_layers):\n",
    "        outputs = decoder_layer(\n",
    "            units=units,\n",
    "            d_model=d_model,\n",
    "            num_heads=num_heads,\n",
    "            dropout=dropout,\n",
    "            name='decoder_layer_{}'.format(i),\n",
    "        )(inputs=[outputs, enc_outputs, look_ahead_mask, padding_mask])\n",
    "\n",
    "    return tf.keras.Model(\n",
    "        inputs=[inputs, enc_outputs, look_ahead_mask, padding_mask],\n",
    "        outputs=outputs,\n",
    "        name=name)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "aware-batch",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "def transformer(vocab_size,\n",
    "                num_layers,\n",
    "                units,\n",
    "                d_model,\n",
    "                num_heads,\n",
    "                dropout,\n",
    "                name=\"transformer\"):\n",
    "    inputs = tf.keras.Input(shape=(None,), name=\"inputs\")\n",
    "    dec_inputs = tf.keras.Input(shape=(None,), name=\"dec_inputs\")\n",
    "\n",
    "    # 인코더에서 패딩을 위한 마스크\n",
    "    enc_padding_mask = tf.keras.layers.Lambda(\n",
    "        create_padding_mask, output_shape=(1, 1, None),\n",
    "        name='enc_padding_mask')(inputs)\n",
    "\n",
    "    # 디코더에서 미래의 토큰을 마스크 하기 위해서 사용합니다.\n",
    "    # 내부적으로 패딩 마스크도 포함되어져 있습니다.\n",
    "    look_ahead_mask = tf.keras.layers.Lambda(\n",
    "        create_look_ahead_mask,\n",
    "        output_shape=(1, None, None),\n",
    "        name='look_ahead_mask')(dec_inputs)\n",
    "\n",
    "    # 두 번째 어텐션 블록에서 인코더의 벡터들을 마스킹\n",
    "    # 디코더에서 패딩을 위한 마스크\n",
    "    dec_padding_mask = tf.keras.layers.Lambda(\n",
    "        create_padding_mask, output_shape=(1, 1, None),\n",
    "        name='dec_padding_mask')(inputs)\n",
    "\n",
    "    # 인코더\n",
    "    enc_outputs = encoder(\n",
    "        vocab_size=vocab_size,\n",
    "        num_layers=num_layers,\n",
    "        units=units,\n",
    "        d_model=d_model,\n",
    "        num_heads=num_heads,\n",
    "        dropout=dropout,\n",
    "    )(inputs=[inputs, enc_padding_mask])\n",
    "\n",
    "    # 디코더\n",
    "    dec_outputs = decoder(\n",
    "        vocab_size=vocab_size,\n",
    "        num_layers=num_layers,\n",
    "        units=units,\n",
    "        d_model=d_model,\n",
    "        num_heads=num_heads,\n",
    "        dropout=dropout,\n",
    "    )(inputs=[dec_inputs, enc_outputs, look_ahead_mask, dec_padding_mask])\n",
    "\n",
    "    # 완전연결층\n",
    "    outputs = tf.keras.layers.Dense(units=vocab_size, name=\"outputs\")(dec_outputs)\n",
    "\n",
    "    return tf.keras.Model(inputs=[inputs, dec_inputs], outputs=outputs, name=name)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "affecting-clock",
   "metadata": {},
   "source": [
    "#### # 모델 생성 및 컴파일"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "blind-exclusion",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"transformer\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "inputs (InputLayer)             [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dec_inputs (InputLayer)         [(None, None)]       0                                            \n",
      "__________________________________________________________________________________________________\n",
      "enc_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "encoder (Functional)            (None, None, 512)    14350848    inputs[0][0]                     \n",
      "                                                                 enc_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "look_ahead_mask (Lambda)        (None, 1, None, None 0           dec_inputs[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dec_padding_mask (Lambda)       (None, 1, 1, None)   0           inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "decoder (Functional)            (None, None, 512)    16454144    dec_inputs[0][0]                 \n",
      "                                                                 encoder[0][0]                    \n",
      "                                                                 look_ahead_mask[0][0]            \n",
      "                                                                 dec_padding_mask[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "outputs (Dense)                 (None, None, 21865)  11216745    decoder[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 42,021,737\n",
      "Trainable params: 42,021,737\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "tf.keras.backend.clear_session()\n",
    "\n",
    "# 하이퍼파라미터\n",
    "NUM_LAYERS = 2 # 인코더와 디코더의 층의 개수\n",
    "D_MODEL = 512 # 인코더와 디코더 내부의 입, 출력의 고정 차원\n",
    "NUM_HEADS = 8 # 멀티 헤드 어텐션에서의 헤드 수 \n",
    "UNITS = 512 # 피드 포워드 신경망의 은닉층의 크기\n",
    "DROPOUT = 0.1 # 드롭아웃의 비율\n",
    "\n",
    "model = transformer(\n",
    "    vocab_size=VOCAB_SIZE,\n",
    "    num_layers=NUM_LAYERS,\n",
    "    units=UNITS,\n",
    "    d_model=D_MODEL,\n",
    "    num_heads=NUM_HEADS,\n",
    "    dropout=DROPOUT)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "endless-cassette",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "def loss_function(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAXLEN - 1)) \n",
    "    # 여기 MAXLEN -1을 해주는 이유는 결과 값에서 START_TOKEN 없는 거로 훈련 돼서 나오니까\n",
    "  \n",
    "    loss = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "        from_logits=True, reduction='none')(y_true, y_pred)\n",
    "\n",
    "    mask = tf.cast(tf.not_equal(y_true, 0), tf.float32)\n",
    "    loss = tf.multiply(loss, mask)\n",
    "\n",
    "    return tf.reduce_mean(loss)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "conceptual-premium",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "\n",
    "    def __init__(self, d_model, warmup_steps=4000):\n",
    "        super(CustomSchedule, self).__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.d_model = tf.cast(self.d_model, tf.float32)\n",
    "\n",
    "        self.warmup_steps = warmup_steps\n",
    "\n",
    "    def __call__(self, step):\n",
    "        arg1 = tf.math.rsqrt(step)\n",
    "        arg2 = step * (self.warmup_steps**-1.5)\n",
    "\n",
    "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "heavy-continent",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'Train Step')"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEGCAYAAABYV4NmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuxklEQVR4nO3de5xcVZnv/8/T1fd7p9O5X8mFEJRrA8KAiuHqLccZ+BnUQ1RmGEccdTzneODoeBzOwdcwOnpGB0dBbiIjIKOSQUYGRS4ikAS5EwJNCCQhJCGXTtJJV3d1P78/9qpOpenqrq7U7upOf9+vV71q16q9135qd/V+aq+19t7m7oiIiBRaSbEDEBGRw5MSjIiIxEIJRkREYqEEIyIisVCCERGRWJQWO4Bimjhxos+ZM6fYYYiIjClPPPHEW+7eMtR84zrBzJkzh9WrVxc7DBGRMcXMXstlPjWRiYhILJRgREQkFkowIiISi1gTjJmdZ2ZrzazNzC4f4P0KM7s9vP+4mc3JeO+KUL7WzM7NKL/BzLaa2XP96ppgZveZ2cvhuSnOzyYiIoOLLcGYWQK4BjgfWAxcZGaL+812CbDT3ecD3wGuDssuBpYBRwPnAd8P9QHcFMr6uxz4rbsvAH4bXouISJHEeQRzMtDm7uvcvQu4DVjab56lwM1h+k5giZlZKL/N3ZPu/irQFurD3R8Cdgywvsy6bgb+SwE/i4iIDFOcCWY6sCHj9cZQNuA87p4C2oHmHJftb7K7bw7TbwKTB5rJzC41s9Vmtnrbtm25fA4REcnDYdnJ79E9CAa8D4G7X+vure7e2tIy5HlCw/LKtr38oe2tgtYpIjJWxZlgNgEzM17PCGUDzmNmpUADsD3HZfvbYmZTQ11Tga15R56nJf/4IB/70eMjvVoRkVEpzgSzClhgZnPNrJyo035Fv3lWAMvD9AXA/eHoYwWwLIwymwssAFYOsb7MupYDdxXgM4iISJ5iSzChT+VzwL3AGuAOd3/ezK40sw+H2a4Hms2sDfgSYeSXuz8P3AG8APwauMzdewDM7KfAo8CRZrbRzC4Jdf09cLaZvQycFV4XRWd3T7FWLSIyasR6LTJ3vwe4p1/Z1zKmO4ELsyx7FXDVAOUXZZl/O7DkUOItlPb93VSWJYaeUUTkMHZYdvIX26593cUOQUSk6JRgYrBrX1exQxARKTolmBjs2q8jGBERJZgCKi+NNme7mshERJRgCqkqdOzv2q8mMhERJZgCKi0xQJ38IiKgBFNQXaleQH0wIiKgBFNQyZ4owagPRkRECaZg3D3jCEZ9MCIiSjAFkgzJBdQHIyICSjAFowQjInIwJZgCSaaiC1xWlJboTH4REZRgCibZHR3BTG2opKOrR1dUFpFxTwmmQNJNZNMaqwDY3qGjGBEZ35RgCiTdRNaXYPYmixmOiEjRKcEUSPoIZnpIMG8pwYjIOKcEUyBdb0swaiITkfFNCaZA3tYHowQjIuOcEkyBJMOoscbqMqrLE2oiE5FxTwmmQNJHMJVlJTTXlquTX0TGPSWYAkknmPJEgom1FeqDEZFxTwmmQPrO5C8robmmQk1kIjLuKcEUSPpM/orSElrqynUEIyLjnhJMgaSbyCpKEzTXVLCjI0lvrxc5KhGR4lGCKZD0eTDlpSVMrC2n12GHLnopIuOYEkyBJFM9lCWMRIkxpSE6F+bN9s4iRyUiUjxKMAWSTPVSUZoAYEpDJaAEIyLjmxJMgSRTPVSURptzajrB7FaCEZHxSwmmQJLdvX0JZmJtBYkS0xGMiIxrSjAFkkz1Uh4STKLEmFRXoSMYERnXlGAKJGoiS/S9ntJQqSMYERnXYk0wZnaema01szYzu3yA9yvM7Pbw/uNmNifjvStC+VozO3eoOs1siZn90cyeMrPfm9n8OD9bf8lULxVlBzbnlPpKNrfvH8kQRERGldgSjJklgGuA84HFwEVmtrjfbJcAO919PvAd4Oqw7GJgGXA0cB7wfTNLDFHnvwAfd/fjgH8FvhrXZxtIZh8MREcwW3brcjEiMn7FeQRzMtDm7uvcvQu4DVjab56lwM1h+k5giZlZKL/N3ZPu/irQFuobrE4H6sN0A/BGTJ9rQF09vQc1kU1tqGRvMsWezu6RDENEZNQojbHu6cCGjNcbgVOyzePuKTNrB5pD+WP9lp0eprPV+efAPWa2H9gNvGugoMzsUuBSgFmzZg3vEw0imeqhsaqs7/Xk+gPnwtRVlmVbTETksHU4dfL/DfB+d58B3Ah8e6CZ3P1ad29199aWlpaCrTzZfXAfTPrOlpt2qR9GRManOBPMJmBmxusZoWzAecyslKhpa/sgyw5YbmYtwLHu/ngovx04rTAfIzeZZ/IDzGyqBmDDTiUYERmf4kwwq4AFZjbXzMqJOu1X9JtnBbA8TF8A3O/uHsqXhVFmc4EFwMpB6twJNJjZwlDX2cCaGD/b2yRTPZQnDmzOSXUVlJeWsHHHvpEMQ0Rk1IitDyb0qXwOuBdIADe4+/NmdiWw2t1XANcDt5hZG7CDKGEQ5rsDeAFIAZe5ew/AQHWG8r8A/s3MeokSzqfj+mwD6T9MuaTEmNFUxetKMCIyTsXZyY+73wPc06/saxnTncCFWZa9CrgqlzpD+S+AXxxiyHnrP0wZYNaEajbsVIIRkfHpcOrkL6r+Z/JD1A/z+nYlGBEZn5RgCiDV00uvM+ARzO7OFO37dC6MiIw/SjAF0He75LKDN+fMCdFQZTWTich4pARTAH0Jpn8T2YQwVFkd/SIyDinBFEAy1QO8vYksnWBeU4IRkXFICaYAkt0DN5HVV5YxsbacV7d1FCMsEZGiUoIpgHQTWXki8bb3jmip5ZVte0c6JBGRolOCKYBsTWQA85RgRGScUoIpgGyjyADmtdSwc183Ozq6RjosEZGiUoIpgL4+mNK3N5HNm1QLoKMYERl3lGAKoKsnexPZ/JaQYLYqwYjI+KIEUwDZRpFBdF+YitISHcGIyLijBFMA2U60BEiUGHMn1tCmIxgRGWeUYApgsFFkAPMn1fLSFiUYERlflGAKoO88mCwJZvG0ejbt2k/7fl30UkTGDyWYAjgwimzgzXnU1HoAXty8e8RiEhEpNiWYAjjQRPb2PhiAo0OCeUEJRkTGESWYAkimejGDsoQN+H5LXQXNNeWsUYIRkXFECaYAulLR7ZLNBk4wZsbiafU6ghGRcUUJpgCSqd6szWNpi6fW89KWvXT39I5QVCIixaUEUwDJVE/WDv60o6bW05Xq1QmXIjJuKMEUQLK7d8Cz+DMdM6MBgGc2tI9ESCIiRacEUwC5NJHNnVhDQ1UZT27YOUJRiYgU15AJxswWmtlvzey58PoYM/tq/KGNHclUD+WJwTelmXHczEaefH3XyAQlIlJkuRzBXAdcAXQDuPszwLI4gxprkqmhm8gAjp/VyNote9ibTI1AVCIixZVLgql295X9yrSHzJDs7h2ykx/g+FlNuMMzG3bFH5SISJHlkmDeMrN5gAOY2QXA5lijGmOiUWSD98EAHDejEYAnlWBEZBwozWGey4BrgUVmtgl4Ffh4rFGNMclUbkcwDdVlzGup4YnX1NEvIoe/XBKMu/tZZlYDlLj7HjObG3dgY0lXqpeKsqGPYABOntvM3U+/Qaqnl9IhBgaIiIxluezh/g3A3TvcfU8ouzO+kMaeXI9gAE6d18yeZIrn39BlY0Tk8Jb1CMbMFgFHAw1m9qcZb9UDlXEHNpbkciZ/2ruOmADAo+u2c+zMxhijEhEprsH2ikcCHwQagQ9lPE4A/iKXys3sPDNba2ZtZnb5AO9XmNnt4f3HzWxOxntXhPK1ZnbuUHVa5Coze8nM1pjZ53OJsRCS3b1ZbzbW36S6ShZMquUPr2yPOSoRkeLKegTj7ncBd5nZqe7+6HArNrMEcA1wNrARWGVmK9z9hYzZLgF2uvt8M1sGXA181MwWE51rczQwDfiNmS0My2Sr85PATGCRu/ea2aThxpyvXM7kz3TqvGbufGIj3T29lKkfRkQOU7ns3Z40s8vM7PtmdkP6kcNyJwNt7r7O3buA24Cl/eZZCtwcpu8Ellh0zfulwG3unnT3V4G2UN9gdf4VcKW79wK4+9YcYjxkvb1OV0/ufTAApx7RzL6uHp7WcGUROYzlsle8BZgCnAs8CMwA9gy6RGQ6sCHj9cZQNuA87p4C2oHmQZYdrM55REc/q83sP8xswUBBmdmlYZ7V27Zty+FjDK4rXH4/lzP5006bN5FEifHA2kNfv4jIaJXLXnG+u/8t0OHuNwMfAE6JN6y8VACd7t5KdHmbAY+y3P1ad29199aWlpZDXmkyFRLMMJrIGqrLOHF2E799cUQOskREiiKXBNMdnneZ2TuABiCX/o1NRH0iaTNC2YDzmFlpqHv7IMsOVudG4Odh+hfAMTnEeMiSqR6AYTWRASxZNIk1m3ezuX1/HGGJiBRdLnvFa82sCfgqsAJ4gagzfiirgAVmNtfMyok67Vf0m2cFsDxMXwDc7+4eypeFUWZzgQXAyiHq/CVwZph+D/BSDjEesmR3+ghmeAnmfYuiHH2/jmJE5DA15Jn87v6jMPkQcASAmc3KYbmUmX0OuBdIADe4+/NmdiWw2t1XANcDt5hZG7CDcJXmMN8dRMksBVzm7j1h3W+rM6zy74FbzexvgL3An+eyAQ5VXxNZjmfyp82fVMuMpip+9+JWPn7K7DhCExEpqkETjJmdStSJ/pC7bzWzY4DLgTM4uKlqQO5+D3BPv7KvZUx3AhdmWfYq4Kpc6gzlu4j6h0ZUvk1kZsZZR03mpytfpyOZoqYil6v2iIiMHVn3imb2TaKO8j8DfmVm/xf4T+BxoiYr4cARTK4nWmZ6/zunkkz18ps1WwodlohI0Q32s/kDwPHu3hn6YDYA73D39SMS2RiRbx8MQOvsJibXV3D3M5tZelz/EdwiImPbYHvFztCEhbvvBF5Wcnm7A01kw+uDASgpMd7/zqk8uHYbezq7h15ARGQMGSzBHGFmK9IPYG6/10LmeTD5XfLlg8dMo6unl/teUDOZiBxeBmsi639Zl3+MM5CxqiskmMphnMmf6YRZjUxvrOKXT73Bn54wo5ChiYgU1WAXu3xwJAMZq/I5kz+TmfFnJ0zne79rY9Ou/UxvrCpkeCIiRaNL+R6ifIcpZ7qwNRrxfefqjQWJSURkNFCCOUQHRpHldwQDMHNCNX8ybyJ3rN5Ab68XKjQRkaJSgjlEh3IeTKaPnjSTTbv288grbxUiLBGRohvy9HEz+3eg/8/qdmA18MP0UObxKt1EdqgJ5pyjJ9NUXcaPH32NMxYc+lWeRUSKLZe94jqia3tdFx67ie4HszC8HteSqV7KEkaixA6pnorSBB8/ZTa/WbOF9W91FCg6EZHiySXBnObuH3P3fw+PTwAnuftlwAkxxzfqJbuHd7vkwVx86mxKS4yb/rC+IPWJiBRTLgmmNvPqyWG6NrzsiiWqMaSrp+eQRpBlmlRfyYePnc4dqzfQvl9n9ovI2JbLnvG/Ab83s9+Z2QPAw8B/N7Ma4OY4gxsLoiOYwo2VuOT0uezr6uEnj71WsDpFRIohl/vB3BPub78oFK3N6Nj/f3EFNlYkU73DvhfMYBZPq+e9R7bwo4fXsfy0OdTqMv4iMkbl+tP7ROBo4Fjg/zOzi+MLaWxJpgrXRJb2xbMWsnNfNzerL0ZExrAh94xmdgvwLeB04KTwaI05rjEjmSpsExnAcTMbOfPIFq57eJ2usiwiY1Yu7S+twGJ31ynmA0h29x7yOTAD+eJZC1l6zSPc+Mh6Pr9E93cTkbEnlz3jc8CUuAMZq6ImssL1waQdO7OR846ewg8efIUtu8f1uawiMkblkmAmAi+Y2b26H8zbxdFElnbF+xeR6nG+ee/aWOoXEYlTLk1kX487iLEsGkUWT4KZ3VzDp/5kDtc+vI7lp87hnTMaYlmPiEgchtwzuvuDAz1GIrixoCtVuDP5B3LZ++Yzobqcr614TldaFpExJWuCMbPfh+c9ZrY747HHzHaPXIijWxzDlDPVV5bxlQ8cxZOv7+Inj+vkSxEZO7LuGd399PBc5+71GY86d68fuRBHtzj7YNI+cvx0zlgwkav/40Xe2LU/1nWJiBRKTntGM0uY2TQzm5V+xB3YWJHsLuyZ/AMxM77xkXfS6/DVXz6HRoyLyFiQy4mWfw1sAe4DfhUed8cc15jg7iRTPZQn4r9v28wJ1Xz5vCO5/8Wt3Pr467GvT0TkUOUyiuwLwJHuvj3uYMaaVK/T68TeRJa2/NQ5PLB2G//n7hc4Ze4EFkyuG5H1iojkI5c94waiO1hKP+nbJcc1TLm/khLjWxceS21FKX/90yfp7O4ZkfWKiOQj1ztaPmBmV5jZl9KPuAMbC5JhBx/nMOX+Wuoq+NaFx/Lim3v42l3qjxGR0SuXBPM6Uf9LOVCX8Rj3unrCEcwINZGlnbloEn/9vvncsXojt+i+MSIySg3aB2NmCWChu398hOIZU5LdI9tElulvzlrIC2/s5sp/f4GFk+t41xHNIx6DiMhgBt0zunsPMNvMyvOp3MzOM7O1ZtZmZpcP8H6Fmd0e3n/czOZkvHdFKF9rZucOo87vmtnefOIdrr4+mBFsIksrKTG+s+w4ZjVX81c/eYJ120bkI4uI5CzXPphHzOxvh9MHE45+rgHOBxYDF5nZ4n6zXQLsdPf5wHeAq8Oyi4FlRDc5Ow/4fjgXZ9A6zawVaMrhMxVEMpXugxn5IxiIzvK/YflJlJhx8Q0r2aqrLovIKJLLnvEVovNeShheH8zJQJu7r3P3LuA2YGm/eZYCN4fpO4ElZmah/DZ3T7r7q0BbqC9rnSH5fBP4cg6xFUQxj2DS5kys4cZPncSOji6W37iK3bpBmYiMEkOeB+Puf5dn3dOJhjinbQROyTaPu6fMrB1oDuWP9Vt2epjOVufngBXuvjnKUQMzs0uBSwFmzTq0CxKk+2DiuOHYcBwzo5EffOJEPn3TKj514ypu+tRJ1FWWFTUmEZFczuRvMbNvmtk9ZnZ/+jESweXKzKYBFwLfG2ped7/W3VvdvbWlpeWQ1lvsJrJM717YwvcuOp6nN+xi+Q0rdatlESm6XPaMtwIvAnOBvwPWA6tyWG4TMDPj9YxQNuA8ZlYKNADbB1k2W/nxwHygzczWA9Vm1pZDjIdkpE+0HMr575zKP3/seJ7Z2M7FN6xUc5mIFFUue8Zmd78e6A73gvk08L4cllsFLDCzuWEU2jKg/50wVwDLw/QFwP0enTm4AlgWRpnNBRYAK7PV6e6/cvcp7j7H3ecA+8LAgVgdOIIpXh9Mf+e9Yyr//LETeHZjOx/94WO63bKIFE0uCSb9M3izmX3AzI4HJgy1kLuniPpF7gXWAHe4+/NmdqWZfTjMdj3QHI42vgRcHpZ9HrgDeAH4NXCZu/dkqzPHz1pwXaninGg5lPPeMYXrP3kSr23v4E+//wfatu4pdkgiMg7ZUJcaMbMPAg8TNU19D6gH/s7d+x+NjDmtra2+evXqvJf/8aPr+dpdz/PEV8+iubaigJEVxrMb2/nUTavo7unluotbOXnukL8LRESGZGZPuHvrUPPlcsvku9293d2fc/cz3f3EwyG5FMKBM/lHTxNZpnfOaOAXnz2N5tpyPnbdY9zy6Hpdu0xERkwuo8gWmtlvzey58PoYM/tq/KGNfqNpFFk2MydU84vP/gnvWdjC3971PF++8xldhVlERkQue8brgCsIfTHu/gxR5/q4l0z1YgalJdnPuxkNGqrKuO7iVj6/ZAE/e2IjF/zgD7q0jIjELpcEU+3uK/uVpeIIZqxJpnqpKC1hsBM7R4uSEuNLZy/kuotb2bhzPx/47u+5beXrajITkdjkkmDeMrN5gAOY2QXA5lijGiOS3T2jaohyLs5ePJlff+HdHD+rkct//iyfvfWP7OzoKnZYInIYyiXBXAb8EFhkZpuALwKfiTOosSJ9BDPWTGmo5CeXnMIV5y/iN2u2cNa3H+SupzbpaEZECiqXUWTr3P0soAVY5O6nAx+JPbIxoCvVO2rO4h+ukhLjL98zj3//69OZMaGaL9z2FJfcvJpNu/YXOzQROUzkvHd09w53T5+xp1smkz6CGVtNZP0tmlLPz//qNP72g4t59JXtnPPtB/mXB17pGyEnIpKvfH9+j/5e7RGQTPWMySay/hIlxiWnz+U//+bdnDpvIlf/+kXO/vZD/Pq5N9VsJiJ5y3fvqL0OY7cPJpuZE6r50fJWbrnkZCpKS/jMT57g4z96nKc37Cp2aCIyBmXdO5rZHjPbPcBjDzBtBGMctZLdY7+JbCBnLGjhP75wBlcuPZo1m3ez9JpHuPTHq3nxzd3FDk1ExpCsNxxz91zuWjmuJVM9NFaXFzuMWJQmSrj41Dl85Pjp3PjIeq57aB3n/9PDfOiYaXx+yXzmT9LXQ0QGN+QdLSW7w62JbCB1lWV8fskCLj51Ntc+tI4bH1nPiqff4OzFk/nMe+Zx4uymYocoIqOUEswhSKZ6R+2FLgutsbqcL5+3iEtOn8vNj77Gjx9dz30vbOHkORP4zHuP4L0LJ1Eyyi+ZIyIjSwnmEHSNgyOY/pprK/jS2Qv5y3cfwW2rNnD9w+v49E2rmd1czSdOmc2FrTMO22ZDERme8bV3LLDDZZhyPmoqSrnk9Lk8+OUz+d5FxzO5rpKr7lnDKd/4Lf/jZ0/z7Mb2YocoIkWmI5hDcLiOIhuOskQJHzp2Gh86dhprNu/mlsde45dPbuJnT2zk6Gn1/NkJM1h63LRReUM2EYnX+Pz5XSDJMXypmDgcNbWeb3zknTz2v5bwdx8+mhIzrrz7BU75xm/5ix+v5tfPvdl3m2kROfzpCCZPvb1OV8/464PJRX1lGctPm8Py0+aw9s09/NsfN/KLJzdx3wtbaKwu49zFU3j/MVM5bV4zZQltP5HDlRJMnrp6ol/i5UowgzpySh3/6/1H8eVzj+Thl9/il09t4lfPbub21RtorC7jnMWT+cAx05RsRA5DSjB5SnZHCWa898HkqjRRwpmLJnHmokl0dvfw0EvbuOfZzdzz7JvcsXojDVVlvPfIFpYcNZn3LGihobqs2CGLyCFSgslT+mrDaiIbvsqyBOccPYVzjp7Sl2x+/fybPLB2G3c99QaJEqN1dhNLjprE+xZNZl5LzZi4a6iIHEwJJk/JVPoIRgnmUGQmm55e56kNu7j/xS3c/+I2vnHPi3zjnheZOaGK0+e38Cfzmzlt3kQm1Og8G5GxQAkmT30JZpycyT8SEiXGibObOHF2E//j3EW8sWs/97+4lQfWbuPup9/gpytfB2Dx1HpOXzCR0+Y1c/LcCVSX62ssMhrpPzNPaiKL37TGKj7xrtl84l2zSfX08uymdh5pe4vft73FTY+s59qH1lGWMI6Z0UjrnCZOmj2BE2c30aQjHJFRQQkmT2oiG1mliRKOn9XE8bOa+Nz7FrC/q4dV63fwyCtvserVHdzw+1f54YPrAFgwqZbWORM4aU4TJ82ZwIymKvXhiBSBEkyeNIqsuKrKE7x7YQvvXtgCQGd3D09v2MXq13ayav0O7n7mQJPaxNpyjpnRyDEzGjg2POvKAiLxU4LJU7qJTOfBjA6VZQlOOaKZU45oBqCn13lpyx5Wr9/BUxvaeWbjLn63divpO0DPaKrqSzbHzGjkHdPrqavU0GiRQlKCyZOayEa3RIlx1NR6jppaz389NSrbm0zx7MYo2TyzsZ2nN+7iV89u7ltm5oQqjppS37fc4qn1zJyg5jWRfCnB5CmdYCp1LbIxo7ailFPnNXPqvOa+su17kzyzqZ3nN7WzZvMe1mzezX1rtvQd6dRWlLJoSl1f0jlySh3zJ9XSUKWjHZGhKMHkKdmdHkWmPpixrLm2gjOPnMSZR07qK9vXlWLtm3v6Es6azbv5xZObuOWx1/rmmVRXwfxJtSyYVMv8SbXMn1THgsm1NNeU64hHJIg1wZjZecA/AQngR+7+9/3erwB+DJwIbAc+6u7rw3tXAJcAPcDn3f3eweo0s1uBVqAbWAn8pbt3x/XZ0tciUxPZ4ae6vLRvxFpab6+zced+Xtqyh7Zte3l5y17atu3lzic20tHV0zdfY3VZX9KZ11LL7OYa5k6sZuaEav0YkXEntgRjZgngGuBsYCOwysxWuPsLGbNdAux09/lmtgy4GviomS0GlgFHA9OA35jZwrBMtjpvBT4R5vlX4M+Bf4nr82kU2fhSUmLMaq5mVnM1ZzG5r9zdeXN3J21bDySdti17uff5Lezo2NA3nxlMa6hi7sQa5kysZk5zTfSYWMPMCVX6HslhKc4jmJOBNndfB2BmtwFLgcwEsxT4epi+E/hni9oXlgK3uXsSeNXM2kJ9ZKvT3e9JV2pmK4EZcX0wyDyTX0cw45mZMbWhiqkNVZyxoOWg93bt6+LVtzp4bfs+Xn2rg/XbO1j/VgcrnnqD3Z2pvvlKDKY2VDFzQhUzm6qZ0VTNjKYqZk6InifXV5IoUbObjD1xJpjpwIaM1xuBU7LN4+4pM2sHmkP5Y/2WnR6mB63TzMqA/wp8YaCgzOxS4FKAWbNm5f5p+ukbpqxLzEsWjdXlHD+r/KCmtrSdHV28ur2D17Z38Opb+3htewcbd+7noZe3sWV38qB5S0uMaY1RAprReHDymdFUTUtdhRKQjEqHYyf/94GH3P3hgd5092uBawFaW1s935UkU72UJ0oo0T+25KGpppymmnJOGCD5dHb38Mau/WzcGT027NwXpvdx/9qtbNtzcAJKlBiT6yqY0lDJ1Iaq8FyZ8VzFpLoK3W9HRlycCWYTMDPj9YxQNtA8G82sFGgg6uwfbNmsdZrZ/wZagL8sQPyDSnb36iRLiUVlWYIjWmo5oqV2wPc7u3v6Es7Gnft5s72Tze2dvLl7P2ve3M39L25lf3fPQcuYQUttRUbiOZCIWuoqmFRXQUtdJfWVpRoFJwUTZ4JZBSwws7lESWAZ8LF+86wAlgOPAhcA97u7m9kK4F/N7NtEnfwLiEaGWbY6zezPgXOBJe4e+43fk6kejSCToqgsS4Sh0QMnIHdnd2cqJJ6MBNTeyRvt+1m3rYM/tG1nTzL1tmUrSktoqauIHrUVTKqvoKW2MjxH5ZPqK2iuqdAPLBlSbAkm9Kl8DriXaEjxDe7+vJldCax29xXA9cAtoRN/B1HCIMx3B9GAgBRwmbv3AAxUZ1jlD4DXgEfDL7Cfu/uVcX2+ZKpXCUZGJTOjoaqMhqoyjpxSl3W+PZ3dbNndydY9Sbb1e2zdk+S17ftYtX4HO/cNPNq/qbqMSXWVfQlpQk05zbXlNNeUM6EmvK4pZ0JtOXUVOjIaj2Ltgwkju+7pV/a1jOlO4MIsy14FXJVLnaF8RPuTkqle3QtGxrS6yjLqKsuYPyl7EgLoSvWyvSPJ1t0hAe0N03s7+5LR+vUd7OjoYl9Xz4B1lCdKaKopY0JNRUhA0WNibUYyqi3vS0r1lWXq3zwMHI6d/COiS01kMk6Ul5b0DcUeSmd3D9s7utixt4u3OpLs2NvFjo6uqKwj2Te9Yec+tu/tYu8AzXQQDVxoqCqjsbqMxqoyGqvLDzxXR+UNVWU0pV9XldNQXaY+pFFGCSZPaiITebvKsgTTG6uY3jh0MoKoL3NHRxfbQyLKTEa79nWza3837fuipry1b+6hfX931qQEGYmpqoyG6pCAwnRjVTkNVaXUV0VHbvWV6enouba8VEdNBaYEk6dkd6/OvhY5RBWliZyPjtK6e3rZta+b9v1d7NrXzc593eza10X7/u7wumvYiQmikXa1FaXUV5YdSDyVZdRXhefMhPS2eaJpDQU/mBJMnpKpHmoqtPlERlpZ4sBIt+HoSvWyp7Ob3Z2p6Hl/it2d3X3T6fd27w/Pnd1s2rWfNZu72d0ZJSgf4sy5qrIE9VWlfUdIdZVl1FaUUlORoKailLqKUmrCozY8+qYro/lqK0qpKkscFk192kPmKZnqpalav1ZExory0hKaayvyvptpb6+ztytKQHs6M547uw+aPlCWYte+Ljbu3EdHsoeOZIq9XUMnKYguH1STkYAOJKfEgMnp4LLEgeXKS6muSBTtyEoJJk/RKDIlGJHxoqTEQlNZ/vcCcnf2dYVkk/FIJ6A9yRQd4bE3mWJvZ4qOrhR7w/vb9iSj+bui91K9uV2MpDxRQlV5gpryBNUVpdSUJ/jeRScwq7k678+SCyWYPEUnWqoPRkRyZ2Z9RySThp59UO5OMtUbElJPRrKKnvd1ReX7ulJ0dPWwLxmeQ/lI/EBWgslT1MmvIxgRKQ4zo7IsQWVZguaBL+pQdNpD5qmrRwlGRGQw2kPmKdmtM/lFRAajBJOHqO1TZ/KLiAxGe8g8pHqdXkcJRkRkENpD5qHvdskaRSYikpUSTB6S4WZOuh+GiEh22kPm4cARjDafiEg22kPmoS/B6Ex+EZGstIfMQzIVNZGpD0ZEJDslmDx0qYlMRGRI2kPmQaPIRESGpgSTh2S3+mBERIaiPWQeDvTBaPOJiGSjPWQe0k1kOg9GRCQ77SHzoFFkIiJDU4LJQ18fjI5gRESy0h4yDzqTX0RkaNpD5qHvPBjdD0ZEJCslmDxoFJmIyNC0h8xDMtVLiUFpiRU7FBGRUUsJJg/JVC8VpQnMlGBERLJRgslDsrtHZ/GLiAxBe8k8JFO9lCe06UREBhPrXtLMzjOztWbWZmaXD/B+hZndHt5/3MzmZLx3RShfa2bnDlWnmc0NdbSFOsvj+lzJVK+OYEREhhDbXtLMEsA1wPnAYuAiM1vcb7ZLgJ3uPh/4DnB1WHYxsAw4GjgP+L6ZJYao82rgO6GunaHuWCRTPTqLX0RkCHH+DD8ZaHP3de7eBdwGLO03z1Lg5jB9J7DEop7zpcBt7p5091eBtlDfgHWGZd4X6iDU+V/i+mDJ7l4NURYRGUJpjHVPBzZkvN4InJJtHndPmVk70BzKH+u37PQwPVCdzcAud08NMP9BzOxS4FKAWbNmDe8TBSfMbmJvMjX0jCIi41icCWZUcvdrgWsBWltbPZ86LjtzfkFjEhE5HMXZzrMJmJnxekYoG3AeMysFGoDtgyybrXw70BjqyLYuEREZQXEmmFXAgjC6q5yo035Fv3lWAMvD9AXA/e7uoXxZGGU2F1gArMxWZ1jmd6EOQp13xfjZRERkCLE1kYU+lc8B9wIJ4AZ3f97MrgRWu/sK4HrgFjNrA3YQJQzCfHcALwAp4DJ37wEYqM6wyv8J3GZm/xd4MtQtIiJFYtGP//GptbXVV69eXewwRETGFDN7wt1bh5pPY21FRCQWSjAiIhILJRgREYmFEoyIiMRiXHfym9k24LU8F58IvFXAcApFcQ2P4hoexTU8h2tcs929ZaiZxnWCORRmtjqXURQjTXENj+IaHsU1POM9LjWRiYhILJRgREQkFkow+bu22AFkobiGR3ENj+IannEdl/pgREQkFjqCERGRWCjBiIhIPNxdj2E+gPOAtUS3cr48hvpnEt1+4AXgeeALofzrRPe5eSo83p+xzBUhnrXAuUPFCswFHg/ltwPlOca2Hng2rH91KJsA3Ae8HJ6bQrkB3w3reAY4IaOe5WH+l4HlGeUnhvrbwrKWQ0xHZmyTp4DdwBeLtb2AG4CtwHMZZbFvo2zrGCSmbwIvhvX+AmgM5XOA/Rnb7Qf5rnuwzzdEbLH/7YCK8LotvD8nh7huz4hpPfDUSG4zsu8bivr9yvq/UOid4+H+ILpNwCvAEUA58DSwuMDrmJr+IgB1wEvA4vBP998HmH9xiKMi/DO9EuLMGitwB7AsTP8A+KscY1sPTOxX9g+Ef2jgcuDqMP1+4D/Cl/xdwOMZX9R14bkpTKf/IVaGeS0se34ef583gdnF2l7Au4ETOHjHFPs2yraOQWI6BygN01dnxDQnc75+n21Y6872+XLYXrH/7YDPEhIB0a1Cbh8qrn7v/yPwtZHcZmTfNxT1+5X1f2G4O7/x/gBOBe7NeH0FcEXM67wLOHuQf7qDYiC6X86p2WINX5y3OLBzOWi+IWJZz9sTzFpgapieCqwN0z8ELuo/H3AR8MOM8h+GsqnAixnlB82XY3znAI+E6aJtL/rtcEZiG2VbR7aY+r33EeDWwebLZ93ZPl8O2yv2v1162TBdGuazweLKKDdgA7CgWNssvJfeNxT9+zXQQ30wwzed6IuVtjGUxcLM5gDHEx3CA3zOzJ4xsxvMrGmImLKVNwO73D3VrzwXDvynmT1hZpeGssnuvjlMvwlMzjOu6WG6f/lwLAN+mvG62NsrbSS2UbZ15OLTRL9W0+aa2ZNm9qCZnZER63DXfSj/L3H/7fqWCe+3h/lzcQawxd1fzigb0W3Wb98wKr9fSjCjmJnVAv8GfNHddwP/AswDjgM2Ex2ij7TT3f0E4HzgMjN7d+abHv288SLERbiN9oeBn4Wi0bC93mYkttFw1mFmXyG6c+ytoWgzMMvdjwe+BPyrmdXHse5BjMq/XYaLOPiHzIhuswH2DXnXlY9c16EEM3ybiDra0maEsoIyszKiL9Ct7v5zAHff4u497t4LXAecPERM2cq3A41mVtqvfEjuvik8byXqGD4Z2GJmU0PcU4k6RvOJa1OY7l+eq/OBP7r7lhBj0bdXhpHYRtnWkZWZfRL4IPDxsNPA3ZPuvj1MP0HUt7Ewz3Xn9f8yQn+7vmXC+w1h/kGFef+UqMM/He+IbbOB9g151DUi3y8lmOFbBSwws7nhF/MyYEUhV2BmBlwPrHH3b2eUT82Y7SPAc2F6BbDMzCrMbC6wgKijbsBYw47kd8AFYfnlRG25Q8VVY2Z16Wmi/o7nwvqXD1DXCuBii7wLaA+H2PcC55hZU2j6OIeoXXwzsNvM3hW2wcW5xJXhoF+Vxd5e/YzENsq2jgGZ2XnAl4EPu/u+jPIWM0uE6SPC9lmX57qzfb5BjdDfLjPmC4D700l2CGcR9VP0NSWN1DbLtm/Io67Yv1+AOvnzeRCNzHiJ6FfKV2Ko/3Siw89nyBimCdxCNHzwmfDHnpqxzFdCPGvJGHmVLVai0TYriYYi/gyoyCGuI4hG5zxNNETyK6G8Gfgt0fDF3wATQrkB14R1Pwu0ZtT16bDuNuBTGeWtRDuTV4B/JodhymG5GqJfnw0ZZUXZXkRJbjPQTdSGfclIbKNs6xgkpjaidvj0dyw9ourPwt/3KeCPwIfyXfdgn2+I2GL/2wGV4XVbeP+IoeIK5TcBn+k374hsM7LvG4r6/cr20KViREQkFmoiExGRWCjBiIhILJRgREQkFkowIiISCyUYERGJhRKMyDCZWbOZPRUeb5rZpozX5UMs22pm3x3m+j5tZs9adNmU58xsaSj/pJlNO5TPIhInDVMWOQRm9nVgr7t/K6Os1A9c++pQ658BPEh0Bd32cImQFnd/1cweILog5OpCrEuk0HQEI1IAZnaTmf3AzB4H/sHMTjazRy26+OEfzOzIMN97zezuMP11iy7k+ICZrTOzzw9Q9SRgD7AXwN33huRyAdEJcbeGI6cqMzvRogstPmFm99qBy3o8YGb/FOZ7zsxOHmA9IgWnBCNSODOA09z9S0Q38jrDo4sffg34RpZlFgHnEl1r639bdJ2pTE8DW4BXzexGM/sQgLvfCawmuobYcUQXq/wecIG7n0h0s6yrMuqpDvN9NrwnErvSoWcRkRz9zN17wnQDcLOZLSC6tEf/xJH2K3dPAkkz20p0CfS+a1y5e0+4ZthJwBLgO2Z2ort/vV89RwLvAO6LLiFFgugyJ2k/DfU9ZGb1Ztbo7rvy/6giQ1OCESmcjozp/wP8zt0/YtF9Ox7IskwyY7qHAf4nPeooXQmsNLP7gBuJbsiVyYDn3f3ULOvp39mqzleJnZrIROLRwIHLnH8y30rMbJqZnZBRdBzwWpjeQ3TbXIgu/NhiZqeG5crM7OiM5T4ayk8nuqJue74xieRKRzAi8fgHoiayrwK/OoR6yoBvheHIncA24DPhvZuAH5jZfqJbAV8AfNfMGoj+t/8f0RV+ATrN7MlQ36cPIR6RnGmYsshhTsOZpVjURCYiIrHQEYyIiMRCRzAiIhILJRgREYmFEoyIiMRCCUZERGKhBCMiIrH4/wH4yodZxlsmGAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 시각화해보기\n",
    "sample_learning_rate = CustomSchedule(d_model=256)\n",
    "\n",
    "plt.plot(sample_learning_rate(tf.range(200000, dtype=tf.float32)))\n",
    "plt.ylabel(\"Learning Rate\")\n",
    "plt.xlabel(\"Train Step\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "female-relationship",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "learning_rate = CustomSchedule(D_MODEL)\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(\n",
    "      learning_rate, beta_1=0.9, beta_2=0.98, epsilon=1e-9)\n",
    "\n",
    "def accuracy(y_true, y_pred):\n",
    "    y_true = tf.reshape(y_true, shape=(-1, MAXLEN - 1))\n",
    "    return tf.keras.metrics.sparse_categorical_accuracy(y_true, y_pred)\n",
    "\n",
    "model.compile(optimizer=optimizer, loss=loss_function, metrics=[accuracy])\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "identical-venice",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/40\n",
      "185/185 [==============================] - 51s 235ms/step - loss: 2.1678 - accuracy: 0.0285\n",
      "Epoch 2/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 1.5722 - accuracy: 0.0771\n",
      "Epoch 3/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 1.2707 - accuracy: 0.0781\n",
      "Epoch 4/40\n",
      "185/185 [==============================] - 43s 234ms/step - loss: 1.1603 - accuracy: 0.0830\n",
      "Epoch 5/40\n",
      "185/185 [==============================] - 43s 232ms/step - loss: 1.0812 - accuracy: 0.0872\n",
      "Epoch 6/40\n",
      "185/185 [==============================] - 43s 232ms/step - loss: 0.9858 - accuracy: 0.0934\n",
      "Epoch 7/40\n",
      "185/185 [==============================] - 43s 231ms/step - loss: 0.8840 - accuracy: 0.1035\n",
      "Epoch 8/40\n",
      "185/185 [==============================] - 43s 234ms/step - loss: 0.7691 - accuracy: 0.1187\n",
      "Epoch 9/40\n",
      "185/185 [==============================] - 43s 235ms/step - loss: 0.6300 - accuracy: 0.1346\n",
      "Epoch 10/40\n",
      "185/185 [==============================] - 43s 232ms/step - loss: 0.5021 - accuracy: 0.1523\n",
      "Epoch 11/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.3779 - accuracy: 0.1707\n",
      "Epoch 12/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.2634 - accuracy: 0.1862\n",
      "Epoch 13/40\n",
      "185/185 [==============================] - 43s 231ms/step - loss: 0.1750 - accuracy: 0.2023\n",
      "Epoch 14/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.1103 - accuracy: 0.2122\n",
      "Epoch 15/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0674 - accuracy: 0.2201\n",
      "Epoch 16/40\n",
      "185/185 [==============================] - 44s 235ms/step - loss: 0.0457 - accuracy: 0.2227\n",
      "Epoch 17/40\n",
      "185/185 [==============================] - 43s 235ms/step - loss: 0.0347 - accuracy: 0.2233\n",
      "Epoch 18/40\n",
      "185/185 [==============================] - 43s 232ms/step - loss: 0.0303 - accuracy: 0.2229\n",
      "Epoch 19/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0278 - accuracy: 0.2233\n",
      "Epoch 20/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0281 - accuracy: 0.2228\n",
      "Epoch 21/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0267 - accuracy: 0.2242\n",
      "Epoch 22/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0250 - accuracy: 0.2235\n",
      "Epoch 23/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0246 - accuracy: 0.2235\n",
      "Epoch 24/40\n",
      "185/185 [==============================] - 43s 235ms/step - loss: 0.0226 - accuracy: 0.2239\n",
      "Epoch 25/40\n",
      "185/185 [==============================] - 43s 232ms/step - loss: 0.0198 - accuracy: 0.2244\n",
      "Epoch 26/40\n",
      "185/185 [==============================] - 43s 234ms/step - loss: 0.0176 - accuracy: 0.2254\n",
      "Epoch 27/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0160 - accuracy: 0.2250\n",
      "Epoch 28/40\n",
      "185/185 [==============================] - 44s 235ms/step - loss: 0.0144 - accuracy: 0.2257\n",
      "Epoch 29/40\n",
      "185/185 [==============================] - 43s 235ms/step - loss: 0.0136 - accuracy: 0.2243\n",
      "Epoch 30/40\n",
      "185/185 [==============================] - 43s 234ms/step - loss: 0.0117 - accuracy: 0.2257\n",
      "Epoch 31/40\n",
      "185/185 [==============================] - 43s 234ms/step - loss: 0.0123 - accuracy: 0.2265\n",
      "Epoch 32/40\n",
      "185/185 [==============================] - 43s 234ms/step - loss: 0.0105 - accuracy: 0.2258\n",
      "Epoch 33/40\n",
      "185/185 [==============================] - 43s 234ms/step - loss: 0.0105 - accuracy: 0.2267\n",
      "Epoch 34/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0085 - accuracy: 0.2276\n",
      "Epoch 35/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0099 - accuracy: 0.2272\n",
      "Epoch 36/40\n",
      "185/185 [==============================] - 44s 235ms/step - loss: 0.0082 - accuracy: 0.2267\n",
      "Epoch 37/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0077 - accuracy: 0.2279\n",
      "Epoch 38/40\n",
      "185/185 [==============================] - 43s 231ms/step - loss: 0.0080 - accuracy: 0.2276\n",
      "Epoch 39/40\n",
      "185/185 [==============================] - 43s 232ms/step - loss: 0.0068 - accuracy: 0.2283\n",
      "Epoch 40/40\n",
      "185/185 [==============================] - 43s 233ms/step - loss: 0.0071 - accuracy: 0.2285\n"
     ]
    }
   ],
   "source": [
    "EPOCHS = 40\n",
    "history = model.fit(dataset, epochs=EPOCHS, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "burning-shakespeare",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEWCAYAAACKSkfIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAA7PklEQVR4nO3deXxc5Xno8d8jjUaSJcuyLdnYkrwI2UZmM2AZCIlZi41KgDbgmJAAAUrTOLe0NLmFCyW5JOl1m/Ym7XWaQFNCEogdQkPiJGBwwJAFHG8YL/Ii75IsIxlrtdaZee4f58geZMnaRnNmeb4f5qOZs8x5xhw9euc973kfUVWMMcYkhxSvAzDGGBM9lvSNMSaJWNI3xpgkYknfGGOSiCV9Y4xJIpb0jTEmiVjSj1EickhEbvA6DmNMYrGkb4wxScSSvjEmbojD8tYI2D9ejBORdBH5logcdR/fEpF0d12eiPxKRBpF5ISI/K7nF0JE/l5EakSkRUT2iMj13n4Sk0hE5BER2e+eXxUi8mdh6/5CRHaFrbvUXV4kIj8TkXoR+UBEVrjLvyIiz4XtP0NEVER87us3ReTrIvIHoA0oFpHPhh3jgIj8Za/4bhWRrSLS7Ma5WETuEJHNvbZ7WER+MXr/UrHH53UAZkCPAVcA8wAFfgE8DvwD8HdANZDvbnsFoCIyB/gCUKaqR0VkBpAa3bBNgtsPfAw4BtwBPCciJcBHga8AtwGbgHOBbhFJBX4FvAF8BggC84dwvM8ANwF7AAHmADcDB4CFwCsislFVt4jIAuCHwO3A68AUYCxwEHhKREpVdVfY+35tGJ8/bllLP/bdBTypqnWqWg/8b5wTFaAb54Serqrdqvo7dSZTCgLpwFwRSVPVQ6q635PoTUJS1Z+q6lFVDanqT4BKYAHwAPDPqrpRHftU9bC7birwJVU9qaodqvr7IRzyWVXdqaoB91z/tarud4/xFvAazh8hgPuBZ1R1rRtfjaruVtVO4CfApwFE5HxgBs4fo6RhST/2TQUOh70+7C4D+AawD3jN/Yr7CICq7gP+BqfFVSciq0RkKsZEiIjc7XafNIpII3ABkAcU4XwL6K0IOKyqgWEesqrX8W8SkfVut2YjUO4ev+dY/TVyfgB8SkQEp/H0gvvHIGlY0o99R4HpYa+nuctQ1RZV/TtVLQZuAR7u6btX1R+r6kfdfRX4p+iGbRKViEwH/hOnC3GiquYCO3C6XapwunR6qwKm9fTT93ISGBP2+pw+tjk1HbB7Teu/gX8BJrvHf9k9fs+x+ooBVV0PdOF8K/gU8KO+tktklvRj30rgcRHJF5E84AngOQARuVlEStxWSxNOt05IROaIyHXuL0cH0A6EPIrfJJ4snCRcDyAin8Vp6QN8D/iiiFzmjrQpcf9IbABqgeUikiUiGSJylbvPVmChiEwTkXHAowMc34/TfVkPBETkJuDGsPX/BXxWRK4XkRQRKRCR88LW/xBYAXQPsYspIVjSj31fw7kgtg3YDmzh9IWnWcBvgFbgHeA/VHUdzi/EcuA4zoW2SQz8i2TMoKhqBfCvOOfc+8CFwB/cdT8Fvg78GGgBfg5MUNUg8HGgBDiCMwDhk+4+a3H62rcBmxmgj11VW4C/Bl4AGnBa7KvD1m8APgt8E6cx9BYf/rb8I5w/Us+RhMSKqBhjkomIZAJ1wKWqWul1PNFmLX1jTLL5K2BjMiZ8sHH6xpgkIiKHcC743uZtJN6x7h1jjEki1r1jjDFJJCa7d/Ly8nTGjBleh2ES1ObNm4+rav7AW0aWnddmNA32vI7JpD9jxgw2bdrkdRgmQYnI4YG3ijw7r81oGux5bd07xhiTRCzpG2NMErGkb4wxScSSvjHGJBFL+sYYk0Qs6RtjTBKxpG+MMUlkwHH6IlKEM//0ZJw5tJ9W1X/rtY0A/4ZTvaYNuFdVt7jr7sGp6QrwNVX9wXACPfzBSV7cXM3SBdMoyM0czluYJLVmzRoeeughgsEgDzzwwBnrReRhnDJ/AZw52u9T1cMiMg/4DpCDU6vg625pQETkWeBqnKl7wTnnt47yRzEJoDMQ5HhrFyc7A5zsDNDWFSQYUlJESBEnybZ0BGjp6KalI0BnIEQwFCIQUoIh5a7Lp3POuIxhH38wN2cFgL9zCw6PBTaLyFp3Tu0eN+HM7T4LuBznF+VyEZkAfBmnALK6+65W1YahBlrb1MH/e2MfVxZPtKRvBi0YDLJs2TLWrl1LYWEhZWVlAL1/Y94F5qtqm4j8FfDPOHO9twF3q2qlW25ys4i8qqqN7n5fUtUXo/RRTAzo6A5S39JJfWun87Olk+Ot7qOli8b2LpraAzS3d9MdDJGTmUZOho/sjDSa2rqoaezgeOvwqzOmCNxQOnl0k76q1uJUvEFVW0RkF1AAhCf9W4EfukW514tIrohMAa4B1qrqCQARWQssxqkGNSTjMtMAaGrvHuquJolt2LCBkpISiouLAVi6dCnbtm3LDd/GLTzTYz1u4WxV3Ru2zVERqQPygcZRDttESXcwRHVDOwePt7K/7iT761s5cqKNzkCIQDBEd1Bp7w7S3O60uruCfRegGz8mjYnZ6Ywfk0ZBbgalU8aSlpJCS2c3ze0Bmtq6GDfGT+mUHKbmZjJpbDrZGT6y/D7G+FPxpQohhWBIESA7w0dORhrZ6T4y/amkpgipIqSkSJ/HH4ohTcMgIjOAS4A/9lpVwIcLF1e7y/pbPmSW9M1w1NTUUFRUdOp1YWEhOOX2+nM/8ErvhSKywN0vvOD210XkCeB14JFkK7AdD1SVIyfaeK+6idrGdj442cXx1k7qmjs5fOIkRxs7CIZOzzQ8McvPtIljnESc7iMtVchIS2VcZhpjM9LIyfSRl51O/th08t2fE7L8pKXGz+XRQSd9EcnGKUb8N6raHOlARORB4EGAadOmnbE+x036zR2W9M3oEJFP43RFXt1r+RScEnv3qGpPU+9RnFKUfuBp4O+BJ/t4z7Oe1yYyQiFlf30rhz5oo7qhjeqGdvbXt/JeVSMNbadzRrovhbzsdPLGpjOvaDy3zRvDtAljKM7P4tz8bHLHnK09kBgGlfRFJA0n4T+vqj/rY5MaoCjsdaG7rAaniyd8+Zt9HUNVn8b55WH+/PlnTPKf5X7FsZa+GYqCggKqqk5/2ayurgbo6r2diNwAPAZcHd5iF5Ec4NfAY6q6vme52+0J0Cki3we+2NfxBzqvzfC0dgZ4r6qRzYcb2Hy4gS1HGmjpCJxan5mWyvSJY7hx7jlcXJTLRYXjmJGXRZY/FWfcSfIazOgdwakuv0tV/28/m60GviAiq3Au5Dapaq2IvAr8o4iMd7e7kWEW6BYRxmWmWdI3Q1JWVkZlZSUHDx6koKCAVatWQa8+eRG5BHgKWKyqdWHL/cBLONerXuy1zxT3HO+pwrRjdD9JfNtX18LOo81UN7RT09hOfUsn4QWcevrOO7qDtHcFCamiAAp+XwrjMtMYP8bPGH8qu461sOdYMyEFEZg9aSwfv3gql04bz6xJ2RSOz2RClj/pk3t/BtPSvwr4DLBdRLa6y/4XMA1AVb8LvIwzXHMfzoiHz7rrTojIV4GN7n5P9lzUHY6cDB9N7YGBNzTG5fP5WLFiBYsWLSIYDHLfffexbdu2DhF5EtikqquBbwDZwE/dRHFEVW8BlgALgYkicq/7lj1DM58XkXyc0ntbgc9F9YPFgUAwxG92vc/3/3CIPx48/Ws/fkwak3MySHUvSqpCmi+FzLQUJmb5SR/nfKvH+Y/OQIimtm4OHG+ltSPAuZOyufG6WVw6fTzzinJPXe8zgxOT5RLnz5+vfc07fuuK35M7xs8P7lvgQVQmUYjIZlWdH+3j9ndeJ6K1Fe/zldU7qWlspyA3k7uvnM51502iYHwmY/wxWcYj7g32vI6rf/0c694xJqaFQsr/e2Mf3/zNXs6fmsOXPz6X60snn2rVG+/FXdKvaWj3OgxjTB9Odgb4uxfeY83OY/z5pQX8459dSEZaqtdhmV7iKumPy0yzIZvGxKBgSFn69Hp2Hm3iH26ey31XzbALqTEq7pJ+U3s3qmonlDExZG3FMbbXNPGvd1zMJy4r9DoccxbxcxsZkJORdmpolzEmdjzz+0MUjs/ktkuGdcO9iaK4Svo9Q7OabdimMTFje3UTGw6d4N6PzLALtnEgLpO+jeAxJnZ8/w8HyfKnsqSsaOCNjecs6Rtjhq2uuYNfbjvKHfOLyMmwm6TiQVwl/ZxM57pzsyV9Y2LCc+sPEwgp93xkhtehmEGKq6RvLX1jYkdHd5Dn/niE68+bxMy8LK/DMYNkSd8YMyy/fO8oJ052cd9VM70OxQxBXCX9sRk2p74xseLl7bVMmzCGK8+d6HUoZgjiKumnpghj033W0jfGYx3dQd7e/wHXnTfJbpSMM3GV9MEmXTMmFryz/wM6AyGuO2+S16GYIYrLpG+jd4zx1ro9dWSmpbJg5gSvQzFDFHdJf1ymz+7INcZDqsobu+u4qiTPZtGMQwMmfRF5RkTqRKTPcnAi8iUR2eo+dohIUEQmuOsOich2d11EqkdYyURjvLW/vpXqhnauPS/f61DMMAympf8ssLi/lar6DVWdp6rzcOrfvtWrJOK17vqIVCrKybCkb4yX3tjtlBG+do7158ejAZO+qv4WGGxd2zuBlSOKaAA2p74x3lq3u57zzhnL1NxMr0MxwxCxPn0RGYPzjeC/wxYr8JqIbBaRBwfY/0ER2SQim+rr6/vdblxmGm1dQbqDoYjEbYwZvOaObjYeOsG1NmonbkXyQu7HgT/06tr5qKpeCtwELBORhf3trKpPq+p8VZ2fn99/X+G4MXZXrhmaNWvWMGfOHEpKSli+fPkZ60XkYRGpEJFtIvK6iEwPW3ePiFS6j3vCll/mXq/aJyL/LkkyWP33lccJhNSGasaxSCb9pfTq2lHVGvdnHfASsGCkB+mZyc+GbZrBCAaDLFu2jFdeeYWKigpWrlwJkNFrs3eB+ap6EfAi8M8A7oCELwOX45y7XxaR8e4+3wH+ApjlPvq97pVI1u2uY1xmGpcU5XodihmmiCR9ERkHXA38ImxZloiM7XkO3Aj0OQJoKGz+HTMUGzZsoKSkhOLiYvx+P0uXLgXIDd9GVdepapv7cj3QU+9vEbBWVU+oagOwFlgsIlOAHFVdr6oK/BC4LQofx1OhkPLm3noWzs7Hlxp3o72Na8AauSKyErgGyBORapyWTxqAqn7X3ezPgNdU9WTYrpOBl9xvvT7gx6q6ZqQB51jSN0NQU1NDUdHp4h6FhYUA/rPscj/wivu8AKgKW1ftLitwn/defgb3WtaDANOmTRta8DHmwPFW6ls6+disPK9DMSMwYNJX1TsHsc2zOEM7w5cdAC4ebmD9Gdczp36H3aBlIktEPg3Mx/nWGhGq+jTwNMD8+fM1Uu/rhS1HGgG4dNr4s29oYlrcfUezlr4ZioKCAqqqTjfWq6urAbp6byciNwCPAbeoaqe7uAYIrwFY6C6r4XQXUPjyhLa1qpGxGT6Kbe78uBZ/Sd8u5JohKCsro7KykoMHD9LV1cWqVasAGsO3EZFLgKdwEn5d2KpXgRtFZLx7AfdG4FVVrQWaReQKd9TO3YRdz0pUW480Mq8olxQrfh7X4i7pZ6Slku5LsaRvBsXn87FixQoWLVpEaWkpS5YsAegQkSdF5BZ3s28A2cBP3SlDVgO4w4+/Cmx0H0+GDUn+PPA9YB+wn9PXARJSW1eA3ceabdROAhiwTz8W2fw7ZijKy8spLy8/9frxxx9HVZ/oea2qN/S3r6o+AzzTx/JNwAURDjVmba9uIqQwb1qu16GYEYq7lj5Y0jcm2t6tagTg4sJcT+MwIxeXST/H5t8xJqq2Hmlk+sQxTMxO9zoUM0JxmfStpW9MdG2tci7imvhnSd8Yc1a1Te0ca+6wpJ8g4jfpt1nSNyYatro3ZV1iN2UlhLhM+jkZPlo6A4RCcX2DozFxYWtVI/7UFEqnjPU6FBMB8Zn0M9NQhZZOm4rBmNH27pFG5k7NId1n9XATQVwm/Z6ZNu0GLWNGVyAYYltNI5fY+PyEEZdJ3+bfMSY6dh9roaM7ZBdxE0hcJn1r6RsTHVvdm7JsZs3EEddJ31r6xoyu96oamZjlp3C8FUFPFAMmfRF5RkTqRKTPqlcico2INLkTVW0VkSfC1i0WkT1uHdFHIhV0T/eO3ZVrzOjaXtPEhYXjSJISwElhMC39Zxm4/ufvVHWe+3gSQERSgW/jFEWfC9wpInNHEmwPa+kbM/o6uoNU1rVywdRxXodiImjApK+qvwVODLRdHxYA+1T1gKp2AauAW4fxPmfI8qeSmiKW9I0ZRbtqmwmGlAsKLOknkkj16V8pIu+JyCsicr67rL/6oiMmIuRk+Ghut3H6xoyWHTVNAFxYaEk/kURiPv0twHRVbRWRcuDnwKyhvslQC0jb/DvGjK7tNU1MyPIzdVyG16GYCBpxS19Vm1W11X3+MpAmInn0X1+0v/d5WlXnq+r8/Pz8AY9rSd+Y0bWjppnzp+bYRdwEM+KkLyLnuHVCEZEF7nt+gFNebpaIzBQRP7AUWD3S4/UYN8ZPY9sZ9a2NMRHQ0R1k7/stXGj9+QlnwO4dEVkJXAPkiUg18GUgDUBVvwvcDvyViASAdmCpqioQEJEv4BSXTgWeUdWdkQp82oRMVh9pQFWtJWJMhO051kIgpJb0E9CASV9V7xxg/QpgRT/rXgZeHl5oZzdr0liaOwLUt3QyKcf6HE3/1qxZw0MPPUQwGOSBBx44Y72ILAS+BVyE02h50V1+LfDNsE3Pc9f/XESeBa4Gmtx196rq1lH7EFG23b2IayN3Ek9cFkYHmDUpG4DKulZL+qZfwWCQZcuWsXbtWgoLCykrKwPofcIcAe4Fvhi+UFXXAfMARGQCsA94LWyTL/X8gUg0O482MS4zze7ETUBxOQ0DQMlkN+m/3+JxJCaWbdiwgZKSEoqLi/H7/SxduhQgN3wbVT2kqtuA0Fne6nbgFVVtG71oY8f2miYuLLA7cRNR3Cb9/Ox0xmWmUVnX6nUoJobV1NRQVHR6EFlhYSGAfxhvtRRY2WvZ10Vkm4h8U0T6rBguIg+KyCYR2VRfXz+Mw0ZfZyDInmMt1rWToOI26YsIsyZlW9I3o05EpgAX4gxK6PEoTh9/GTAB+Pu+9h3qUORYsPdYK91Bu4ibqOI26QPMmpzNPkv65iwKCgqoqjp9Y3h1dTXAUMf6LgFeUtVTN4aoaq06OoHv40w7khB2HO25iJvjcSRmNMR10i+ZNJYTJ7v4oLXT61BMjCorK6OyspKDBw/S1dXFqlWrABqH+DZ30qtrx239496jchvQ5yy08Wh7TRM5GT6mTRjjdShmFMR10g8fwWNMX3w+HytWrGDRokWUlpayZMkSgA4ReVJEbgEQkTL3HpQ7gKdE5NT9JCIyA+fO8rd6vfXzIrId2A7kAV+LwseJih01TVxgF3ETVtwO2QSnewecpH9F8USPozGxqry8nPLy8lOvH3/8cVT1VN0HVd2IM03IGVT1EH1MFKiq10U+Uu91BULsrm3hs1fN8DoUM0riuqV/Tk4G2ek+9tmwTWMiorKuha5giPPtIm7CiuukLyKU2AgeYyJmV63TgJo7xS7iJqq4TvqADds0JoJ21TaTkZbCzLwsr0MxoyT+k/7kbOpbOm3GTWMioOJoM3POySE1xS7iJqr4T/qTxgLYeH1jRkhV2XWsmblTxnodihlFcZ/0S2zYpjERcay5g8a2buvPT3Bxn/QLcjPJTEul8n1L+saMRMXRZgBKLekntLhP+ikpPSN4bNimMSOxq9ZJ+udZ0k9oAyZ9EXlGROpEpM/bzEXkLnemwe0i8raIXBy27pC7fKuIbIpk4OFmTbI5eIwZqV21LUyfOIbs9Li+Z9MMYDAt/WeBxWdZfxC4WlUvBL4KPN1r/bWqOk9V5w8vxIGVTM6mtqmDlg4rlG7McFXUNlN6jrXyE92ASV9VfwucOMv6t1W1wX25nn5uZx9NNoLHmJFp6wpw6IOT1p+fBCLdp38/8ErYawVeE5HNIvLg2XYcSbGJ86c6J+rb+z8YYrjGGIDdx1pQhblTLeknuoglfbeI9P18uJjER1X1UuAmYJlbgLpPIyk2MTU3kyuLJ/KTjVWEQjqc8I1JaqdH7tgY/UQXkaQvIhcB3wNuVdVTzW1VrXF/1gEvMYqFJpYuKOLIiTbeOWCtfWOGaldtMzkZPgpyrRB6ohtx0heRacDPgM+o6t6w5VkiMrbnOXAjo1hoYtH555A7Jo2VG46M1iGMSVi7apspnZJjc+gngQHHZonISuAaIM8tNPFlIA1AVb8LPAFMBP7DPWEC7kidycBL7jIf8GNVXTMKnwGAjLRU/uySAp5ff4QTJ7uYkDWc2tfGJJ9QSNl9rIUl84sG3tjEvQGTvqreOcD6B4AH+lh+ALj4zD1Gz9KyaXz/D4f42ZZqHvhYcTQPbUzcOnyijbauoE2/kCTi/o7ccHPOGcsl03JZtbEKVbuga8xg9NyJayN3kkNCJX2AO8umsa+ulS1HGgbe2BhDxdFmUt3pTEziS7ik/6cXTSHLn8rKDVVeh2JixJo1a5gzZw4lJSUsX778jPUislBEtohIQERu77Uu6E4jslVEVoctnykifxSRfSLyExGJ24tIu2qbOTc/i4y0VK9DMVGQcEk/K93HrZcU8KttR62wiiEYDLJs2TJeeeUVKioqWLlyJUBGr82OAPcCP+7jLdrdaUTmqeotYcv/CfimqpYADTj3qMSlitpm689PIgmX9AE+c8V0OrpDvLDJWvvJbsOGDZSUlFBcXIzf72fp0qUAueHbqOohVd0GhAbznuIMSbsOeNFd9APgtogFHUUnTnZR29Rh/flJJCGTfumUHBbMnMCP1h8maHfoJrWamhqKik4PRSwsLAQYSldMhjs9yHoRuc1dNhFoVNWA+7oaKOhr55FMLxINpy7iThnncSQmWhIy6QPcc+UMqk608+aeOq9DMfFtunvfyaeAb4nIuUPZeSTTi0SDTb+QfBI26d94/mTOycng2bcPeR2K8VBBQQFVVae7+aqrqwEGfbEnbCqRA8CbwCXAB0CuiPTc51II1EQm4uiqqG3mnJwMJmanex2KiZKETfppqSncdfk0fld5nP31NuVysiorK6OyspKDBw/S1dXFqlWrABoHs6+IjBeRdPd5HnAVUKHOTSDrgJ6RPvcAv4h48FFQcbTZ+vOTTMImfYClC6bhT03hR+8c9joU4xGfz8eKFStYtGgRpaWlLFmyBKBDRJ4UkVsARKTMnWLkDuApEdnp7l4KbBKR93CS/HJVrXDX/T3wsIjsw+nj/69ofq5I6OgOsq++1UbuJJmErouWPzadP71oCi9uruaLi+ZYGbgkVV5eTnl5+anXjz/+OKr6RM9rVd1IH8V/VPVt4MK+3tPt7hm1WWOjofL9VoIhtZZ+kknolj7A3VdOp7UzwEvvxmWXqzGjpqK2CcBa+kkm4ZP+vKJcSqfk8MJGG7NvTLiKo81k+VOZNmGM16GYKEr4pC8iLJlfyPaaplPD04wxbiH0KTmkpNgc+skk4ZM+wG3zCvCnptgdusa4QiFlV22L9ecnoaRI+uOz/Nx4/mR+vrWGzkDQ63CM8VxVQxutnQHrz09Cg0r6IvKMiNSJSJ/lDsXx7+6Mg9tE5NKwdfeISKX7uCdSgQ/VkvlFNLZ1s7bifa9CMCZm9HR1Wks/+Qy2pf8ssPgs628CZrmPB4HvAIjIBJzyipfjDG/7soiMH26wI/HRkjwKcjP5iV3QNYaKWmcO/dmTbfqFZDOopK+qvwVOnGWTW4EfqmM9zi3qU4BFwFpVPaGqDcBazv7HY9SkpAi3X1bI7/cdp7qhzYsQjIkZFUdtDv1kFak+/QIgvAndM+tgf8vPEI3ZCG+/zLn/5r8325h9k9xsDv3kFTMXcqMxG2HRhDFcdW4eP91cRcimXDZJyubQT26RSvo1QFHY655ZB/tb7plPXFZAdUM771ZZDV2TnGwO/eQWqaS/GrjbHcVzBdCkqrXAq8CN7myF44Eb3WWeuaF0Mn5fCr98r9bLMIzxzI4aZ/qF862ln5QGO2RzJfAOMEdEqkXkfhH5nIh8zt3kZeAAsA/4T+DzAKp6AvgqsNF9POku88zYjDSunZPPr7fXWlUtk5R2HG2mIDeT8VlxW8vdjMCgpp1U1TsHWK/Asn7WPQM8M/TQRs/HL57KqzvfZ8PBE1x57kSvwzEmqnbWNHFBgbXyk1XMXMiNpuvOm0RmWiq/3HbU61CMiaqWjm4OHD/JBVOtPz9ZJWXSH+P3ccPcyazZcYzuYMjrcIyJml21LQBcUGBJP1klZdIHuPmiKZw42cXb+z/wOhRjoubURVzr3klaSZv0r56dz9h0H796z7p4TPLYcbSJSWPTmTQ2w+tQjEeSNulnpKXyJ+dPZs3OYzbzpkkaO2uarWsnySVt0gf4+EVTaekI8Lu9x70OxYyiNWvWMGfOHEpKSli+fPkZ60VkoYhsEZGAiNwetnyeiLwjIjvd2WM/GbbuWRE5KCJb3ce86Hya4WvvClJZ18IFNj4/qSV1pfCrSvIYl5nGyztquWHuZK/DMaMgGAyybNky1q5dS2FhIWVlZQC9+zaOAPcCX+y1vA24W1UrRWQqsFlEXlXVRnf9l1T1xVEMP6J2H2smpHC+tfSTWlK39P2+FK6dk8+63XV2o1aC2rBhAyUlJRQXF+P3+1m6dClAbvg2qnpIVbcBoV7L96pqpfv8KFAHjM7EUFGww51D37p3kltSJ32AG+ZOpqGtmy1HbC6eRFRTU0NR0enpnwoLCwGGfCuqiCxw99sftvjrbrfPN0UkfaSxjradNU2MH5PG1HF2ETeZJX3SXzg7n7RU4TdWUcv0w60N8SPgs6ra823gUeA8oAyYAPx9P/uO+pThg7XjaBMXFIxDxAqhJ7OkT/o5GWlcUTyRtbss6SeigoICqqpOl3Sorq4G6Brs/iKSA/waeMwtEASAqta6RYM6ge/jVIY7QzSmDB+MrkCIPcdaON/uxE16SZ/0wZl580D9SfbXt3odiomwsrIyKisrOXjwIF1dXaxatQqgcTD7iogfeAmnKtyLvdZNcX8KcBvQZ/3oWLH3/Ra6g2pz7hhL+sCpkTuvW2s/4fh8PlasWMGiRYsoLS1lyZIlAB0i8qSI3AIgImUiUg3cATwlIjvd3ZcAC4F7+xia+byIbAe2A3nA16L5uYZq51HnTlybc8ck9ZDNHgW5mcydksNvKup4cOG5XodjIqy8vJzy8vJTrx9//HFU9Yme16q6EafAz4eo6nPAc329p6peNwqhjpodNc2MTfcxbcIYr0MxHrOWvuuGuZPZdPgEJ04OurvXmLixvaaJ0qk5pKTYRdxkN9giKotFZI+I7BORR/pY/82wr797RaQxbF0wbN3qCMYeUX9SOpmQwrrddV6HYkxEdQaCVBxtZl5RrtehmBgwYPeOiKQC3wb+BKgGNorIalWt6NlGVf82bPv/AVwS9hbtqjovYhGPkgsKcpick85vdr3PJy4745u+MXFr59FmuoIhLp2W63UoJgYMpqW/ANinqgdUtQtYBdx6lu3vBFZGIrhoEhFuKJ3MW3vrbQI2k1DePdIIwCXTxnsbiIkJg0n6BUBV2Otqd9kZRGQ6MBN4I2xxhntzynoRua2/g8TCTSzXl06irSvIhoOelvE1JqK2HGmgIDeTyTl2J66J/IXcpcCLqhreVJ6uqvOBTwHfEpE+h8fEwk0sVxRPxJ+awlt7vL1z0phIevdwA5dY145xDSbp1wBFYa8L3WV9WUqvrh1VrXF/HgDe5MP9/TFljN/H5cUTeHOvJX2TGI41dXC0qcO6dswpg0n6G4FZIjLTvUNxKXDGKBwROQ8YD7wTtmx8z0RUIpIHXAVU9N43llw9O599da1UN7R5HYoxI/auO5GgXcQ1PQZM+qoaAL4AvArsAl5Q1Z3hdzS6lgKrVDV8juJSYJOIvAesA5aHj/qJRdfMmQTAm9bFYxLAu1WN+H0pNueOOWVQd+Sq6svAy72WPdHr9Vf62O9t4MIRxBd15+ZnUTg+kzf31PPpK6Z7HY4xI7LlcAMXTM3B77P7MI3DzoReRIRr5uTz9v7jNnTTxLWuQIjtNU1cav35Jowl/T5cM9sZurnpkBVWMfFrV20znYGQXcQ1H2JJvw9XnusM3Xxzj03JYOJXTzW4S6fnehuIiSmW9PuQle6jbOZ4u5hr4tq7RxqZMi6DKeMyvQ7FxBBL+v24ZvYkKutaqWls9zoUY4ZlyxG7KcucyZJ+P66Z49wVbF08Jh7VtXRQ3dBuF3HNGSzp96NkUjYFuZnWxWPi0pbDjQDW0jdnsKTfDxFh4ex83tn/AV2BkNfhGDMkv62sJzvdx4UFuV6HYmKMJf2zuHp2Pq2dgVOjIIyJB6rKm7vr+GhJnt2UZc5gZ8RZfKRkIr4U4S2bgM3Ekb3vt3K0qYNrz/NmtloT2yzpn0VORhqXTh9vUy3HuTVr1jBnzhxKSkpYvnz5GetFZKGIbBGRgIjc3mvdPSJS6T7uCVt+mYhsd0uI/ruIxEzx2XXu4IOeeaSMCWdJfwBXz86noraZupYOr0MxwxAMBlm2bBmvvPIKFRUVrFy5EqB3NZEjwL3Aj8MXisgE4MvA5TgV5L4sIj3DYb4D/AUwy30sHrUPMUTrdtcxd0qOFU0xfbKkP4CrZztfkX+397jHkZjh2LBhAyUlJRQXF+P3+1m6dClAbvg2qnpIVbcBva/YLwLWquoJVW0A1gKLRWQKkKOq691ZZX8I3Dban2Uwmju62XS4wbp2TL8s6Q9g7pQc8rLTrV8/TtXU1FBUdLoGUGFhIYB/kLv3Vyq0wH3ee/kZol0G9PeVxwmGlGuta8f0w5L+AFJShIWz8/hdZT3BkA68gzFhol0GdN3uOsZlpjGvKHfUj2XikyX9Qbh6dj4Nbd1sr2nyOhQzRAUFBVRVnW6sV1dXA3QNcvf+SoXWuM97L/dUKKS8ubeehbPz8aXar7bp26DODBFZLCJ73JEKj/Sx/l4RqReRre7jgbB1fY5+iCcfm5WPCDaKJw6VlZVRWVnJwYMH6erqYtWqVQCNg9z9VeBGt+zneOBG4FVVrQWaReQKd9TO3cAvRiP+oaiobaa+pZNr51h/vunfgElfRFKBbwM3AXOBO0Vkbh+b/kRV57mP77n7nm30Q9yYkOXnosJc3tpr8/DEG5/Px4oVK1i0aBGlpaUsWbIEoCO83KeIlIlINXAH8JSI7ARQ1RPAV3HqRG8EnnSXAXwe+B6wD9gPvBLVD9aHdbvrEIGFsy3pm/4NplziAmCfqh4AEJFVwK0MrsD5qdEP7r5rcYa2rRxeuN65elYeK9bto7Gti9wxg70OaGJBeXk55eXlp14//vjjHyr3qaob+XB3DWHrngGe6WP5JuCCyEc7fG/sqeOiwlzystO9DsXEsMF07/Q3gqG3T4jINhF5UUR6+kEHu2/URzkM1TXnTSKk8NtKG7ppYs+OmibePdJI+QXneB2KiXGRutrzS2CGql6EM5b5B0N9g2iPchiqiwtzmZjl541d73sdijFn+O5b+xmb7uPOy6d5HYqJcYNJ+v2NYDhFVT9Q1U735feAywa7b7xITRGumTOJdXvqCQRt1k0TOw5/cJKXt9dy1xXTyclI8zocE+MGk/Q3ArNEZKaI+IGlwOrwDdw7FHvcAuxyn/c5+mHkYXvj+tJJNLV3s+VIo9ehGHPK0789gC8lhfuumuF1KCYODHghV1UDIvIFnGSdCjyjqjtF5Elgk6quBv7aHQkRAE7gzGOCqp4QkZ7RD/Dh0Q9x52Oz8vClCK/vfp8FMyd4HY4x1Ld08tPN1XzisgIm2Vw7ZhAGM3oHVX0ZeLnXsvDRD48Cj/azb5+jH+LR2Iw0Li+ewBu76nj0plKvwzGG7//hIN3BEA8uPNfrUEycsNv2hui68yZTWdfKkQ/avA7FJLmWjm5+tP4wN11wDjPzsrwOx8QJS/pDdEOpM5HVG7ttFI/x1op1+2jpCPC5q62VbwbPkv4QTZ+Yxbn5Wby+2+7ONd55fdf7PPXWAe5cUMRFhbleh2PiiCX9Ybi+dDJ/PHCC1s6A16GYJFR1oo2//clWzp+aw5c/fr7X4Zg4Y0l/GK47bxJdwRC/r4y9O4dNYusMBPn881tQ4D/uupSMtFSvQzJxxpL+MFw2fTw5GT5+s8u6eEx0ffVXFWyvaeJf77iY6RPt4q0ZOkv6w5CWmsL1pZNZW/E+3XZ3romSdbvreG79Ef7iYzO58XybY8cMjyX9YfrTC6fQ1N7NH/bZBGxm9DW1d/Poz7Yze3I2X1w0x+twTByzpD9MH5udx9gMH7/aVut1KCYJfP3XFdS3dvIvd1xMus/68c3wWdIfpnRfKjfOPYdXdx6jK2BdPGb0vLmnjhc2VfO5q4tteKYZMUv6I3DzRVNo6Qjw+302iseMjuYOp1tn1qRs/vr6WV6HYxKAJf0RuKokjxzr4jGjQFV5becxPvEfb/N+c4d165iIGdSEa6Zvfl8Ki84/hzU7jtHRHbQx0yYiNh46wT++vIt3jzRSnJfF05+Zz8VFuV6HZRKEtfRH6OaLp9LSGeB3VkYxZq1Zs4Y5c+ZQUlLC8uXLz1gvIuki8hMR2ScifxSRGe7yu0Rka9gjJCLz3HVvisiesHWTIhHriZNd3PWff6S2sYPlf34hr/3tQm6YOzkSb20MYC39EfvIuRPJHZPGr7cd5U/slzPmBINBli1bxtq1ayksLKSsrAyg98Tz9wMNqloiIkuBfwI+qarPA88DiMiFwM9VdWvYfne5BdIj5uDxk3QFQ/zjn1/AdefZ+WQib1AtfRFZ7LZq9onII32sf1hEKtzC6K+LyPSwdcGw1tDq3vvGu7TUFBaffw5rK96nozvodTimlw0bNlBSUkJxcTF+v5+lS5cC5Pba7FZO13V+EbheRKTXNncCq0Y1WKC6wZmyu3D8mNE+lElSAyZ9EUkFvg3cBMwF7hSRub02exeY7xZGfxH457B17ao6z33cEqG4Y8rNF03lZFeQ1ypsuuVYU1NTQ1HR6TLNhYWFAP5emxUAVeBUigOagIm9tvkksLLXsu+7jZl/6OOPBAAi8qCIbBKRTfX1A4/yqm5odwLKzRxwW2OGYzAt/QXAPlU9oKpdOK2dW8M3UNV1qtpTVWQ9TgH0pPGRcycybcIYnnvnsNehmFEgIpcDbaq6I2zxXap6IfAx9/GZvvZV1adVdb6qzs/Pzx/wWNUN7UzI8pOVbj2vZnQMJumfagW5qt1l/bkfeCXsdYbb0lkvIrf1t9NQW0SxJCVF+PQV09hw6AS7jzV7HY4JU1BQQFXV6dO3uroaoKvXZjVAEYCI+IBxwAdh65fSq5WvqjXuzxbgxziNoxGrbmijcLy18s3oiejoHRH5NDAf+EbY4umqOh/4FPAtEemzzM9QW0Sx5o7Likj3pfAja+3HlLKyMiorKzl48CBdXV2sWrUKoLHXZquBe9zntwNvqKoCiEgKsISw/nwR8YlInvs8DbgZ2EEE1DS0W9I3o2owSf9UK8hV6C77EBG5AXgMuEVVO3uWh7WIDgBvApeMIN6YNT7Lz8cvnspL79bQ3NHtdTjG5fP5WLFiBYsWLaK0tJQlS5YAdIjIkyLSc43pv4CJIrIPeBgIH6ywEKhyz98e6cCrIrIN2Irz+/CfI401FFKqG9vtIq4ZVYNJ+huBWSIyU0T8OF91PzQKR0QuAZ7CSfh1YcvHi0i6+zwPuAqoiFTwsebuK6fT1hXkpS1n/E00HiovL2fv3r3s37+fxx57DABVfUJVV7vPO1T1DlUtUdUF4QleVd9U1SvC309VT6rqZap6kaqer6oPqeqIh24db+2kKxCylr4ZVQMmfXc0wxeAV4FdwAuqurNXS+kbQDbw015DM0uBTSLyHrAOWK6qCZv0LyrM5eLCcfxo/WHc3gFjBq3KHbljSd+MpkENEVDVl4GXey17Iuz5Df3s9zZw4UgCjDefuXIGX/zpe7yz/wM+UpLndTgmjtgYfRMNNg1DhN180RRyx6Tx7NuHvA7FxBkbo2+iwZJ+hGWkpXL3lTN4reJ9Nh9u8DocE0dsjL6JBkv6o+AvFxYzaWw6X/1VBaGQ9e2bwbEx+iYaLOmPgqx0H19aNIetVY38cttRr8MxccLG6JtosKQ/Sj5xaSEXFOTwT6/spr3LJmIzZ2dj9E20WNIfJSkpwj/86VyONnXwvd8dGHgHk9RsjL6JFkv6o+jy4oksPv8cvvPWfo41dXgdjolhNkbfRIsl/VH2aPl5hFT5/PObbb590y8bo2+ixZL+KJs+MYtvfXIeW4408sh/b7M7dU2fbIy+iRZL+lGw+IIpfGnRHH6+9SjfXrfP63BMDKpptDH6JjrsDIuSz19zLvvqWvmX1/YyMy+bP71oitchmRhSbcM1TZRYSz9KRIT/8+cXctn08Xxh5RYee2k7jW29a3mYZGU3ZplosaQfRRlpqfzgvgV89iMzWbWxiuv+9S1e2FRld+0mOVV1b8yyi7hm9FnSj7LsdB9PfHwuv/zCR5mZl8X/fHEbl/+f13n859v5w77jBIIhr0M0UVbf2kmnjdE3UWJ9+h6ZOzWHn/7llazZeYxfvneUFzdX89z6I2T5UynOz6Y4P4vivGzyxvrJTvcxNsNHlt9Heloq6b4U/L4U/Kkp+FIFX0oKqSlCSJVQSAmqMibNR06mDxHx+qOaAVTbGH0TRZb0PZSSIpRfOIXyC6fQ3hXkrb11rD9wgv31rWw61MDq944ykhGefl8K+dnp5I9NZ2KWnwlZfiZmpzN+TBrjMk8/csf4GZ+VxvgxfjLSUiP3Ac2gnE761r1jRt+gkr6ILAb+DUgFvqeqy3utTwd+CFwGfAB8UlUPueseBe4HgsBfq+qrEYs+gWT6U1l8wRQWX3B6VE9Hd5Cm9m5aOgK0dgY42RmgMxCkKxCiMxCiO6gEgiG6Q0owGCIlRUgRITVFONkZoL6lk/qWTupaOjna1MGOo02cONlFd7D/vyRpqUKGL5X0tBTSfalkpacyNiON7HQf2ek+0tNSyAj7tpGemkJaagppPuenP1VIS00hPS2FzDQfY/ypZPpT8aU4caWmCP5U95uK+20FoCeiFBHnfX0p+FIkKb6p9NyYZWP0TTQMmPRFJBX4NvAnQDWwUURW9yp7eD/QoKolIrIU+CfgkyIyF6em7vnAVOA3IjI7EvVEk0FGWioZaalMzonce6oqJ7ucPyZNbd00tXfT2NZFQ1s3DW1dtHQE3D8qQTq6Q5zsDNDS6ayramijs/v0uq5giK7A6F2DEIG0lJ4uLOcPRvgfAV+K8wfG7zvdvaUKIVVuvXgqD984B4A1a9bw0EMPEQwGeeCBB/o4Tt+NFhGZgVMidI+76XpV/Zy7z2XAs0AmTlW5h3SYd97ZPPommgZzli0A9vUUixaRVcCtfLjA+a3AV9znLwIrxPntvBVYpaqdwEER2ee+3zuRCd8MlYicarVHomWpqgRDSlfQ+ebR7f4h6AyEaOsK0N4VpL07SCCkBINKIORs1xX48B8NERA49V6d7h+Vnm8zgZBznFPHxT1uwDlmMKSION8UUgSKJjhdJcFgkGXLlrF27VoKCwspKysDyOj1MfpstLjr9qvqvD4++neAvwD+iJP0FwOvDOffcPakbNLsvg0TJYNJ+gVAVdjrauDy/rZR1YCINAET3eXre+1b0NdBRORB4EGAadOmDSZ2EwNExGmJp8bmQLANGzZQUlJCcXExAEuXLmXbtm25vTbrr9HSJxGZAuSo6nr39Q+B2xhm0r/3qpnD2c2YYYmZ31RVfVpV56vq/Pz8fK/DMQmipqaGoqKiU68LCwsB/L02+1CjBehptADMFJF3ReQtEflY2PbVYfuftTEjIptEZFN9ff1IP44xIzaYpF8DFIW9LnSX9bmNiPiAcTh9o4PZ15hYVQtMU9VLgIeBH4vIkK6wWGPGxJrBJP2NwCwRmSkifpwLs6t7bbMauMd9fjvwhntRazWwVETSRWQmMAvYEJnQjRlYQUEBVVWneyerq6sBes9/0WejRVU7VfUDAFXdDOwHZrvbF4btb40ZEzcGTPru190vAK/ijGR4QVV3isiTInKLu9l/ARPdC7UPA4+4++4EXsC56LsGWGYjd0w0lZWVUVlZycGDB+nq6mLVqlUAjb0267PRIiL57ug1RKQYp9FyQFVrgWYRucLt+78b+EU0Po8xIzWoMWKq+jLOCIXwZU+EPe8A7uhn368DXx9BjMYMm8/nY8WKFSxatIhgMMh9993Htm3bOkTkSWCTqq7GabT8yG20nMD5NguwEHhSRLqBEPA5VT3hrvs8p4dsvsIwL+IaE20Si0U95s+fr5s2bfI6DJOgRGSzqs6P9nHtvDajabDndcyM3jHGGDP6LOkbY0wSicnuHRGpBw73sSoPOB7lcM4mluKJpVggtuLpHct0VY36+MmznNcQW/9eEFvxxFIsEFvxhMcyqPM6JpN+f0Rkkxd9sf2JpXhiKRaIrXhiKZb+xFqMsRRPLMUCsRXPcGKx7h1jjEkilvSNMSaJxFvSf9rrAHqJpXhiKRaIrXhiKZb+xFqMsRRPLMUCsRXPkGOJqz59Y4wxIxNvLX1jjDEjYEnfGGOSSNwkfRFZLCJ7RGSfiDziwfGfEZE6EdkRtmyCiKwVkUr35/goxVIkIutEpEJEdorIQ17FIyIZIrJBRN5zY/nf7vKZIvJH9//XT9wZWqNGRFLdefB/FQvxnI2X57ad12eNJ+bO7Uic13GR9OV0nd6bgLnAneLU342mZ3FK4oV7BHhdVWcBr7uvoyEA/J2qzgWuAJa5/x5exNMJXKeqFwPzgMUicgVOycFvqmoJ0IBTkjCaHsKZFbaH1/H0KQbO7Wex87o/sXhuj/y8VtWYfwBXAq+GvX4UeNSDOGYAO8Je7wGmuM+nAHs8+vf5BU7hek/jAcYAW3DKaR4HfH39/4tCHIU4yeE64Fc45Xc9i2eAWD0/t+28HlQsnp/bkTqv46KlT991evssTxdlk9WZWx3gGDA52gGIyAzgEpwC3Z7E437l3ArUAWtxio00qlOLAaL//+tbwP/EmQ4ZnNKHXsZzNrF4btt5fTqOWDq3v0UEzut4SfoxT50/tVEd/yoi2cB/A3+jqs1exaOqQVWdh9MSWQCcF43j9kVEbgbq1Kl0ZUYomc9r93gxcW5H8rweVBGVGBCrtXbfF5EpqlorIlNwWgNRISJpOL8Yz6vqz7yOB0BVG0VkHc7XzFwR8bmtkGj+/7oKuEVEyoEMIAf4Nw/jGUgsntt2XvcSA+d2xM7reGnpD6ZOrxfCy+zdQ5RK5omI4FR72qWq/9fLeMQpKZjrPs/E6YPdBazDKT0YtVgAVPVRVS1U1Rk458kbqnqXV/EMQiye20l/XrvxxMy5HdHzOtoXREZwEaMc2IvTp/aYB8dfCdQC3Th9Z/fj9Km9DlQCvwEmRCmWj+J8xd0GbHUf5V7EA1wEvOvGsgN4wl1eDGwA9gE/BdI9+H92DfCrWInnLHF6dm7beX3WeGLy3B7peW3TMBhjTBKJl+4dY4wxEWBJ3xhjkoglfWOMSSKW9I0xJolY0jfGmCRiST9Bicg1PTPxGZNI7NweGUv6xhiTRCzpe0xEPu3O2b1VRJ5yJ3hqFZFvunN4vy4i+e6280RkvYhsE5GXeuYVF5ESEfmNO+/3FhE51337bBF5UUR2i8jz7h2PiMhyd87ybSLyLx59dJPg7NyOUdG8k8weZ9xZVwr8EkhzX/8HcDfOXYl3ucueAFa4z7cBV7vPnwS+5T7/I/Bn7vMMnGlgrwGacObjSAHewbnjcSLOVLU9N+blev3vYI/Ee9i5HbsPa+l763rgMmCjO33r9Ti3VYeAn7jbPAd8VETG4ZzEb7nLfwAsFJGxQIGqvgSgqh2q2uZus0FVq1U1hHNL+wycX5YO4L9E5M+Bnm2NiSQ7t2OUJX1vCfADVZ3nPuao6lf62G64c2V0hj0P4hRbCOBMEfsicDOwZpjvbczZ2Lkdoyzpe+t14HYRmQSnaoFOx/n/0jNz3qeA36tqE9AgIh9zl38GeEtVW4BqEbnNfY90ERnT3wHducrHqerLwN8CF4/C5zLGzu0YFS/z6SckVa0QkceB10QkBWemw2XASWCBu64O+KS7yz3Ad90T/wDwWXf5Z4CnRORJ9z3uOMthxwK/EJEMnNbYwxH+WMbYuR3DbJbNGCQiraqa7XUcxkSandves+4dY4xJItbSN8aYJGItfWOMSSKW9I0xJolY0jfGmCRiSd8YY5KIJX1jjEki/x/5EuOap0txfgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_loss = history.history['loss']\n",
    "train_accuracy = history.history['accuracy']\n",
    "\n",
    "plt.figure()\n",
    "plt.subplot(1, 2, 1)\n",
    "plt.plot(train_loss, label='train loss')\n",
    "plt.title('loss')\n",
    "plt.xlabel('epochs')\n",
    "\n",
    "plt.subplot(1, 2, 2)\n",
    "plt.plot(train_accuracy, label='train accuracy')\n",
    "plt.title('accuracy')\n",
    "plt.xlabel('epochs')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "offensive-interaction",
   "metadata": {},
   "source": [
    "#### Step 5. 모델 평가하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "sound-theme",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "# 위의 과정을 모두 담은 decoder_inference() 함수 선언\n",
    "def decoder_inference(sentence):\n",
    "    sentence = preprocess_sentence(sentence)\n",
    "\n",
    "    # 입력된 문장을 정수 인코딩 후, 시작 토큰과 종료 토큰을 앞뒤로 추가.\n",
    "    # ex) Where have you been? → [[8331   86   30    5 1059    7 8332]]\n",
    "    sentence = tf.expand_dims(\n",
    "        START_TOKEN + tokenizer.encode(sentence) + END_TOKEN, axis=0)\n",
    "\n",
    "    # 디코더의 현재까지의 예측한 출력 시퀀스가 지속적으로 저장되는 변수.\n",
    "    # 처음에는 예측한 내용이 없음으로 시작 토큰만 별도 저장. ex) 8331\n",
    "    output_sequence = tf.expand_dims(START_TOKEN, 0)\n",
    "\n",
    "    # 디코더의 인퍼런스 단계\n",
    "    for i in range(MAXLEN):\n",
    "        # 디코더는 최대 MAX_LENGTH의 길이만큼 다음 단어 예측을 반복합니다.\n",
    "        predictions = model(inputs=[sentence, output_sequence], training=False)\n",
    "        predictions = predictions[:, -1:, :]\n",
    "\n",
    "        # 현재 예측한 단어의 정수\n",
    "        predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)\n",
    "\n",
    "        # 만약 현재 예측한 단어가 종료 토큰이라면 for문을 종료\n",
    "        if tf.equal(predicted_id, END_TOKEN[0]):\n",
    "            break\n",
    "\n",
    "        # 예측한 단어들은 지속적으로 output_sequence에 추가됩니다.\n",
    "        # 이 output_sequence는 다시 디코더의 입력이 됩니다.\n",
    "        output_sequence = tf.concat([output_sequence, predicted_id], axis=-1)\n",
    "\n",
    "    return tf.squeeze(output_sequence, axis=0)\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "mathematical-utility",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "슝=3\n"
     ]
    }
   ],
   "source": [
    "def sentence_generation(sentence):\n",
    "    # 입력 문장에 대해서 디코더를 동작 시켜 예측된 정수 시퀀스를 리턴받습니다.\n",
    "    prediction = decoder_inference(sentence)\n",
    "\n",
    "    # 정수 시퀀스를 다시 텍스트 시퀀스로 변환합니다.\n",
    "    predicted_sentence = tokenizer.decode(\n",
    "          [i for i in prediction if i < tokenizer.vocab_size])\n",
    "\n",
    "    print('입력 : {}'.format(sentence))\n",
    "    print('출력 : {}'.format(predicted_sentence))\n",
    "\n",
    "    return predicted_sentence\n",
    "print(\"슝=3\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "average-edmonton",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 오늘은 좀 힘든 날이었어\n",
      "출력 : 공부 도와달라고 해보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'공부 도와달라고 해보세요 .'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('오늘은 좀 힘든 날이었어')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "different-program",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 어디 놀러가볼까?\n",
      "출력 : 달라지는게 없다면 지금이 좋을지도 몰라요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'달라지는게 없다면 지금이 좋을지도 몰라요 .'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('어디 놀러가볼까?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "motivated-marble",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 사랑한다 말해줘!\n",
      "출력 : 마음의 정리가 끝났길 바랍니다 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'마음의 정리가 끝났길 바랍니다 .'"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('사랑한다 말해줘!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "senior-hours",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 카페 갈까?\n",
      "출력 : 카페 데이트 좋죠 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'카페 데이트 좋죠 .'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('카페 갈까?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "basic-system",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : $%**\n",
      "출력 : 사랑의 빠르게 흘러 가죠 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'사랑의 빠르게 흘러 가죠 .'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('$%**')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "atlantic-liverpool",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 어디 카페가 좋아?\n",
      "출력 : 연락 안되면 걱정된다고 꼭 중간중간 연락 달라고 해보세요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'연락 안되면 걱정된다고 꼭 중간중간 연락 달라고 해보세요 .'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('어디 카페가 좋아?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "anonymous-collar",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 : 12시 땡!\n",
      "출력 : 하루가 또 가네요 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'하루가 또 가네요 .'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_generation('12시 땡!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "italian-crash",
   "metadata": {},
   "source": [
    "#### 결론 : 약간은 미숙한 응답들이 많다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unknown-central",
   "metadata": {},
   "source": [
    "## 회고\n",
    "#### Point 1) 한국어 전처리 및 예측\n",
    "- 영어 전처리만 보다가 한국어 전처리하려니까 많은 부분이 다르다.\n",
    "- 일단 사용하는 punctuation이 살짝 다르고(한국말에서는 소유격을 잘 안 쓴다)\n",
    "- 그리고 대문자도 없다는 점이 어쩌면 더 단순한 것 같으면서도 뭔가 빼먹은 듯한 느낌을 준다...\n",
    "- 실습 시 영어 문장 대답은 별 기대를 안했는데 한국어는 좋은 예측이 나오길 많이 기대했다.(그래서 `d_model`도 512로 올렸다.)\n",
    "- 그리고 결과를 봤는데 정말 간단한 문장 혹은 훈련 데이터와 정확히 맞아떨어지는 것들만 그럭저럭 괜찮은 답변이 온 것 같다.\n",
    "  - 입력 : `카페 갈까?` 출력 : `카페 데이트 좋죠 .`\n",
    "- 설명 보니까 BERT도 1달 동안 훈련시킨 거라고 하니까 어쩌면 현재 학습하는 수준에서는 이 정도 성과가 기대할 수 있는 수준인 것 같다.\n",
    "\n",
    "#### Point 2) `len(string)`, `len(list)`는 다르다...\n",
    "- 한 문장 내 '단어'의 갯수를 확인하는 게 우리는 중요하다.\n",
    "- 왜냐하면 그것에 따라 패딩할 때 `MAX_LENGTH`를 지정해주는 것이니..\n",
    "- 그런데 아래와 같이 코드 하니 밑의 그래프와 같은 결과가 나왔다.\n",
    "```python\n",
    "q_lengths = [len(sentence) for sentence in df['Q']]\n",
    "a_lengths = [len(sentence) for sentence in df['A']]\n",
    "```\n",
    "![](https://images.velog.io/images/moondeokjong/post/de6a608a-51be-4560-8133-b70b15f6c4a1/Screenshot%202022-02-17%20at%203.52.42%20PM.png)\n",
    "- 그리고 이것이 문제되는 줄도 모르고 패딩하다가 전처리가 잘됐는지 확인하는 '습관'을 실행하던 중 알게됐다.\n",
    "- 내가 당시 정한 `MAX_LENGTH`(65) 덕분에 아무 데이터도 제외되지 않았음을...(물론 나중엔 데이터가 적어서 일부러 다 포함하려 하긴 했지만...)\n",
    "- 그리고 다시 확인해 보니 맥시멈으로서 76개의 len을 가졌다던 그 문장은 실은 아래의 문장이었다.\n",
    "```python\n",
    "10808    연애는 문제가 있으면 문제를 해결 하기 위해 함께 노력하고 그 속에서 더욱 단단해지...\n",
    "Name: A, dtype: object\n",
    "```\n",
    "- 시각화가 아닌 텍스트로 이루어지는 프로그래밍에서는 내가 실행한 것과 그것의 결과를 확인하는 습관이 참 중요한 것 같다...\n",
    "\n",
    "#### Point 3) 갈수록 어려워지는 LMS 노드...\n",
    "- 저번주 언젠가를 기점으로 한번에 이해하기 어려운 개념과 코드들이 많이 등장하고 있다.\n",
    "- 그런데 해커톤 같은 건 실제 적용스럽게 하려면 더 어려울 텐데...\n",
    "- 이럴 때일수록 건강한 정신이 더 중요해지는 것 같다.\n",
    "- 욕심 부리지 말고 내 페이스에 맞게 오래오래 행복하게 공부하려던 초심을 잃지 말자"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}