tutorial/notebook/go_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这是一个使用 CodeFuse-Query 分析 Go 项目的教程。在教程中，你将体验到使用命令行工具对代码仓库进行数据化，然后使用 Godel 语言来分析这个仓库。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "检查cli是否就绪"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/sparrow-cli/sparrow\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!which sparrow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "STEP 0: 克隆要分析的仓库。我们以 [gorm](https://github.com/go-gorm/gorm.git) 项目为例。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-22T08:30:58.387715Z",
     "start_time": "2023-11-22T08:30:44.572634Z"
    }
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/go-gorm/gorm.git -q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "STEP 1: 代码数据化。使用 `sparrow database create` 命令创建一个db文件，指定待分析的仓库地址（当前目录下的gorm子目录），分析的语言（go），以及db文件的存储路径（放置在当前目录下的/db/gorm）。执行该命令之后，就会生成一份db文件，该文件存储着代码仓库的结构化数据，之后的分析就是针对这份数据进行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-23T03:46:32.220317Z",
     "start_time": "2023-11-23T03:46:12.785705Z"
    }
   },
   "outputs": [],
   "source": [
    "!sparrow database create --source-root gorm --data-language-type go --output ./db/gorm --overwrite > /dev/null"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "STEP 2: 使用Godel分析语言分析db文件。在本教程中，可以点击代码左侧的执行按钮，或使用快捷键：`Shift+Enter`，直接运行分析脚本。这里使用 `%db /path/to/db` 魔法命令来设置COREF db路径，内核会读取这个值来进行query查询。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<b>示例</b> 查询 [gorm](https://github.com/go-gorm/gorm.git) 的文件代码复杂度信息。\n",
    "\n",
    "第一行通过内核魔法命令指定分析的db路径，后面写查询文件代码复杂度Godel脚本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/workspaces/CodeFuse-Query/tutorial/notebook/db/gorm\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mSparrow database is set to: /workspaces/CodeFuse-Query/tutorial/notebook/db/gorm\n",
      "\u001b[0m"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-12-06 07:42:29,249 INFO: sparrow 2.0.0\n",
      " will start\n",
      "2023-12-06 07:42:29,249 INFO: database /workspaces/CodeFuse-Query/tutorial/notebook/db/gorm/coref_go_src.db size: 42.32 MB\n",
      "2023-12-06 07:42:29,250 INFO: execute : /sparrow-cli/godel-script/usr/bin/godel /tmp/godel-jupyter-t7lsy12n/query.gdl -p /sparrow-cli/lib-1.0 -o /tmp/tmphp2bm3pi.gdl\n",
      "2023-12-06 07:42:29,314 INFO: godel-script compile time: 0.06s\n",
      "2023-12-06 07:42:29,315 INFO: execute : /sparrow-cli/godel-1.0/usr/bin/godel /tmp/tmphp2bm3pi.gdl --run-souffle-directly --package-path /sparrow-cli/lib-1.0 --souffle-fact-dir /workspaces/CodeFuse-Query/tutorial/notebook/db/gorm --souffle-output-format json --souffle-output-path /tmp/godel-jupyter-t7lsy12n/query.json\n",
      "2023-12-06 07:42:39,655 INFO: Task /tmp/godel-jupyter-t7lsy12n/query.gdl is success, result is NOT-EMPTY, execution time is  10.41s.\n",
      "2023-12-06 07:42:39,656 INFO: run success\n",
      "\n",
      "Total results: 942\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>func</th>\n",
       "      <th>cmplx</th>\n",
       "      <th>sl</th>\n",
       "      <th>el</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>association.go</td>\n",
       "      <td>Delete</td>\n",
       "      <td>20</td>\n",
       "      <td>197</td>\n",
       "      <td>361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>association.go</td>\n",
       "      <td>Unscoped</td>\n",
       "      <td>1</td>\n",
       "      <td>44</td>\n",
       "      <td>51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>association.go</td>\n",
       "      <td>Find</td>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>association.go</td>\n",
       "      <td>Association</td>\n",
       "      <td>4</td>\n",
       "      <td>21</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>association.go</td>\n",
       "      <td>Count</td>\n",
       "      <td>2</td>\n",
       "      <td>367</td>\n",
       "      <td>372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>937</th>\n",
       "      <td>tests/preload_test.go</td>\n",
       "      <td>TestPreloadEmptyData</td>\n",
       "      <td>5</td>\n",
       "      <td>215</td>\n",
       "      <td>235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>938</th>\n",
       "      <td>tests/preload_test.go</td>\n",
       "      <td>TestPreloadGoroutine</td>\n",
       "      <td>2</td>\n",
       "      <td>237</td>\n",
       "      <td>253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>939</th>\n",
       "      <td>tests/preload_test.go</td>\n",
       "      <td>TestPreloadWithDiffModel</td>\n",
       "      <td>2</td>\n",
       "      <td>255</td>\n",
       "      <td>271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>940</th>\n",
       "      <td>tests/preload_test.go</td>\n",
       "      <td>TestNestedPreloadWithUnscoped</td>\n",
       "      <td>4</td>\n",
       "      <td>273</td>\n",
       "      <td>308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>941</th>\n",
       "      <td>tests/preload_test.go</td>\n",
       "      <td>TestEmbedPreload</td>\n",
       "      <td>2</td>\n",
       "      <td>310</td>\n",
       "      <td>445</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%db ./db/gorm\n",
    "// script\n",
    "use coref::go::*\n",
    "\n",
    "fn default_db() -> GoDB {\n",
    "    return GoDB::load(\"coref_go_src.db\")\n",
    "}\n",
    "\n",
    "/**\n",
    " * @param name: 文件名\n",
    " * @param func: 函数名\n",
    " * @param cmplx: 函数圈复杂度\n",
    " * @param sl,el,sc,ec: 函数位置信息，依次为函数起始行，结束行\n",
    " */\n",
    "fn out(name: string, func: string, cmplx: int, sl: int, el: int) -> bool {\n",
    "    for(f in GoFile(default_db()), function in Function(default_db())) {\n",
    "        if ((!f.isAutoGenereatedFile()) &&\n",
    "            f.key_eq(function.getBelongsFile()) &&\n",
    "            name = f.getName() &&\n",
    "            func = function.getName() &&\n",
    "            cmplx = function.getCyclomaticComplexity() &&\n",
    "            sl = function.getLocation().getStartLineNumber() &&\n",
    "            el = function.getLocation().getEndLineNumber()) {\n",
    "            return true\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "fn main() {\n",
    "    output(out())\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "保存上一次运行的 query 结果保存到一个JSON文件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query result saved to /workspaces/CodeFuse-Query/tutorial/notebook/query.json\n"
     ]
    }
   ],
   "source": [
    "%%save_to ./query.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "STEP 3: 好了，你可以针对分析生成的结果，进行进一步的代码分析了，比如你可以结合pandas库，使用刚刚生成的 query.json 实现最大函数复杂度Top 10的排序查询："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-11-23T03:54:56.998681Z",
     "start_time": "2023-11-23T03:54:56.976694Z"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                      name                       func  cmplx   sl   el\n",
      "428        schema/field.go                 ParseField    110   99  436\n",
      "18     callbacks/update.go       ConvertToAssignments     69  131  304\n",
      "309       schema/schema.go  ParseWithSpecialTableName     68  121  370\n",
      "130    callbacks/create.go      ConvertToCreateValues     64  217  385\n",
      "234                scan.go                       Scan     62  125  342\n",
      "30            statement.go             BuildCondition     61  284  465\n",
      "429        schema/field.go       setupValuerAndSetter     46  439  969\n",
      "729  tests/migrate_test.go         TestMigrateColumns     44  517  645\n",
      "11      callbacks/query.go              BuildQuerySQL     41   33  266\n",
      "281   migrator/migrator.go              MigrateColumn     41  441  542\n"
     ]
    }
   ],
   "source": [
    "%%python\n",
    "import pandas as pd\n",
    "data = pd.read_json('./query.json')\n",
    "data.sort_values('cmplx', ascending=False, inplace=True)\n",
    "top_10 = data.head(10)\n",
    "print(top_10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Enjoy！"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Godel kernel",
   "language": "rust",
   "name": "godel-jupyter"
  },
  "language_info": {
   "file_extension": ".gdl",
   "help_links": [
    {
     "text": "Godel kernel Magics",
     "url": "https://sparrow.alipay.com"
    }
   ],
   "mimetype": "text/rust",
   "name": "rust",
   "version": "0.0.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}