examples/Magics_and_rpy2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "892997dd-bbd9-4fea-a9a2-88288beff542",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "**To view the examples in this notebook as intended, you need both R and Python language servers up and running.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc05e0ae-c6bf-42d3-88c3-e1505285aa2d",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b78a3fd6-494b-4551-84d8-1bdada1c09fd",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "Magics support is implemented with kernel-specific definitions. The idea is to:\n",
    "- extract the code inside of magics to another virtual document if they contain foreign code (e.g. code in another language or code which is a standalone script), or\n",
    "- mask the existence of other magics from the LSP server so that it does not complain of invalid syntax\n",
    "\n",
    "By default, a small number of rules for extractors and overrides is provided, showcased below on the example of IPython kernel and rpy2 integration. The user will be able to specify custom overrides and extractors in the settings.\n",
    "\n",
    "#### Technical notes\n",
    "\n",
    "The extraction is performed by implementations of `IForeignCodeExtractor` extractors, with `RegExpForeignCodeExtractor` provided as a simple regular expression-based implementation for the default rules. These rules and this extractor are not designed to be comprehensive, but rather to provide a nice initial experience to the majority of users. More advanced use-cases will require a custom implementation of the `IForeignCodeExtractor` interface.\n",
    "\n",
    "With custom implementation of extractors it is easily possible to extract HTML code from within Python (or any other language) strings or CSS from within HTML for analysis by the LSP server, thus the implementation of extractors is not limited for the use with cell or line magics.\n",
    "\n",
    "The overrides are implemented with regular expressions defined in objects obeying `IMagicOverride` interface.\n",
    "\n",
    "In the future the \"included batteries\" may be moved out to separate extensions i.e. *jupyterlab-lsp-ipython* for default IPython magics support and *jupyterlab-lsp-rpy2* for rpy2 support."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "652b41ab-0ec1-4767-928e-702f2315ce51",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "## Magics support for IPython kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c24369f3-339e-4167-b85e-c6599a9d0798",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "### Shell assignments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d48f659-318b-4db1-99dd-ffd375aefc0c",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "Shell assignments are masked by default so that no \"Invalid syntax\" message is shown (regardless of the linter used):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "981bc665-1993-42f8-a61d-7e9502de4c07",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "!ls\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee4c21f4-6d9c-4ab0-849c-befd3f7dce94",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "### `%%python` magic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44a1a7a1-345c-4428-b31a-bd09fce526dc",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "from statistics import mean\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dde13506-267d-4421-b106-714dbf6e2025",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "x = mean([1, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "165f1c62-477d-4152-88eb-a2a47f062e8b",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "The content of script magics with aliases defining their language is moved to separate virtual document for LSP linting, completion and all other features; this is then transparently mapped to the top level virtual document and then back to the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc35f5a3-648b-4c93-a46d-accf3adcbddc",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "%%python\n",
    "# x is not defined in this scope, should be underlined\n",
    "x\n",
    "y = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ef72ad2-085a-4142-bc2a-2120f1b3c5d1",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "# y is not defined in this scope\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f66e7bfb-aba3-458a-8a25-dc1f3450b6bf",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "Consecutive calls to the `%%python` magic will result in separate namespaces. In the extension codebase this is referred as the magic being *standalone*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3288f463-ed66-402a-9e1c-2127280436c1",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "%%python\n",
    "# y is not defined here (separate namespace from the fist %%python magic)\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d626105-e406-4908-bdc8-d186ce0cde38",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "Following IPython magics are treated as standalone:\n",
    "- `%%python`, `%%python2`, `%%python3`, `%%pypy` - use *python* server if available\n",
    "- `%%perl` - uses *perl* server if available\n",
    "- `%%ruby` - uses *ruby* server if available\n",
    "- `%%html --isolated` - uses *html* server if available\n",
    "- `%%sh` - uses *sh* server if available\n",
    "\n",
    "Following IPython magics are treated as parts of a continious virtual document (not standalone):\n",
    "- `%%js`, `%%javascript` - use *javascript* server if available\n",
    "- `%%html` (when without `--isolated` switch) - uses *html* server if available\n",
    "- `%%latex` - uses *latex* server if available\n",
    "- `%%markdown` - uses *markdown* server if available"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac54cc45-36f1-49e1-8cc6-00227360779b",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "### `%%timeit` and all other built-in cell magics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "509cf809-91a5-4907-8163-306f96b6ed97",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "`%%timeit` reads from the global namespace but it does not write to it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "787f9d95-5150-4c71-bc69-c375ef452cef",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "# note: x is accessible from global namespace and hence not underlied her:\n",
    "y = x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "effd3c4a-a2b2-4f46-8bb2-6712f7e32c18",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "024b580f-6e28-4fac-b206-6d04f73e0b7b",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "Contrary to the `%%python` magic, the `%%timeit` is masked out for the LSP server rather than moved to a separate virtual document. This means that the above example seems to work perfectly only because no linting is active in `%%timeit` magic:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65b84737-0d21-4c1e-a420-9adc0cbd1d3a",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "this_is_undefined_but_wont_be_highlighted"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c690888d-2cc3-49e5-9d05-6a2b5fac9417",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "The user can change this behaviour in settings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35c902ba-0603-4357-a99b-5dbf112a1a1d",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "### Line and cell magic names can be presented to the LSP server as function calls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d0e0140-f0b7-4b19-9416-61c4c0eb7a18",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "By default the name of the magic, will be kept and presented in the LSP document, to prevent a false positive of \"defined but unused\" on magic imports or magic function/class definitions. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f09f7110-2368-41c9-9d48-8a9529b1d3fe",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "from jupyter_helpers.selective_import import skip_on_import\n",
    "from jupyter_helpers.selective_import import skip_all_below\n",
    "\n",
    "%skip_all_below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aa9c7b6-0ab4-470f-a7a9-1a65a87a7936",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "`%skip_all_below` was transformed into `skip_all_below()` with the built-in regular expression, thus preventing the LSP linter from wrongly falgging the import as unused."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8b080ba-26f6-41e8-b5c6-dcd53b5c15d0",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "All of this will be made customizable to the user by the modification of regular expressions set in the settings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1d073c1-c27d-4cab-823b-d77982d21d31",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "### Line and cell magics are not chekced"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34ad94fb-29bf-4f4f-afb8-8d25043c48a3",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "We *could* restrict the matches of the default override to only known IPython magics in the potential future jupyterlab-lsp-ipython client extension, allowing the user to disable this feature, but as for now the names are not validated in any way."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f3fd158-398b-406d-bff4-89063d6864cc",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "## rpy2 intergation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e86e4ed-98be-43c7-97eb-c48bccd32711",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "Both cell and line magics are linted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebe2b93c-a81f-4434-952f-3fea812c7e0c",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "# notice the undeline after 1 (\"Trailing whitespaces is superfluous\")\n",
    "%R test = 1     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5eb0da3a-eb55-4c3b-8703-dfe3d8d1725c",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "%%R\n",
    "# again, some trailing spaces added for visibility\n",
    "test2 = data.frame()    \n",
    "x = test\n",
    "x = t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "709ed9df-ac14-4875-94e0-d68c7d6d33ed",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "%%R\n",
    "# try <tab> after labell - it should complete to ggplot2::labeller,\n",
    "# and then, offer you a signature suggestion upon '(' insertion.\n",
    "ggplot2::labell"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43d00554-4fe0-4c43-b657-dd7740f68d2f",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "source": [
    "Line magics and the \"host\" (here Python code) can be interspersed within a single cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1b2b271-8d81-4a25-ac87-ac37ba3efc77",
   "metadata": {
    "libroFormatter": "formatter-string"
   },
   "outputs": [],
   "source": [
    "undefined_python_variable\n",
    "%R invalid R syntax\n",
    "valid_python = 'syntax'\n",
    "%R valid_r = c('syntax')\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}