10-PureHDF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PureHDF\n",
    "\n",
    "__[PureHDF](https://github.com/Apollo3zehn/PureHDF)__ is a .NET library implemented in C# for reading and writing HDF5 files. It is a pure .NET implementation of the HDF5 file format, and does not rely on any native libraries.  In other words, it is an independent (of the HDF5 library) implementation of an API and the HDF5 file format specification. It is based on the __[HDF5 File Format Specification Version 3.0](https://docs.hdfgroup.org/hdf5/develop/_f_m_t3.html)__.\n",
    "\n",
    "<a href=\"https://en.wikipedia.org/wiki/.NET\"><img src=\"img/Microsoft_.NET_logo.png\" title=\".NET logo\" /></a>\n",
    "\n",
    "C# code can be run in Jupyter notebooks using the __[dotnet-interactive](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.dotnet-interactive-vscode), aka `ms-dotnettools.dotnet-interactive-vscode`__ VS Code extension. Before installing the extension, you must install the .NET SDK 8.0 on your system.  The following instructions are for Debian 11, which      is the OS used for this container.\n",
    "\n",
    "## Install the .NET SDK 8.0 on Debian 11\n",
    "\n",
    "Run the following commands in a terminal window: \n",
    "\n",
    "```bash\n",
    "wget https://packages.microsoft.com/config/debian/11/packages-microsoft-prod.deb -O packages-microsoft-prod.deb\n",
    "sudo dpkg -i packages-microsoft-prod.deb\n",
    "rm packages-microsoft-prod.deb\n",
    "sudo apt-get update && sudo apt-get install -y dotnet-sdk-8.0\n",
    "```\n",
    "\n",
    "## Install the VS Code extension\n",
    "\n",
    "__[Polyglot Notebooks](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.dotnet-interactive-vscode)__, aka `ms-dotnettools.dotnet-interactive-vscode`\n",
    "\n",
    "Finally install the __[PureHDF NuGet package](https://www.nuget.org/packages/PureHDF/)__  .\n",
    "\n",
    "## Install the PureHDF NuGet package\n",
    "\n",
    "Just execute the following command and the PureHDF NuGet package will be installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "#r \"nuget: PureHDF, 2.1.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to use PureHDF in a Jupyter notebook.\n",
    "\n",
    "## Let's go!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "using PureHDF;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the generation of sample paths, we need  random numbers drawn from a normal distribution.  We could use the __[MathNet.Numerics](https://numerics.mathdotnet.com/)__ library for this purpose. However, we will just roll our own for the sake of simplicity. \n",
    "\n",
    "The code to generate the sample paths is almost identical to all the examples we have seen so far. Once that's out of the way, the conciseness of the C# code for writing the HDF5 entities is striking."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "ulong path_count = 100, step_count = 1000;\n",
    "double dt = 0.01, theta = 1.0, mu = 0.0, sigma = 0.1;\n",
    "\n",
    "// How do C# programmers get through life without the normal distribution?\n",
    "\n",
    "var normal = (Random random, double mean, double stddev) =>\n",
    "{\n",
    "    double x1 = 1 - random.NextDouble();\n",
    "    double x2 = 1 - random.NextDouble();\n",
    "    double y1 = Math.Sqrt(-2.0 * Math.Log(x1)) * Math.Cos(2.0 * Math.PI * x2);\n",
    "    return y1 * stddev + mean;\n",
    "};\n",
    "\n",
    "var random = new Random();\n",
    "\n",
    "var data = new double[path_count, step_count];\n",
    "\n",
    "for (ulong i = 0; i < path_count; i++)\n",
    "{\n",
    "    data[i, 0] = 0.0d;\n",
    "    for (ulong j = 1; j < step_count; j++)\n",
    "    {\n",
    "        double dW = normal(random, 0.0, Math.Sqrt(dt));\n",
    "        data[i, j] = data[i, j - 1] + theta * (mu - data[i, j - 1]) * dt + sigma * dW;\n",
    "    }\n",
    "}\n",
    "\n",
    "var dataset = new H5Dataset(data, fileDims: [path_count, step_count]);\n",
    "\n",
    "dataset.Attributes = new()\n",
    "{\n",
    "    [\"comment\"] = \"This dataset contains sample paths of an Ornstein-Uhlenbeck process.\",\n",
    "    [\"Wikipedia\"] = \"https://en.wikipedia.org/wiki/Ornstein%E2%80%93Uhlenbeck_process\",\n",
    "    [\"rows\"] = \"path\",\n",
    "    [\"columns\"] = \"time\",\n",
    "    [\"dt\"] = dt,\n",
    "    [\"θ\"] = theta,\n",
    "    [\"μ\"] = mu,\n",
    "    [\"σ\"] = sigma\n",
    "};\n",
    "\n",
    "var file = new H5File()\n",
    "{\n",
    "    Attributes = new()\n",
    "    {\n",
    "        [\"source\"] = \"https://github.com/HDFGroup/hdf5-tutorial\"\n",
    "    },\n",
    "\n",
    "    [\"dataset\"] = dataset\n",
    "};\n",
    "\n",
    "file.Write(\"ou_purehdf.h5\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's visit an old friend\n",
    "\n",
    "In October 2023, I hosted an HDF clinic [PureHDF: HDF5 on a free, cross-platform, open-source developer platform](https://www.youtube.com/watch?v=-v8790jQ5xk). Let's revisit the code from that clinic!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var file = new H5File()\n",
    "{\n",
    "    Attributes = new()\n",
    "    {\n",
    "        [\"attr1\"] = new string(\"string attribute\")\n",
    "    },\n",
    "\n",
    "    [\"dset1\"] = new int[,]\n",
    "    {\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 },\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 },\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 },\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 },\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 },\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 },\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 },\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 },\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 },\n",
    "        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }\n",
    "    },\n",
    "\n",
    "    [\"dset2\"] = new []\n",
    "    {\n",
    "    new {a=1, b=0.1F, c=0.01D},\n",
    "    new {a=2, b=0.2F, c=0.02D},\n",
    "    new {a=3, b=0.3F, c=0.03D},\n",
    "    new {a=4, b=0.4F, c=0.04D},\n",
    "    new {a=5, b=0.5F, c=0.05D}\n",
    "    },\n",
    "\n",
    "    [\"dset3\"] = new int[][]\n",
    "    {\n",
    "        new int[] {0},\n",
    "        new int[] {10, 11},\n",
    "        new int[] {20, 21, 22},\n",
    "        new int[] {30, 31, 32, 33}\n",
    "    },\n",
    "\n",
    "    // Soft links are supported!\n",
    "    [\"slink1\"] = new H5SoftLink(\"somevalue\")\n",
    "\n",
    "    // Q: How do I create a comitted datatype?\n",
    "    // A: Currently unsupported.\n",
    "};\n",
    "\n",
    "file[\"group1\"] = new H5Group()\n",
    "{\n",
    "    Attributes = new()\n",
    "    {\n",
    "        [\"numerical-attribute\"] = new double[] { 2.0, 3.1, 4.2 },\n",
    "        [\"string-attribute\"] = new string[] { \"One\", \"Two\", \"Three\" }\n",
    "    },\n",
    "\n",
    "    [\"dset3\"] = new []\n",
    "    {\n",
    "    new {a=1, b=0.1F, c=0.01D, d=new {a=\"string_1\"}},\n",
    "    new {a=2, b=0.2F, c=0.02D, d=new {a=\"string_2\"}},\n",
    "    new {a=3, b=0.3F, c=0.03D, d=new {a=\"string_3\"}},\n",
    "    new {a=4, b=0.4F, c=0.04D, d=new {a=\"string_4\"}},\n",
    "    new {a=5, b=0.5F, c=0.05D, d=new {a=\"string_5\"}}\n",
    "    }\n",
    "};\n",
    "\n",
    "file[\"group2\"] = file[\"group1\"];\n",
    "\n",
    "file.Write(\"./example.h5\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good news! The code just works, and soft links are now supported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "PureHDF is an independent implementation of HDF5 in C#/.NET. This is no small achievement and offers a fresh perspective on what’s possible with modern languages and platforms. If you thought that .NET was a Windows-only Microsoft thing, then this is your chance to experience a modern, free, cross-platform, open-source developer platform for building many different types of applications, and HDF5 is a first-class citizen."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "polyglot-notebook"
  },
  "polyglot_notebook": {
   "kernelInfo": {
    "defaultKernelName": "csharp",
    "items": [
     {
      "aliases": [],
      "name": "csharp"
     }
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}