Skip to content

Latest commit

 

History

History
168 lines (142 loc) · 9.16 KB

DEV.md

File metadata and controls

168 lines (142 loc) · 9.16 KB

DISCO developer guide

build status build status


Welcome to the DISCO developer guide. Here you will have a first overview of the project, how to install and run an instance of DISCO and links to further documentation.

Structure

The DISCO project is composed of multiple parts. At the root level, there are four main folders: discojs, server, web-client and cli.

  • discojs, or Disco.js, is the JavaScript library that contains federated and decentralized learning logic. The library allows to train and use machine learning models in a distributed fashion. The library itself is composed of the disco-node and disco-web modules, both of them extending the platform-agnostic code in disco-core. In other words, disco-core contains most of the implementation but can't be used by itself, while disco-web and disco-node allow using disco-core via different technologies. To some extents, you can think of disco-core as an abstract class extended by disco-weband disco-node.

    • disco-node lets you use Disco.js with Node.js. For example, the server and the cli rely on disco-node. A user can also directly import the disco-node package in their Node.js programs.
    • disco-web allows using Disco.js through a browser. The web-client, discussed below, relies on disco-web to implement a browser UI.

    The main difference between the two is how they handle storage: a browser doesn't have access to the file system (for security reasons) while a Node.js application does.

  • server contains the server implementation necessary to use Disco.js. Indeed, while the federated and decentralized learning logic is implemented by Disco.js, we still need a server to orchestrate users in both paradigms. In decentralized learning, the server exposes an API for users to query the necessary information to train models in a decentralized fashion, such as the list of other peers. Thus, the server never receives training data or model parameters. In federated learning, the server receives model updates but never training data. It keeps track of participants and updates the model weights. A server instance is always necessary to use DISCO, whether one is using a browser UI, the CLI or directly programming with disco-node.

  • web-client implements a browser User Interface. In other words, it implements a website allowing users to use DISCO without coding. Via the browser, a user can create and participate in federated and decentralized training sessions, evaluate models, etc.

  • cli contains the Command Line Interface for Disco.js. For example, the CLI allows a user to create and join training sessions from the command line, benchmark performance by emulating multiple clients, etc.

Here is a summary diagram:

flowchart LR
  subgraph discojs
    discojs-node-->|extends|discojs-core;
    discojs-web-->|extends|discojs-core;
  end
  subgraph User Interface
    web-client-->|uses|discojs-web;
    custom_browser["custom browser implementation"]-->|uses|discojs-web;
    server-->| uses |discojs-node;
    cli-->|uses|discojs-node;
    custom_node["custom Node.js scripts"]-->|uses|discojs-node;
  end
Loading

Installation guide

The following instructions will install the required dependencies, build Disco.js and launch a DISCO server and a web client. If you run into any sort of trouble check our FAQ; otherwise please create a new issue or feel free to ask on our slack.

  1. We recommend using nvm (Node Version Manager) to handle multiple Node.js versions. Start by installing nvm by following their installation instructions. After installation, you should be able to run
nvm -v
0.39.7 # my nvm version at the time
  1. Install Node.js version 16
nvm install 16

You can now choose which Node.js version to use:

nvm use 16

Using Node.js v16 should automatically set your npm (Node Package Manager, different from nvm) version to 8:

npm --version
8.xx.xx

nvm manages your different Node.js versions while npm handles your different Node.js project packages within one version.

  1. Clone the repository
git clone [email protected]:epfml/disco.git
cd disco
  1. Run the installation script
sh install.sh
What does install.sh do?
The installation script installs the dependencies required by the different parts of the project, which are described in the Structure section. It first installs the Disco.js library dependencies, notably, `TensorFlow.js`, and anything else required for federated and decentralized learning logic. The script then builds the library, a step necessary to compile TypeScript into JavaScript. 
cd discojs
npm ci # stands for `clean install`, to ensure than only expected dependencies are being installed.
npm run build

The script then installs dependencies for the web client, which implements a browser UI. By default, the project points to the @epfml/disco-web package published on the npm remote repository. In a development environment, we want to use the local web client in the discojs/web-client folder. To do so, we need to link the local folder as the actual dependency.

cd ../web-client
npm ci
npm link ../discojs/discojs-web

You can verify than the link is effective by checking that npm ls lists @epfml/[email protected] -> ./../discojs/discojs-web.

Similarly, we install the server dependencies, and then the discojs-node dependency to the local folder rather than the remote npm package @epfml/disco-node:

cd ../server
npm ci
npm link ../discojs/discojs-node

Install the CLI dependencies:

cd ../cli
npm ci

Finally, we install ts-node globally in order to compile and run TypeScript code in a single command from anywhere.

npm install -g ts-node
  1. Launch DISCO

As you may have seen, there are many ways to use DISCO. Here we will run a server and a web client. From there, a user can use DISCO from their browser.

  • First launch a server instance, which is used for federated and decentralized learning tasks, e.g. to list peers participating in a decentralized task.
cd server
npm run dev

The server should be listening on http://localhost:8080/.

  • Secondly, start a web client, which will allow you to use DISCO from your browser. You may have to do so from another terminal since the previous one is now used by the server.
cd web-client
npm run dev

The web client should be running on http://localhost:8081, if not first restart the server and then the web client.

Important

Make sure to first start the server to ensure that it is listening to port 8080.

You can now access DISCO at http://localhost:8081/

Further documentation

  • Next you may want to read our onboarding guide which lists the following steps to onbaord DISCO. Additionally the Architecture guide gives more information on our use of TypeScript and Vue.js, the frontend framework.
  • If you are only planning to use DISCO in your own scripts, you can find a stand-alone example relying on discojs-node here. The example runs with Node.js outside any browser, using the @epfml/discojs-node NPM package and the server module. A DISCO server is launched by the script itself and the data is already available in the repo.

Table of contents

As there are many guides in the project, here is a table of contents referencing them all: