Twitch metrics (manifoldmarkets#1366)

* Manifold user data now cached within bot rather than using update hooks. This reduces CPU overhead when many remote documents are updated simultaneously.

* Changed yarn command for local web server to "dev". Added local debugger support for NodeJS.

* Moved timing utilities to utils.ts.

* Corrected new contract ante.

* Bot Firestore user data is now cached in the bot. This reduces connection latency for the dock and overlay, and reduces the delay when placing a bet.

* Only users declared as "admin: true" in the bot Firebase can use the group controls UI in the dock.

* Updated engine.io to fix security issue.

* Hackathon project: dashboard metrics prototype.

* Fixed Twitch bot building and deployment using common manifold files. Converted Twitch sub-repository to CJS. Imports now have consistent and unambiguous resolutions due to using import aliases.

* Server-side metrics working and connected to Firestore.

* Metrics UI working with backend to serve latest real data. Improved icons for percentage difference to previous day when handing edge cases.

* Improved metrics behaviour with sparse data. Fixed styling issue with dock "Create new question" popup.

* Added colors to logging to make errors and warnings clearer.

* Removed legacy windows scripts. Made unix scripts the default.

* Fixed auto-refresh not working for dock and overlay after migrating server build to esbuild.

* Normalized manifold envs. Removed broken "use latest git" option from deploy script. Fixed manifold logo overflowing overlay page.

* Added Twitch bot debug configuration to VSCode config.

* Attempt at fixing deployment script when running through Git Bash on Windows. Also updated README with latest instructions for debugging, building and deploying as well as viewing the logs and metrics of the deployed bot.

* Fixed bug with colors being sent to gcloud logger.

.vscode/launch.json

@@ -11,6 +11,12 @@
       "port": 9222, // chrome needs to be started with the parameter "--remote-debugging-port=9222"
       "urlFilter": "http://localhost:3000/*",
       "webRoot": "${workspaceFolder}/web"
+    },
+    {
+      "name": "Debug Twitch bot",
+      "type": "node",
+      "request": "attach",
+      "port": 9229
     }
   ]
 }

twitch-bot/Dockerfile

@@ -11,8 +11,9 @@ COPY twitch-bot/common/package.json common/package.json
 RUN yarn
 COPY twitch-bot/common common
 COPY twitch-bot/web web
+RUN yarn --cwd web build
 COPY twitch-bot/server server
-RUN npx concurrently -n WEB,SERVER "yarn --cwd web build" "yarn --cwd server build" -g
+RUN yarn --cwd server build
 
 # -- Stage 2 -- #
 FROM node:16

twitch-bot/README.md

@@ -2,13 +2,7 @@
 
 This repo has everything required to host the Manifold Twitch Bot, and associated overlay and dock browser sources for OBS.
 
-![OBS example](/docs/OBS.png)
-
-## Live demo
-
-There is a live demo of this project hosted as a [DigitalOcean App](https://www.digitalocean.com/products/app-platform). The dock demo is available [here](https://king-prawn-app-5btyw.ondigitalocean.app/dock) and the overlay is available [here](https://king-prawn-app-5btyw.ondigitalocean.app/overlay).
-
-It is worth noting that since this is a public demo with no associated Manifold account, you will not be able to create or resolve questions. You will, however, be able to search for existing questions and feature them on the overlay.
+![OBS example](./docs/OBS.png)
 
 ## Environmental variables
 
@@ -33,22 +27,53 @@ The following environmental variables are available but optional:
 
 These can either be defined as global environmental variables on the system, or as a `.env` file in the root of the repository.
 
-## Getting started
+## Starting development
 
 - Ensure the [environmental variables](#environmental-variables) are correctly configured
 - Ensure [Yarn](https://classic.yarnpkg.com/lang/en/docs/install/#windows-stable) is installed
-- `$ yarn`
-- `$ yarn dev:fullstack`
+- Run the following commands from the root of the Manifold repository:
+  - `$ yarn`
+  - `$ cd twitch-bot`
+  - `$ yarn`
+  - `$ yarn dev:fullstack`
+
+The server automatically enables debugging in development mode. If using VSCode, you can attach to the process by pressing F5 and launching the `Debug Twitch bot` session.
+
+## Quick deployment to prod or dev servers
+
+In order to deploy to the Manifold Twitch servers, you will need to have the appropriate permissions on your Google account, and the following software installed:
+
+- [NodeJS](https://nodejs.org/en/download/current/) 16 or higher
+- [Yarn](https://classic.yarnpkg.com/lang/en/docs/install/#windows-stable)
+- [gcloud CLI](https://cloud.google.com/sdk/docs/install) (must be initialized with `gcloud auth login`)
+- [Docker](https://docs.docker.com/get-docker/)
+
+If you are using Windows, it is also recommended to have [Git Bash](https://git-scm.com/downloads) installed, as the deployment script is targeted for use on Unix OSs.
 
-## Deploying
+Launch `twitch-bot/scripts/deploy-to-remote.sh`. On Windows, this must be done through Git Bash. The script will ask whether you wish to deploy to the development or production server, and should then handle everything else.
 
-This repo can be built into a Docker image for deployment to a hosting site. The container host must have all the [environmental variables](#environmental-variables) set.
+The first time this script is run it will need to download docker images, Yarn dependencies and build all the source from scratch, so be patient! Subsequent runs should only take a matter of seconds to complete.
+
+## Alternative deployment
+
+This repo can be built into a Docker image ready for deployment to a hosting site as-is. The container host must have all the [environmental variables](#environmental-variables) set for this to work.
 
 The Docker image can be built with `docker build -t {IMAGE_NAME} .` in the root of the repository, and run with `docker run --env-file .env -p 9172:9172 -it {IMAGE_NAME}`
 
+## Viewing logs in production
+
+There are two ways to view the logs of the deployed bot:
+
+1.  Go to https://console.cloud.google.com/logs/query?project=mantic-markets and select either `bot-DEV` or `bot-PROD` under `Log name`.
+2.  SSH into the remote server using `gcloud compute ssh dev-twitch-bot` OR `gcloud compute ssh twitch-bot` and run `docker logs $(docker ps -q) -n 100`
+
+## Viewing usage metrics
+
+- Development: https://dev-twitch-bot.manifold.markets/metrics
+- Production: https://twitch-bot.manifold.markets/metrics
+
 ## Future development
 
-- [ ] Integrate access to Manifold's Firestore fully to decrease latency when loading markets, detecting new bets and detecting market resolution
 - [ ] Port the overlay to a [Twitch Extension](https://www.twitch.tv/p/en/extensions/) to decrease viewer latency when viewing bets
 - [ ] Support market types other than binary
 - [ ] Rate limit management in the Twitch bot to prioritize outgoing messages when there is a risk of Twitch dropping them

twitch-bot/common/types/metric-types.ts

@@ -0,0 +1,12 @@
+export type MetricDay = {
+  uniqueUserFeatures: number; // Users that have featured a market today
+  featuredQuestions: number; // Total number of features questions today
+  newBots: number; // Number of users that have added the bot to their channel for the first time
+  twitchLinks: number; // Number of users that have linked their Manifold account to Twitch
+  commandsUsed: number; // Times a Twitch bot command has been used
+  activeUsers: number; // Users that have used a command today
+};
+
+export function getCurrentEpochDay() {
+  return (Date.now() / 8.64e7) >> 0;
+}

twitch-bot/scripts/build-docker.sh

@@ -0,0 +1,8 @@
+cd ../..
+start=`date +%s`
+docker build -f twitch-bot/Dockerfile -t mb .
+end=`date +%s`
+echo "================================="
+echo "= Build completed in $((end-start)) seconds ="
+echo "================================="
+sleep 3

twitch-bot/scripts/deploy-to-remote.sh

@@ -0,0 +1,105 @@
+#!/bin/sh
+
+# This script is a quick way to deploy a docker image to the remote GCloud server.
+# It works by building the default repository Docker image first, extracting the build
+# artifacts, zipping them up along with a mini Dockerfile, copying them to the server,
+# and finally building the runtime image there.
+
+# ============
+#    CONFIG   
+# ============
+ROUTE_TO_SCRIPTS_DIR=.
+PROJECT=mantic-markets
+
+DEV_INSTANCE=dev-twitch-bot
+DEV_BUILD_DIR=build/dev
+DEV_ZONE=europe-west2-c
+
+PROD_INSTANCE=twitch-bot
+PROD_BUILD_DIR=build/prod
+PROD_ZONE=us-central1-a
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+# ============
+#    SCRIPT   
+# ============
+error () {
+	echo An error occuring during the deployment process.
+	exit 1
+}
+
+pushd () {
+    command pushd "$@" > /dev/null
+}
+
+popd () {
+    command popd "$@" > /dev/null
+}
+
+gcloud() {
+	command node "$SCRIPT_DIR/gcloud.mjs" "$@"
+}
+
+build() {
+	pushd ../..
+	echo Building code...
+	docker rmi mb
+	docker build -f twitch-bot/Dockerfile -t mb . || error
+	popd
+
+	pushd $BUILD_DIR
+	echo Preparing files...
+	CONTAINER_ID=$(docker create mb)
+	docker cp $CONTAINER_ID:deploy/. out/
+	docker rm $CONTAINER_ID
+	popd
+
+	cp $BUILD_DIR/.env $BUILD_DIR/out/
+	cp Dockerfile $BUILD_DIR/out/
+
+	pushd $BUILD_DIR
+	echo Copying files to server...
+	tar -czf out.tar.gz out
+	rm -r out
+	gcloud compute scp --recurse --zone $ZONE out.tar.gz Phil@$INSTANCE_NAME:. || error
+	rm out.tar.gz
+
+	COMMAND="tar -zxf out.tar.gz out && \
+	rm out.tar.gz && \
+	echo Rebuilding docker image... && \
+	docker build -t bot out && \
+	echo Launching docker image... && \
+	docker run -d --env-file=out/.env --restart on-failure --network=host bot && \
+	echo Cleaning up... && \
+	rm -r out && \
+	docker system prune -a -f"
+
+	gcloud compute ssh --zone $ZONE $INSTANCE_NAME --command "$COMMAND" || error
+
+	exit 0
+}
+
+init_dev () {
+	echo Deploying to DEV
+	INSTANCE_NAME=$DEV_INSTANCE
+	BUILD_DIR=$DEV_BUILD_DIR
+	ZONE=$DEV_ZONE
+	build
+}
+
+init_prod () {
+	echo Deploying to PROD
+	INSTANCE_NAME=$PROD_INSTANCE
+	BUILD_DIR=$PROD_BUILD_DIR
+	ZONE=$PROD_ZONE
+	build
+}
+
+cd "$SCRIPT_DIR"
+read -p "Which server do you want to deploy to [DEV/prod]? " de
+case $de in
+	prod|p ) init_prod; break;;
+	dev|d|"" ) init_dev; break;;
+	* ) echo "Invalid option entered. Exiting..."; error; break;;
+esac

twitch-bot/scripts/gcloud.mjs

@@ -0,0 +1,13 @@
+import { spawnSync } from 'child_process';
+
+const gcloudArgs = process.argv.splice(2);
+const newArgs = [];
+for (let arg of gcloudArgs) {
+  arg = arg.replaceAll(/[\n\r\t]/g, '');
+  if (arg.includes(' ')) {
+    newArgs.push(`"${arg}"`);
+  } else {
+    newArgs.push(arg);
+  }
+}
+spawnSync('gcloud', newArgs, { shell: true, stdio: 'inherit' });

twitch-bot/scripts/launch-docker.sh

@@ -0,0 +1,2 @@
+cd ..
+docker run -it --env-file=.env -p 9172:9172 mb

twitch-bot/server/build.mjs

@@ -81,7 +81,7 @@ function cleanupPreviousProc() {
 }
 
 function startProc() {
-  const newProc = spawn('node', [config.WAIT_FOR_DEBUGGER.value ? '--inspect-brk' : '--inspect', config.BUILD_FILE.value], { stdio: 'inherit' });
+  const newProc = spawn('node', [config.WAIT_FOR_DEBUGGER.value ? '--inspect-brk' : '--inspect', '--enable-source-maps', config.BUILD_FILE.value], { stdio: 'inherit' });
   currentlyRunningProc = newProc;
 }
 
@@ -112,6 +112,9 @@ esbuild
     ignoreAnnotations: true,
     treeShaking: true,
     logLevel: config.BUILD_ONLY.value ? 'info' : 'silent',
+    define: {
+      'process.env.__BUILD_ID__': JSON.stringify(new Date().toISOString()),
+    },
     watch: !config.BUILD_ONLY.value && {
       onRebuild: (error) => {
         cleanupPreviousProc();