ESIM Web App Playground

A Web App for Generating Events from Upsampled Video Frames & Visualization of Events from Live Video of a Webcam!

Siddhant Sahu | Daniel Gehrig | Nico Messikommer

Table of Contents

1. About The Project
2. Getting Started
   • Prerequisites & Installation
   • Running the Web App Locally
   • Running the Web App on Google Colab
3. Roadmap

About The Project

The goal of this project is to make ESIM and VID2E available to researchers who do not possess a real event camera. For this we want to deploy VID2E and ESIM as an interactive web app. The app should be easy to use and have the following functional requirements:

• Generation of events from video through dragging and dropping a video into the browser. The resulting events should be downloadable as raw events and rendered video.
• Incorporate video interpolation via an existing video interpolation method (e.g. Super SloMo) before event generation with ESIM.
• Visualization and inspection of the event stream in the browser.
• The event generation should be configurable by changing ESIM parameters.

(back to top)

Getting Started

Follow these steps in order to make this app run in your local system.

This code base assumes you have access to a NVIDIA GPU in your system with proper drivers installed along with the cuda-toolkit version 10.1.

Prerequisites & Installation

Make sure that you have followed the Instalation with Anaconda Instruction and have created the vid2e Conda environment with the esim_torch package installed in this environment.

1. Activate the Conda environment

conda activate vid2e
conda list | grep 'esim'

The above command should output the following

esim-cuda                 0.0.0                    pypi_0    pypi
esim-py                   0.0.1                    pypi_0    pypi

2. Install the streamlit package in this Conda environment, along with a couple of additional packages.

pip install streamlit stqdm numba h5py

Running the Web App Locally

If you want to run this app locally on your system then follow these steps.

1. Run the App

streamlit run web_app.py

2. Then you can access the app in Local URL: http://localhost:8501

When using the 'video upload' feature, the uploaded file is saved in the 'data/original/video_upload' directory. Subsequently, upon clicking the 'Generate Events' button, two new directory, namely, 'data/upsampled/video_upload' and 'data/events/video_upload' are created. Additionally, after the process has finished, you will see a 'Download Events' button, you can click this button to download the events. Events can be downloaded in three different formats: as .h5 file, as .npz or as a rendered video.

(back to top)

Running the Web App on Google Colab

To run this web app on Google Colab follow these steps in order.

Make sure you select the Run time type as GPU in Google Colab

1. Register and create an account in the remote.it.
2. Install remote.it service in your Google Colab instance.

!curl -LkO https://raw.githubusercontent.com/remoteit/installer/master/scripts/auto-install.sh
! chmod +x ./auto-install.sh
! sudo ./auto-install.sh

3. Install Miniconda in your Google Colab Instance

! wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
! chmod +x Miniconda3-py37_4.8.2-Linux-x86_64.sh
! bash ./Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -f -p /usr/local

4. Update Python Path

import sys
sys.path.insert(0,'/usr/local/lib/python3.7/site-packages/')

5. Install Streamlit package

!pip install streamlit

6. Run the remote.it service

!sudo connectd_installer

• In the menu selection choose 1
• Enter your remote.it email address and password
• Enter the name of your device esim
• Choose 1 Attach/reinstall a remote.it Service to an application
• Protocol Selection Menu, Choose 2, Web (HTTP) on port 80
• Enter a name for this remote.it service, esim
• Main menu, choose, 5 Exit

7. Mount your Google Drive to this Google Colab Instance
8. Change directory to where to want this project to reside.
9. Clone the repo

   git clone https://github.com/uzh-rpg/rpg_vid2e.git

10. Check if the esim_torch package is installed

import esim
print(esim_torch.__path__)

11. Built the esim_torch package with pybind11

pip install esim_torch/

12. Change directory inside web_app and run the app on port number 80 as a webservice which can then be accessed for the remote.it service.

!streamlit run --server.port 80 web_app.py&>/dev/null&

Finaly, you can navigate your remote.it dashboard and open these service and use the app!

Note: Remeber when running this app on Google Colab you need to change the paths for input and output directories for the upsampling to the specific directory in Google Drive and also other paths and esim import.

(back to top)

Roadmap

• Add video upload dragging and dropping a video into the browser.
• Add video interpolation via an existing video interpolation method (e.g. Super SloMo) before event generation with ESIM.
• Add events download button (output format)
  • HDF5
  • NPZ
  • Rendered Video
• Add webcam functionality
• Run this app on Google Colab