Hole in the Wall game

A browser game reminiscent of Hole in the Wall gameshow powered by Tensorflow Pose Estimation library (aka Posenet).

Try out the demo! or watch my demo

This project is a submission for Hack&Roll 2020 which wins The Most Entertaining Hack!

Getting Started

Installing

npm install

Serving the repository on localhost

You can use any tool that does this.

• Since node js is presumably installed, you may run npm install http-server -g to install http-server globally, then run http-server -p 8080 in the project directory
• Alternatively, if you have python 3 installed, you can run python -m http.server 8080
• Alternatively, if you have ruby intalled, you can run ruby -run -e httpd . -p 8080

Once it is running, you can just go to http://localhost:8080/ and try it out. It should request for access to the camera/webcam to which you can just accept it. Note that starting up will most likely take a few seconds to load the video, depending on your GPU.

The pose estimation is all running locally, so there is no further communication to any server after loading of the necessary scripts. You can check out the network tab in your browser devtools, so your video feed is not being uploaded at all.

Make sure only 1 person is in the camera view when it is running, as I am using single pose detection for the sake of the game (I only support single player).

index.html

As this is a hackathon, I coded most of the application in this monolithic file. Most of the code are adapted from posenet's demo camera.js, demo_utils.js and analysing the actual demo.

Here I will briefly explain what the code does.

1. Selectively include functions dependencies from demo_utils.js which are specified here.
2. Include samplePoses.js. This specifies an array of predefined poses which are manually captured by logging the poses and stopping the requestAnimationFrame loop. The poses are made by myself, Bing Huang and Zainul.
3. Load camera/webcam feed into the video source to display what it sees.
4. Initialize posenet and run the poseDetectionFrame in a requestAnimationFrame loop. The game logic is all in the poseDetectionFrame function.
5. poseDetectionFrame will draw 2 overlays on a canvas: the wall with the expected pose zooming in, and the detected a single pose from the camera feed.
6. When the wall is zoomed in 100%, the expected pose is compared with the detected pose using a pose similarity algo library to decide whether to score or fail.
7. When the player fails 3 times, the game stops. Reload the browser to replay.

Challenges I faced

Tensorflow js pose estimation does not have a API documentation. So I had to manually analyse and experiment with the demo source code.

I was stucked for some time on combining HTML5 canvas context scaling and translations to create the zoom in effect originating from the middle. The canvas context's origin is at the top left corner, so scaling is relative to the top left corner and will shift the wall away/towards the corner. So some maths is involved to solve this issue.

Built With

• Tensorflow Pose Estimation - Generates coordinates of a pose skeleton
• PosenetSimilarity - Calculates similarity between 2 poses
• HTML5 canvas - Drawing skeletons and bounding boxes overlay

Contributing

I will be honest, this project is littered with bad practices but hey its from a hackathon :p So if you can understand my code and want to contribute or fork it out, feel free to do so! If you do, please let me know as I will be interested in what you will do with it.

Authors

• David Choo - Quick 24 hour hack

License

This project is licensed under the MIT License - see the LICENSE file for details

Acknowledgments

• Another year of amazing Hack&Roll 2020 for giving me an opportunity to learn something new over the weekend.