Scrcpy is a tool that streams your android screen to your computer, and allows you to send mouse or keyboard input back to the android device. This project modifies scrcpy to also send back automatically generated input by passing the frame on to the OpenCV computer vision library. To see scrcpy-opencv in action, check out this Youtube video!
Install dependencies to build scrcpy:
runtime dependencies
sudo apt install ffmpeg libsdl2-2.0-0 adb
client build dependencies
sudo apt install gcc git pkg-config meson ninja-build libavcodec-dev libavformat-dev libavutil-dev libsdl2-dev
server build dependencies
sudo apt install openjdk-8-jdk
Some more dependencies you might not have installed:
sudo apt install g++ cmake libswscale-dev
Clone this repository:
git clone https://github.com/RobbertH/scrcpy-opencv.git
Go into the folder you just cloned:
cd scrcpy-opencv
Save the directory of the prebuilt server in an environment variable, we'll need this later:
PREBUILT_SERVER_PATH=$PWD/prebuilt
Change directory to the desktop part of the project, called scrcpy,
cd scrcpy
then run the following command, making sure $PREBUILT_SERVER_PATH contains the prebuilt scrcpy server:
meson x --buildtype release --strip -Db_lto=true -Dprebuilt_server=$PREBUILT_SERVER_PATH/scrcpy-server-v1.12.1
to configure the build, and then
ninja -Cx
to build the application, so you can finally
./run x
to run the modified scrcpy.
Also make sure your android phone is plugged in over USB and has adb enabled.
If the meson step above fails because of OpenCV, you can install it as follows. It's a slight variation on the OpenCV tutorial.
wget https://github.com/opencv/opencv/archive/4.4.0.zip
unzip 4.4.0.zip
cd opencv-4.4.0
mkdir build
cd build
cmake -D OPENCV_GENERATE_PKGCONFIG=ON -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
make -j3
sudo make install
pkg-config --cflags --libs opencv4
sudo ldconfig -v to refresh dynamic linker run-time bindings
After this, meson should find opencv4 and you're good to go.
When making changes, make sure to run meson reconfigure and rebuild using ninja -Cx. Alternatively, you could delete the x directory and run the commands above again.
The files that this repository changes versus the original scrcpy repository are the following:
- added:
app/src/opencv_injection.cppThis is where all new code is written. It is recommended that you alter this file. - added:
app/src/opencv_injection.hpp - modified:
app/src/scrcpy.cDefines a method that can send a tap to the phone (used in opencv_injection.cpp) - modified:
app/src/scrcpy.h - modified:
app/src/screen.cThis is where the 'hook' is that sends the AVFrame to the opencv_injection function - modified:
app/meson.buildto add dependencies and source files to be compiled - modified:
meson.buildto enable C++ compilation instead of only C
I wanted to win a soccer game on android, where you have to tap a ball to keep it in the air. To do so, the 'AVFrame' used in scrcpy is converted to an OpenCV 'mat' so that OpenCV's image processing functions can be run on it. Then, a circle the size of the football is extracted, using "HoughCircles". Lastly, an input tap is sent back to the android device, in the center of that circle.
The main modifications to scrcpy include:
- Modifying the meson config files to compile both scrcpy (C) and the additional functions (C++) that depend on OpenCV (C++).
- Converting the scrcpy image format to one that OpenCV can understand.
- Writing the additional functions to use OpenCV and send input to the android device.
Since most work done was part of "making it work" rather than writing the actual code, I figured this might help other people in achieving similar goals. The concrete case of the soccer game is just one example of what can be achieved with this powerful tool: any OpenCV function can be run to produce input to the android device, based on the frames.
If you want to read more, there is a blog post. For questions, simply open an issue.