To download the movenet_lightning_f16 neural network from Tensorflow, run :
wget -q -O movenet.tflite https://tfhub.dev/google/lite-model/movenet/singlepose/lightning/tflite/float16/4?lite-format=tflite
Note that multiple variants can be found to https://www.tensorflow.org/hub/tutorials/movenet if you want a different trade-off between precision and inference speed.
To get tennis videos, you can simply download them from any youtube converter, e.g. https://en1.onlinevideoconverter.pro/11/
Movenet human pose estimation
To make your annotation, you can use the annotator.py file, e.g
$ python annotator.py dataset/nadal/nadal.mp4
which will output a csv file, named annotation_nadal.csv containing something like this:
Shot,FrameId
serve,257
forehand,294
backhand,329
forehand,374
forehand,415
backhand,450
where each line corresponds to a shot at a specified frame.
$ python extract_shots_as_features.py dataset/nadal/nadal.mp4 annotation_nadal.csv shots/ --show
You might need to create the shots/ directory before.
This will extract human poses from the video, and record them as tennis shots thanks to the previous annotation. You will capture backhands, forehands, serves and neutral. Neutral is not a shot but this is a crucial thing to be able to detect that the player is not currently hitting the ball when we will do the training/inference.
We consider that a tennis shot lasts 30 frames (~1 second).
$ ls shots/
backhand_001.csv forehand_001.csv forehand_002.csv forehand_003.csv forehand_004.csv neutral_001.csv neutral_002.csv neutral_003.csv neutral_004.csv
$ cat shots/forehand_001.csv
nose_y,nose_x,left_shoulder_y,left_shoulder_x,right_shoulder_y,right_shoulder_x,left_elbow_y,left_elbow_x,right_elbow_y,right_elbow_x,left_wrist_y,left_wrist_x,right_wrist_y,right_wrist_x,left_hip_y,left_hip_x,right_hip_y,right_hip_x,left_knee_y,left_knee_x,right_knee_y,right_knee_x,left_ankle_y,left_ankle_x,right_ankle_y,right_ankle_x,shot
0.162207,0.6012776,0.21824716,0.54470015,0.20131558,0.47970787,0.29950634,0.63309443,0.24733938,0.43873972,0.21634884,0.6599996,0.20992465,0.5716669,0.46811688,0.41779175,0.46534866,0.41446403,0.6377884,0.42666733,0.65294665,0.5091464,0.82741517,0.34416345,0.88804334,0.44243777,forehand
0.11285974,0.59998506,0.20404208,0.55788124,0.16721159,0.4570608,0.2858228,0.6353053,0.22715563,0.41733202,0.19021843,0.6887855,0.1892693,0.57292336,0.44883227,0.40839133,0.4449601,0.39647135,0.6343681,0.43327686,0.6405092,0.49266624,0.83009136,0.32465157,0.88660365,0.43830386,forehand
0.18672426,0.5594809,0.23171018,0.523953,0.21895005,0.43091992,0.29559258,0.61327755,0.25287876,0.5207798,0.22299193,0.6759945,0.21183936,0.5540659,0.46609396,0.40087023,0.4604697,0.38435063,0.63056093,0.39763328,0.6450049,0.481231,0.82833445,0.32043666,0.88719803,0.43460035,forehand
0.122355275,0.5629583,0.18595593,0.5040643,0.18126898,0.4215086,0.2520818,0.6160585,0.22247656,0.3967939,0.20359102,0.68833303,0.19821078,0.5250137,0.43757656,0.38526922,0.4362464,0.36810756,0.6144553,0.3815326,0.62870055,0.4712695,0.82959497,0.3016157,0.88982064,0.43042964,forehand
0.15612131,0.56811064,0.20406301,0.5271659,0.1897412,0.42148355,0.26531354,0.6210118,0.2472065,0.48218417,0.22588617,0.7004272,0.2364195,0.5609894,0.44792235,0.38700035,0.44255504,0.36812404,0.6186205,0.38358772,0.62739605,0.47094434,0.83019346,0.3146945,0.882129,0.43377566,forehand
0.15578315,0.5565451,0.20175111,0.5018753,0.19719487,0.40738672,0.2516997,0.5919501,0.2534607,0.4977278,0.23936568,0.6850488,0.23811159,0.5775441,0.4431228,0.3646868,0.44162205,0.352458,0.6207051,0.36826694,0.6283074,0.46201554,0.83522564,0.30127928,0.8858893,0.4271728,forehand
0.14873266,0.5621277,0.17849365,0.51233685,0.18800208,0.40776357,0.24231735,0.6156765,0.25448534,0.41536257,0.22609894,0.6979466,0.24268046,0.5459729,0.42966476,0.3692935,0.42637375,0.34841412,0.61549723,0.37143,0.62061346,0.46910533,0.8359508,0.30034524,0.8807858,0.43032792,forehand
0.12997259,0.5664855,0.17004256,0.5173207,0.19318573,0.41867685,0.23417976,0.61116785,0.28315043,0.445243,0.24077037,0.70852345,0.31008887,0.53605074,0.42414507,0.36229688,0.42469478,0.3399179,0.61522204,0.36888158,0.62607384,0.46483827,0.83934456,0.30602366,0.8878883,0.43199226,forehand
0.10537767,0.56108826,0.17601924,0.5023125,0.191344,0.41378558,0.2273116,0.579749,0.29920375,0.4306217,0.22087473,0.7082849,0.3494972,0.5449365,0.4255584,0.34314293,0.42950112,0.32858106,0.6203235,0.3560501,0.6319325,0.44651416,0.83847004,0.30349055,0.8858669,0.42307374,forehand
0.16222504,0.54625624,0.20033462,0.47103462,0.2281443,0.41840762,0.23881578,0.5565716,0.35972512,0.445435,0.22646071,0.68761384,0.3997776,0.548746,0.44396904,0.3300644,0.45124656,0.33838513,0.6339686,0.33965522,0.64963615,0.4412527,0.8393668,0.3030287,0.8856738,0.41851878,forehand
0.13632877,0.54586107,0.16717008,0.42215812,0.20609526,0.4503647,0.31252617,0.4541881,0.37438887,0.48436615,0.18901862,0.60591567,0.41250837,0.59278476,0.43206906,0.29706722,0.44154367,0.3518083,0.62492985,0.3378265,0.64021146,0.43990603,0.83652055,0.30154347,0.88716775,0.42188057,forehand
0.123052165,0.5687044,0.15792535,0.42875302,0.2210781,0.516862,0.2844538,0.51203716,0.35936075,0.5535483,0.3665406,0.588864,0.3888768,0.61536026,0.43750554,0.36060083,0.45120245,0.43236256,0.62274235,0.38027024,0.63875204,0.50256574,0.83054894,0.35333607,0.8730721,0.47187904,forehand
0.1313534,0.55945617,0.17354995,0.44143113,0.25493783,0.561159,0.31856343,0.47561768,0.3811879,0.61699957,0.33190602,0.5921471,0.4122482,0.71234584,0.4491526,0.37141114,0.4630425,0.4585234,0.63813156,0.3920718,0.65501887,0.5097074,0.84080786,0.37062332,0.89155185,0.495416,forehand
0.121629976,0.51105255,0.17542179,0.3746134,0.24170336,0.52589625,0.31145847,0.36193207,0.3503183,0.5787143,0.26731518,0.50980604,0.36962909,0.72513485,0.4371491,0.33746395,0.4514583,0.43784356,0.6275245,0.35791036,0.65191245,0.4828551,0.8361564,0.34165606,0.87946296,0.46578094,forehand
0.14191693,0.47350967,0.1958375,0.35192865,0.24752453,0.49331152,0.32929885,0.3205982,0.34589502,0.53451735,0.2609776,0.4533733,0.28410387,0.5141061,0.46000233,0.31666544,0.4724379,0.42065448,0.639074,0.3288596,0.66999096,0.45918855,0.8515998,0.31277528,0.8855833,0.44097936,forehand
0.11332863,0.52512544,0.19596393,0.40992993,0.23828283,0.57073164,0.34211025,0.3743957,0.30849192,0.6170265,0.32831448,0.4312694,0.2528241,0.6436677,0.45438927,0.39788714,0.46120733,0.5066512,0.63019556,0.39824465,0.6579548,0.53447855,0.85127616,0.39115512,0.8790374,0.5250485,forehand
0.12933628,0.5151641,0.20207709,0.40946776,0.24406736,0.57646227,0.30992508,0.3673021,0.30761972,0.59117824,0.28502947,0.4490541,0.280934,0.5159697,0.46357372,0.39875573,0.4715815,0.51015264,0.6563021,0.3917818,0.6752563,0.5341938,0.86140937,0.39091176,0.8775115,0.5239768,forehand
0.117655344,0.5299771,0.19176333,0.44129393,0.22841677,0.599087,0.2785872,0.36511168,0.29973295,0.56939435,0.20717171,0.42022356,0.23083383,0.479621,0.46055368,0.4306441,0.47035632,0.5399396,0.6567126,0.41054007,0.681158,0.56665456,0.87256765,0.40769127,0.8853823,0.5557887,forehand
0.13280487,0.54744923,0.20804316,0.46037716,0.23059413,0.611563,0.30143163,0.38280836,0.3021171,0.59004545,0.22413328,0.44101655,0.24057071,0.51146376,0.47077146,0.45685127,0.47833744,0.56158835,0.6663076,0.43124703,0.6850996,0.5815654,0.87454396,0.43226427,0.87565506,0.58870953,forehand
0.11962753,0.54240835,0.20131546,0.4695021,0.21714498,0.6092738,0.3066715,0.37429598,0.29945976,0.56655043,0.20335497,0.42121205,0.20161949,0.4649276,0.46678254,0.46749127,0.46776482,0.5707978,0.67952824,0.4401246,0.6835979,0.592365,0.8805767,0.44437715,0.88169533,0.59868985,forehand
0.12789832,0.55127954,0.21347162,0.48475313,0.22069171,0.6234296,0.32261714,0.3975584,0.30513966,0.56833273,0.2132519,0.40522665,0.21793887,0.46661428,0.4771195,0.48214015,0.47867426,0.583399,0.6929398,0.45430294,0.69239223,0.60769916,0.88471085,0.45614374,0.88280916,0.6186236,forehand
0.13211174,0.5283826,0.21462001,0.49171364,0.22254492,0.6100712,0.323471,0.39783502,0.31807095,0.5674504,0.21908379,0.3754104,0.22647189,0.4490074,0.47811544,0.48215723,0.47726604,0.57887185,0.6930829,0.45137346,0.6925592,0.6033112,0.89061207,0.45417792,0.8792889,0.62763834,forehand
0.12730542,0.52605003,0.21441358,0.49554554,0.21553579,0.61051977,0.3313136,0.40502122,0.3089007,0.5559311,0.21263632,0.37119916,0.22711737,0.45185816,0.476293,0.4830446,0.47075894,0.58015406,0.68582493,0.45357087,0.68043995,0.61140954,0.88420194,0.45470294,0.8661247,0.64065796,forehand
0.13907279,0.5192146,0.2220459,0.49224636,0.22221202,0.60962015,0.33397073,0.40366513,0.3168744,0.5508518,0.22854869,0.36428615,0.23188305,0.45360583,0.4808035,0.48267245,0.47561276,0.5743089,0.68510085,0.45092988,0.68310636,0.6087561,0.88395345,0.45477325,0.870935,0.65014386,forehand
0.13044491,0.51391304,0.21544524,0.49006945,0.21925555,0.6081438,0.3399014,0.40109918,0.3199339,0.55098075,0.22557105,0.35375586,0.23332529,0.4569447,0.48395765,0.48813555,0.48164523,0.57841146,0.6946267,0.4431994,0.69409096,0.61248404,0.8896532,0.45154086,0.87917036,0.6551453,forehand
0.1399526,0.50999,0.22023128,0.4885532,0.22353378,0.61061275,0.3425629,0.40016556,0.32230347,0.54783195,0.23246041,0.34256977,0.23037408,0.46040943,0.4874474,0.49044216,0.48488757,0.5792735,0.69872683,0.43988112,0.6922801,0.61311316,0.887391,0.44767004,0.8850891,0.6616773,forehand
0.13926607,0.51253325,0.22139907,0.49186283,0.22271205,0.6164401,0.34039086,0.41488537,0.32183906,0.55259866,0.22566272,0.3673197,0.22595322,0.462297,0.4877644,0.4924857,0.48322004,0.58525604,0.69098043,0.44098923,0.69156176,0.6172858,0.8818898,0.4491933,0.87827003,0.6792749,forehand
0.13151215,0.50184685,0.21903923,0.47833472,0.21883199,0.59715325,0.340546,0.4002956,0.32312274,0.54590374,0.2246443,0.36028475,0.2236349,0.4384642,0.48507276,0.47846657,0.48199463,0.57085687,0.6885189,0.42096385,0.69228876,0.60491055,0.89279157,0.42542508,0.88523793,0.66767585,forehand
0.13379067,0.49248594,0.2208226,0.4784049,0.22206064,0.5878116,0.34517848,0.39959538,0.33899796,0.54213655,0.22647646,0.36144915,0.22740869,0.4300209,0.48806074,0.4771488,0.48363167,0.5660192,0.6845876,0.41580927,0.68914336,0.5943135,0.8838772,0.4181738,0.8767559,0.6575803,forehand
0.13606904,0.48398906,0.22113709,0.47489744,0.2250625,0.59259063,0.35737422,0.40498376,0.34462595,0.54788846,0.23919058,0.36728457,0.24672493,0.42578894,0.48894736,0.48176247,0.48369023,0.5692743,0.68760705,0.41633543,0.6937238,0.5923523,0.8902998,0.4050037,0.88128424,0.6583273,forehand
You can visualize your results by running:
python visualize_features.py shots/forehand_001.csv
Forehand example
Backhand example
Neutral/Idle example
Serve example
See SingleFrameShotClassifier.ipynb
In the notebook, we load our annotated datasets (csv files containing 1 second shot) with the position of each key points of the player pose. Each sample is here a set of features from a single frame (instantaneous). Possible classes are :
- backhand
- forehand
- neutral (or idle)
- serve
With a fully connected layers, we can reach a validation accuracy of ~80% (see also the confusion matrix).
And we export the neural network to tennis_fully_connected.h5
python track_and_classify_frame_by_frame.py path/to/dimitrov_alcaraz.mp4 tennis_fully_connected.h5
This will read the video of your choice, infer the movenet then feed it to your trained network at each frame. Probabilities of each class are displayed as vertical bars.
Probabilities at each frame
where classes are S(erve), B(ackhand), N(eutral) and F(orehand).
As you can see, classification is very unstable on a single frame.
python track_and_classify_frame_by_frame.py path/to/dimitrov_alcaraz.mp4 tennis_fully_connected.h5
Same priciple than before. But not we do an averaging of the shot probabilities over a sliding window of 10 frames. We add a basic shot counter to be able to detect (and not only classify) shots.
Averaging and shot counter
Proabilities are now smoother, and it s possible to have a decently working shot counter.
See RNNShotClassifier.ipynb
In the notebook, we load our annotated datasets (csv files containing 1 second shot) as a temporal sequence of the human pose. We then use keras GRU recurrent neural network to train it.
I get close to ~100% accuracy.
Probabilities with RNN
It runs faster than real-time on my GPU.