To download the movenet_lightning_f16 neural network from Tensorflow, run :
wget -q -O movenet.tflite https://tfhub.dev/google/lite-model/movenet/singlepose/lightning/tflite/float16/4?lite-format=tflite
Note that multiple variants can be found to https://www.tensorflow.org/hub/tutorials/movenet if you want a different trade-off between precision and inference speed.
To get tennis videos, you can simply download them from any youtube converter, e.g. https://en1.onlinevideoconverter.pro/11/
Movenet human pose estimation
To make your annotation, you can use the annotator.py file, e.g
$ python annotator.py dataset/nadal/nadal.mp4
which will output a csv file, named annotation_nadal.csv containing something like this:
Shot,FrameId
serve,257
forehand,294
backhand,329
forehand,374
forehand,415
backhand,450
where each line corresponds to a shot at a specified frame.
$ python extract_shots_as_features.py dataset/nadal/nadal.mp4 annotation_nadal.csv shots/ --show
You might need to create the shots/ directory before.
This will extract human poses from the video, and record them as tennis shots thanks to the previous annotation. You will capture backhands, forehands, serves and neutral. Neutral is not a shot but this is a crucial thing to be able to detect that the player is not currently hitting the ball when we will do the training/inference.
We consider that a tennis shot lasts 30 frames (~1 second).
$ ls shots/
backhand_001.csv forehand_001.csv forehand_002.csv forehand_003.csv forehand_004.csv neutral_001.csv neutral_002.csv neutral_003.csv neutral_004.csv
$ cat shots/forehand_001.csv
nose_y,nose_x,left_shoulder_y,left_shoulder_x,right_shoulder_y,right_shoulder_x,left_elbow_y,left_elbow_x,right_elbow_y,right_elbow_x,left_wrist_y,left_wrist_x,right_wrist_y,right_wrist_x,left_hip_y,left_hip_x,right_hip_y,right_hip_x,left_knee_y,left_knee_x,right_knee_y,right_knee_x,left_ankle_y,left_ankle_x,right_ankle_y,right_ankle_x,shot
0.162207,0.6012776,0.21824716,0.54470015,0.20131558,0.47970787,0.29950634,0.63309443,0.24733938,0.43873972,0.21634884,0.6599996,0.20992465,0.5716669,0.46811688,0.41779175,0.46534866,0.41446403,0.6377884,0.42666733,0.65294665,0.5091464,0.82741517,0.34416345,0.88804334,0.44243777,forehand
0.11285974,0.59998506,0.20404208,0.55788124,0.16721159,0.4570608,0.2858228,0.6353053,0.22715563,0.41733202,0.19021843,0.6887855,0.1892693,0.57292336,0.44883227,0.40839133,0.4449601,0.39647135,0.6343681,0.43327686,0.6405092,0.49266624,0.83009136,0.32465157,0.88660365,0.43830386,forehand
0.18672426,0.5594809,0.23171018,0.523953,0.21895005,0.43091992,0.29559258,0.61327755,0.25287876,0.5207798,0.22299193,0.6759945,0.21183936,0.5540659,0.46609396,0.40087023,0.4604697,0.38435063,0.63056093,0.39763328,0.6450049,0.481231,0.82833445,0.32043666,0.88719803,0.43460035,forehand
0.122355275,0.5629583,0.18595593,0.5040643,0.18126898,0.4215086,0.2520818,0.6160585,0.22247656,0.3967939,0.20359102,0.68833303,0.19821078,0.5250137,0.43757656,0.38526922,0.4362464,0.36810756,0.6144553,0.3815326,0.62870055,0.4712695,0.82959497,0.3016157,0.88982064,0.43042964,forehand
0.15612131,0.56811064,0.20406301,0.5271659,0.1897412,0.42148355,0.26531354,0.6210118,0.2472065,0.48218417,0.22588617,0.7004272,0.2364195,0.5609894,0.44792235,0.38700035,0.44255504,0.36812404,0.6186205,0.38358772,0.62739605,0.47094434,0.83019346,0.3146945,0.882129,0.43377566,forehand
0.15578315,0.5565451,0.20175111,0.5018753,0.19719487,0.40738672,0.2516997,0.5919501,0.2534607,0.4977278,0.23936568,0.6850488,0.23811159,0.5775441,0.4431228,0.3646868,0.44162205,0.352458,0.6207051,0.36826694,0.6283074,0.46201554,0.83522564,0.30127928,0.8858893,0.4271728,forehand
0.14873266,0.5621277,0.17849365,0.51233685,0.18800208,0.40776357,0.24231735,0.6156765,0.25448534,0.41536257,0.22609894,0.6979466,0.24268046,0.5459729,0.42966476,0.3692935,0.42637375,0.34841412,0.61549723,0.37143,0.62061346,0.46910533,0.8359508,0.30034524,0.8807858,0.43032792,forehand
0.12997259,0.5664855,0.17004256,0.5173207,0.19318573,0.41867685,0.23417976,0.61116785,0.28315043,0.445243,0.24077037,0.70852345,0.31008887,0.53605074,0.42414507,0.36229688,0.42469478,0.3399179,0.61522204,0.36888158,0.62607384,0.46483827,0.83934456,0.30602366,0.8878883,0.43199226,forehand
0.10537767,0.56108826,0.17601924,0.5023125,0.191344,0.41378558,0.2273116,0.579749,0.29920375,0.4306217,0.22087473,0.7082849,0.3494972,0.5449365,0.4255584,0.34314293,0.42950112,0.32858106,0.6203235,0.3560501,0.6319325,0.44651416,0.83847004,0.30349055,0.8858669,0.42307374,forehand
0.16222504,0.54625624,0.20033462,0.47103462,0.2281443,0.41840762,0.23881578,0.5565716,0.35972512,0.445435,0.22646071,0.68761384,0.3997776,0.548746,0.44396904,0.3300644,0.45124656,0.33838513,0.6339686,0.33965522,0.64963615,0.4412527,0.8393668,0.3030287,0.8856738,0.41851878,forehand
0.13632877,0.54586107,0.16717008,0.42215812,0.20609526,0.4503647,0.31252617,0.4541881,0.37438887,0.48436615,0.18901862,0.60591567,0.41250837,0.59278476,0.43206906,0.29706722,0.44154367,0.3518083,0.62492985,0.3378265,0.64021146,0.43990603,0.83652055,0.30154347,0.88716775,0.42188057,forehand
0.123052165,0.5687044,0.15792535,0.42875302,0.2210781,0.516862,0.2844538,0.51203716,0.35936075,0.5535483,0.3665406,0.588864,0.3888768,0.61536026,0.43750554,0.36060083,0.45120245,0.43236256,0.62274235,0.38027024,0.63875204,0.50256574,0.83054894,0.35333607,0.8730721,0.47187904,forehand
0.1313534,0.55945617,0.17354995,0.44143113,0.25493783,0.561159,0.31856343,0.47561768,0.3811879,0.61699957,0.33190602,0.5921471,0.4122482,0.71234584,0.4491526,0.37141114,0.4630425,0.4585234,0.63813156,0.3920718,0.65501887,0.5097074,0.84080786,0.37062332,0.89155185,0.495416,forehand
0.121629976,0.51105255,0.17542179,0.3746134,0.24170336,0.52589625,0.31145847,0.36193207,0.3503183,0.5787143,0.26731518,0.50980604,0.36962909,0.72513485,0.4371491,0.33746395,0.4514583,0.43784356,0.6275245,0.35791036,0.65191245,0.4828551,0.8361564,0.34165606,0.87946296,0.46578094,forehand
0.14191693,0.47350967,0.1958375,0.35192865,0.24752453,0.49331152,0.32929885,0.3205982,0.34589502,0.53451735,0.2609776,0.4533733,0.28410387,0.5141061,0.46000233,0.31666544,0.4724379,0.42065448,0.639074,0.3288596,0.66999096,0.45918855,0.8515998,0.31277528,0.8855833,0.44097936,forehand
0.11332863,0.52512544,0.19596393,0.40992993,0.23828283,0.57073164,0.34211025,0.3743957,0.30849192,0.6170265,0.32831448,0.4312694,0.2528241,0.6436677,0.45438927,0.39788714,0.46120733,0.5066512,0.63019556,0.39824465,0.6579548,0.53447855,0.85127616,0.39115512,0.8790374,0.5250485,forehand
0.12933628,0.5151641,0.20207709,0.40946776,0.24406736,0.57646227,0.30992508,0.3673021,0.30761972,0.59117824,0.28502947,0.4490541,0.280934,0.5159697,0.46357372,0.39875573,0.4715815,0.51015264,0.6563021,0.3917818,0.6752563,0.5341938,0.86140937,0.39091176,0.8775115,0.5239768,forehand
0.117655344,0.5299771,0.19176333,0.44129393,0.22841677,0.599087,0.2785872,0.36511168,0.29973295,0.56939435,0.20717171,0.42022356,0.23083383,0.479621,0.46055368,0.4306441,0.47035632,0.5399396,0.6567126,0.41054007,0.681158,0.56665456,0.87256765,0.40769127,0.8853823,0.5557887,forehand
0.13280487,0.54744923,0.20804316,0.46037716,0.23059413,0.611563,0.30143163,0.38280836,0.3021171,0.59004545,0.22413328,0.44101655,0.24057071,0.51146376,0.47077146,0.45685127,0.47833744,0.56158835,0.6663076,0.43124703,0.6850996,0.5815654,0.87454396,0.43226427,0.87565506,0.58870953,forehand
0.11962753,0.54240835,0.20131546,0.4695021,0.21714498,0.6092738,0.3066715,0.37429598,0.29945976,0.56655043,0.20335497,0.42121205,0.20161949,0.4649276,0.46678254,0.46749127,0.46776482,0.5707978,0.67952824,0.4401246,0.6835979,0.592365,0.8805767,0.44437715,0.88169533,0.59868985,forehand
0.12789832,0.55127954,0.21347162,0.48475313,0.22069171,0.6234296,0.32261714,0.3975584,0.30513966,0.56833273,0.2132519,0.40522665,0.21793887,0.46661428,0.4771195,0.48214015,0.47867426,0.583399,0.6929398,0.45430294,0.69239223,0.60769916,0.88471085,0.45614374,0.88280916,0.6186236,forehand
0.13211174,0.5283826,0.21462001,0.49171364,0.22254492,0.6100712,0.323471,0.39783502,0.31807095,0.5674504,0.21908379,0.3754104,0.22647189,0.4490074,0.47811544,0.48215723,0.47726604,0.57887185,0.6930829,0.45137346,0.6925592,0.6033112,0.89061207,0.45417792,0.8792889,0.62763834,forehand
0.12730542,0.52605003,0.21441358,0.49554554,0.21553579,0.61051977,0.3313136,0.40502122,0.3089007,0.5559311,0.21263632,0.37119916,0.22711737,0.45185816,0.476293,0.4830446,0.47075894,0.58015406,0.68582493,0.45357087,0.68043995,0.61140954,0.88420194,0.45470294,0.8661247,0.64065796,forehand
0.13907279,0.5192146,0.2220459,0.49224636,0.22221202,0.60962015,0.33397073,0.40366513,0.3168744,0.5508518,0.22854869,0.36428615,0.23188305,0.45360583,0.4808035,0.48267245,0.47561276,0.5743089,0.68510085,0.45092988,0.68310636,0.6087561,0.88395345,0.45477325,0.870935,0.65014386,forehand
0.13044491,0.51391304,0.21544524,0.49006945,0.21925555,0.6081438,0.3399014,0.40109918,0.3199339,0.55098075,0.22557105,0.35375586,0.23332529,0.4569447,0.48395765,0.48813555,0.48164523,0.57841146,0.6946267,0.4431994,0.69409096,0.61248404,0.8896532,0.45154086,0.87917036,0.6551453,forehand
0.1399526,0.50999,0.22023128,0.4885532,0.22353378,0.61061275,0.3425629,0.40016556,0.32230347,0.54783195,0.23246041,0.34256977,0.23037408,0.46040943,0.4874474,0.49044216,0.48488757,0.5792735,0.69872683,0.43988112,0.6922801,0.61311316,0.887391,0.44767004,0.8850891,0.6616773,forehand
0.13926607,0.51253325,0.22139907,0.49186283,0.22271205,0.6164401,0.34039086,0.41488537,0.32183906,0.55259866,0.22566272,0.3673197,0.22595322,0.462297,0.4877644,0.4924857,0.48322004,0.58525604,0.69098043,0.44098923,0.69156176,0.6172858,0.8818898,0.4491933,0.87827003,0.6792749,forehand
0.13151215,0.50184685,0.21903923,0.47833472,0.21883199,0.59715325,0.340546,0.4002956,0.32312274,0.54590374,0.2246443,0.36028475,0.2236349,0.4384642,0.48507276,0.47846657,0.48199463,0.57085687,0.6885189,0.42096385,0.69228876,0.60491055,0.89279157,0.42542508,0.88523793,0.66767585,forehand
0.13379067,0.49248594,0.2208226,0.4784049,0.22206064,0.5878116,0.34517848,0.39959538,0.33899796,0.54213655,0.22647646,0.36144915,0.22740869,0.4300209,0.48806074,0.4771488,0.48363167,0.5660192,0.6845876,0.41580927,0.68914336,0.5943135,0.8838772,0.4181738,0.8767559,0.6575803,forehand
0.13606904,0.48398906,0.22113709,0.47489744,0.2250625,0.59259063,0.35737422,0.40498376,0.34462595,0.54788846,0.23919058,0.36728457,0.24672493,0.42578894,0.48894736,0.48176247,0.48369023,0.5692743,0.68760705,0.41633543,0.6937238,0.5923523,0.8902998,0.4050037,0.88128424,0.6583273,forehand
You can visualize your results by running:
python visualize_features.py shots/forehand_001.csv
Forehand example
Backhand example
Neutral/Idle example
Serve example
See SingleFrameShotClassifier.ipynb
In the notebook, we load our annotated datasets (csv files containing 1 second shot) with the position of each key points of the player pose. Each sample is here a set of features from a single frame (instantaneous). Possible classes are :
- backhand
- forehand
- neutral (or idle)
- serve