Effnet docs release (DeepLabCut#1074)

Effnet updates

README.md

@@ -196,7 +196,7 @@ VERSION 1.0: The initial, Nature Neuroscience version of [DeepLabCut](https://ww
 
 ## News (and in the news):
 
-- Jan 2021: [Pretraining boosts out-of-domain robustness for pose estimation](https://openaccess.thecvf.com/content/WACV2021/papers/Mathis_Pretraining_Boosts_Out-of-Domain_Robustness_for_Pose_Estimation_WACV_2021_paper.pdf) published in the IEEE Winter Conference on Applications of Computer Vision. We also added EfficientNet backbones to DeepLabCut, those are best trained with cosine decay (see paper). To use them, just pass "efficientnet-b0" to "efficientnet-b6" when creating the trainingset!
+- Jan 2021: [Pretraining boosts out-of-domain robustness for pose estimation](https://openaccess.thecvf.com/content/WACV2021/html/Mathis_Pretraining_Boosts_Out-of-Domain_Robustness_for_Pose_Estimation_WACV_2021_paper.html) published in the IEEE Winter Conference on Applications of Computer Vision. We also added EfficientNet backbones to DeepLabCut, those are best trained with cosine decay (see paper). To use them, just pass "efficientnet-b0" to "efficientnet-b6" when creating the trainingset!
 - Dec 2020: We released a real-time package that allows for online pose estimation and real-time feedback. See [DLClive.deeplabcut.org](http://DLClive.deeplabcut.org).
 - 5/22 2020: We released 2.2beta5. This beta release has some of the features of DeepLabCut 2.2, whose major goal is to integrate multi-animal pose estimation to DeepLabCut.
 - Mar 2020: Inspired by suggestions we heard at this weeks CZI's Essential Open Source Software meeting in Berkeley, CA we updated our [docs](docs/UseOverviewGuide.md). Let us know what you think!

deeplabcut/pose_estimation_tensorflow/evaluate_multianimal.py

@@ -1,5 +1,5 @@
 """
-DeepLabCut2.0 Toolbox (deeplabcut.org)
+DeepLabCut 2.2 Toolbox (deeplabcut.org)
 © A. & M. Mathis Labs
 https://github.com/AlexEMG/DeepLabCut
 

deeplabcut/pose_estimation_tensorflow/nnet/pose_net_efficientnet.py

@@ -1,8 +1,19 @@
 '''
+DeepLabCut 2.1.9 Toolbox (deeplabcut.org)
+© A. & M. Mathis Labs
+https://github.com/AlexEMG/DeepLabCut
+
+Please see AUTHORS for contributors.
+https://github.com/DeepLabCut/DeepLabCut/blob/master/AUTHORS
+Licensed under GNU Lesser General Public License v3.0
+
 Adopted from DeeperCut by Eldar Insafutdinov
 https://github.com/eldar/pose-tensorflow
 
 Effnet added by T. Biasi & AM
+Efficient Nets added by T. Biasi & AM
+See https://openaccess.thecvf.com/content/WACV2021/html/Mathis_Pretraining_Boosts_Out-of-Domain_Robustness_for_Pose_Estimation_WACV_2021_paper.html
+
 '''
 
 import re

deeplabcut/pose_estimation_tensorflow/predict_multianimal.py

@@ -150,13 +150,15 @@ def GetPoseandCostsF(
     det_min_score = dlc_cfg.minconfidence
 
     num_idchannel = dlc_cfg.get("num_idchannel", 0)
+    # TODO Fix the code below...
+    #  We can't just break the whole thing if there is one corrupted frame
+    #  in the middle of the video. Rather iterate over all frames and simply skip corruptions
     while cap.video.isOpened():
         if counter % step == 0:
             pbar.update(step)
         frame = cap.read_frame(crop=cfg["cropping"])
         if frame is not None:
             frames[batch_ind] = img_as_ubyte(frame)
-            inds.append(counter)
             if batch_ind == batchsize - 1:
                 # PredicteData['frame'+str(counter)]=predict.get_detectionswithcosts(frame, dlc_cfg, sess, inputs, outputs, outall=False,nms_radius=dlc_cfg.nmsradius,det_min_score=dlc_cfg.minconfidence)
                 D = predict.get_batchdetectionswithcosts(
@@ -173,14 +175,20 @@ def GetPoseandCostsF(
                     inputs,
                     outputs,
                 )
-                for ind, data in zip(inds, D):
-                    PredicteData["frame" + str(ind).zfill(strwidth)] = data
+                for l in range(batchsize):
+                    # pose = predict.getposeNP(frames,dlc_cfg, sess, inputs, outputs)
+                    # PredicteData[batch_num*batchsize:(batch_num+1)*batchsize, :] = pose
+                    PredicteData[
+                        "frame" + str(batch_num * batchsize + l).zfill(strwidth)
+                    ] = D[l]
+
                 batch_ind = 0
-                inds.clear()
                 batch_num += 1
             else:
                 batch_ind += 1
-        elif counter >= nframes:
+        else:
+            nframes = counter
+            print("Detected frames: ", nframes)
             if batch_ind > 0:
                 # pose = predict.getposeNP(frames, dlc_cfg, sess, inputs, outputs) #process the whole batch (some frames might be from previous batch!)
                 # PredicteData[batch_num*batchsize:batch_num*batchsize+batch_ind, :] = pose[:batch_ind,:]
@@ -199,11 +207,14 @@ def GetPoseandCostsF(
                     outputs,
                     c_engine=c_engine,
                 )
-                for ind, data in zip(inds, D):
-                    PredicteData["frame" + str(ind).zfill(strwidth)] = data
+                for l in range(batch_ind):
+                    # pose = predict.getposeNP(frames,dlc_cfg, sess, inputs, outputs)
+                    # PredicteData[batch_num*batchsize:(batch_num+1)*batchsize, :] = pose
+                    PredicteData[
+                        "frame" + str(batch_num * batchsize + l).zfill(strwidth)
+                    ] = D[l]
             break
         counter += 1
-
     cap.close()
     pbar.close()
     PredicteData["metadata"] = {
@@ -249,7 +260,8 @@ def GetPoseandCostsS(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes, c_engine
                 det_min_score=dlc_cfg.minconfidence,
                 c_engine=c_engine,
             )
-        elif counter >= nframes:
+        else:
+            nframes = counter
             break
         counter += 1
 
@@ -264,4 +276,6 @@ def GetPoseandCostsS(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes, c_engine
         ],
         "nframes": nframes,
     }
+
+    # print(PredicteData)
     return PredicteData, nframes

deeplabcut/pose_estimation_tensorflow/predict_videos.py

@@ -394,21 +394,26 @@ def GetPoseF(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes, batchsize):
                 )
             else:
                 frames[batch_ind] = img_as_ubyte(frame)
-            inds.append(counter)
+
             if batch_ind == batchsize - 1:
                 pose = predict.getposeNP(frames, dlc_cfg, sess, inputs, outputs)
-                PredictedData[inds] = pose
+                PredictedData[
+                    batch_num * batchsize : (batch_num + 1) * batchsize, :
+                ] = pose
                 batch_ind = 0
-                inds.clear()
                 batch_num += 1
             else:
                 batch_ind += 1
-        elif counter >= nframes:
+        else:
+            nframes = counter
+            print("Detected frames: ", nframes)
             if batch_ind > 0:
                 pose = predict.getposeNP(
                     frames, dlc_cfg, sess, inputs, outputs
                 )  # process the whole batch (some frames might be from previous batch!)
-                PredictedData[inds[:batch_ind]] = pose[:batch_ind]
+                PredictedData[
+                    batch_num * batchsize : batch_num * batchsize + batch_ind, :
+                ] = pose[:batch_ind, :]
             break
         counter += 1
 
@@ -446,7 +451,8 @@ def GetPoseS(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes):
             ] = (
                 pose.flatten()
             )  # NOTE: thereby cfg['all_joints_names'] should be same order as bodyparts!
-        elif counter >= nframes:
+        else:
+            nframes = counter
             break
         counter += 1
 
@@ -491,7 +497,8 @@ def GetPoseS_GTF(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes):
             ] = (
                 pose.flatten()
             )  # NOTE: thereby cfg['all_joints_names'] should be same order as bodyparts!
-        elif counter >= nframes:
+        else:
+            nframes = counter
             break
         counter += 1
 
@@ -530,7 +537,7 @@ def GetPoseF_GTF(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes, batchsize):
                 )
             else:
                 frames[batch_ind] = img_as_ubyte(frame)
-            inds.append(counter)
+
             if batch_ind == batchsize - 1:
                 # pose = predict.getposeNP(frames,dlc_cfg, sess, inputs, outputs)
                 pose = sess.run(pose_tensor, feed_dict={inputs: frames})
@@ -540,19 +547,26 @@ def GetPoseF_GTF(cfg, dlc_cfg, sess, inputs, outputs, cap, nframes, batchsize):
                 pose = np.reshape(
                     pose, (batchsize, -1)
                 )  # bring into batchsize times x,y,conf etc.
-                PredictedData[inds] = pose
+                PredictedData[
+                    batch_num * batchsize : (batch_num + 1) * batchsize, :
+                ] = pose
+
                 batch_ind = 0
-                inds.clear()
                 batch_num += 1
             else:
                 batch_ind += 1
-        elif counter >= nframes:
+        else:
+            nframes = counter
+            print("Detected frames: ", nframes)
             if batch_ind > 0:
                 # pose = predict.getposeNP(frames, dlc_cfg, sess, inputs, outputs) #process the whole batch (some frames might be from previous batch!)
                 pose = sess.run(pose_tensor, feed_dict={inputs: frames})
                 pose[:, [0, 1, 2]] = pose[:, [1, 0, 2]]
                 pose = np.reshape(pose, (batchsize, -1))
-                PredictedData[inds[:batch_ind]] = pose[:batch_ind]
+                PredictedData[
+                    batch_num * batchsize : batch_num * batchsize + batch_ind, :
+                ] = pose[:batch_ind, :]
+
             break
         counter += 1
 
@@ -631,7 +645,8 @@ def GetPoseDynamic(
                 detected = False
 
             PredictedData[counter, :] = pose
-        elif counter >= nframes:
+        else:
+            nframes = counter
             break
         counter += 1
 

deeplabcut/pose_estimation_tensorflow/train.py

@@ -222,8 +222,12 @@ def train(
     train_writer = TF.summary.FileWriter(cfg.log_dir, sess.graph)
 
     if cfg.get("freezeencoder", False):
-        print("Freezing...")
-        learning_rate, _, train_op = get_optimizer_with_freeze(loss_op, cfg)
+        if 'efficientnet' in cfg.net_type:
+            print("Freezing ONLY supported MobileNet/ResNet currently!!")
+            learning_rate, train_op, tstep = get_optimizer(total_loss, cfg)
+
+        print("Freezing encoder...")
+        learning_rate, _, train_op = get_optimizer_with_freeze(total_loss, cfg)
     else:
         learning_rate, train_op, tstep = get_optimizer(total_loss, cfg)
 

deeplabcut/utils/auxfun_models.py

@@ -47,7 +47,7 @@ def Check4weights(modeltype, parent_path, num_shuffles):
                     + modeltype.replace('_','-')))
     else:
         print(
-            "Currently ResNet (50, 101, 152) and MobilenetV2 (1, 0.75, 0.5 and 0.35) are supported, please change 'resnet' entry in config.yaml!"
+                        "Currently ResNet (50, 101, 152), MobilenetV2 (1, 0.75, 0.5 and 0.35) and EfficientNet (b0-b6) are supported, please change 'resnet' entry in config.yaml!"
         )
         num_shuffles = -1  # thus the loop below is empty...
         model_path = parent_path

deeplabcut/version.py

@@ -8,5 +8,5 @@
 Licensed under GNU Lesser General Public License v3.0
 """
 
-__version__ = "2.1.9"
+__version__ = "2.1.10"
 VERSION = __version__

docs/roadmap.md

@@ -1,4 +1,4 @@
-## A development roadmap for DeepLabCut 
+## A development roadmap for DeepLabCut
 
 
 :loudspeaker: :hourglass_flowing_sand: :construction:
@@ -7,7 +7,7 @@
 - [X] alpha testing complete (early May 2020)
 - [X] beta release: 2.2.b5 on 5 / 22 / 20 :smile:
 - [X] beta release: 2.2b8 released 9/2020 :smile:
-- [X] beta release 2.2b9 --> candidate release, slotted for Oct 2020 
+- [X] beta release 2.2b9 (rolled into 2.1.9_ --> candidate release, slotted for Oct 2020
 - [ ] full 2.2 stable release
 - [ ] Manuscripts forthcoming!
 
@@ -19,10 +19,13 @@
 - [X] BETA release with 2.1.8b0: http://www.mousemotorlab.org/dlc-modelzoo
 - [X] full release with 2.1.8.1 http://www.mousemotorlab.org/dlc-modelzoo
 - [ ] Manuscript forthcoming!
-- [X] new models added with 2.2b8! 
+- [X] new models added with 2.2b8!
 - [ ] contribution module
- 
+
 **DeepLabCut GUI and DeepLabCut-core:**
-- [X] to make DLC more modular, we will move core functions to https://github.com/DeepLabCut/DeepLabCut-core 
+- [X] to make DLC more modular, we will move core functions to https://github.com/DeepLabCut/DeepLabCut-core
 - [ ] new GUI for DeepLabCut; due to extended issues with wxPython, we will be moving to napari https://github.com/napari/napari
 - [ ] tensorflow 2.2 support: https://github.com/DeepLabCut/DeepLabCut/issues/601
+
+**General Improvements:**
+- [X] Efficient Net backbones added (currently SOTA on ImageNet). https://openaccess.thecvf.com/content/WACV2021/html/Mathis_Pretraining_Boosts_Out-of-Domain_Robustness_for_Pose_Estimation_WACV_2021_paper.html https://github.com/DeepLabCut/DeepLabCut/commit/96da2cacf837a9b84ecdeafb50dfb4a93b402f33

examples/test.sh

@@ -6,11 +6,13 @@ rm -r OUT
 cd ..
 pip uninstall deeplabcut
 python3 setup.py sdist bdist_wheel
-pip install dist/deeplabcut-2.1.9-py3-none-any.whl
+pip install dist/deeplabcut-2.1.10-py3-none-any.whl
 
 cd examples
 
-python3 testscript.py
+#python3 testscript.py
 #python3 testscript_3d.py #does not work in container
 #python3 testscript_mobilenets.py
 #python3 testscript_multianimal.py
+
+python3 testscript_openfielddata_netcomparison.py