Recommended weighted IoU for imbalanced labels (single class)

[JulienMaille]

Hello, I figured I would get the ball rolling and create a first post in this shiny, new discussion section.
Here is the thing, let's say you have a dataset for single-class segmentation with:

• many images with small labels
• many images without any label
• a few images with very large labels

Now let's say you compute IoU on the validation dataset with batch_size = 1
Regular IoU will compute the IoU of each image and average across all images Sum(I/U)/N
-> in that case, images without any label will either contribute with a score of 1 (no prediction) or 0 (even a single pixel false-positive will lead to 0) i.e.. you get artificially high IoU as long as empty images are correctly predicted

My first approach was to code a partial-IoU: compute I & U for each image and then Sum(I)/Sum(U)
-> in that case empty images do not inflate the score anymore (but they might not contribute enough)
-> but images with very large labels will now steer the score due to being the largest contributors to partial-IoU

So now I'm wondering if there are any SotA techniques to properly weight the metric and avoid large contirbution of extremes images (no label and large label). My research did not let me find anything yet.
Approaches I had in mind

• partial-IoU: weight each image using the inverse of the number of pixels labels in that image
• IoU: increase epsilon to change the penalty shape of images without labels

[Mirzyaaliii]

Hello @JulienMaille,

I am working on the same problem as yours where most data samples are negative (i.e., without any object) and others have small or large labels. I am also wondering which loss function or approach is suitable for such an imbalanced dataset.

(It's been a long since this discussion thread) still wanted to check if anyone found the solution.