Going from yolov5 results to expected folder.

[topherbuckley]

I have a finetuned yolo_v5 model, and can get the results from it just fine as a <class 'yolov5.models.common.Detections'> object, but how can I save these in a format that can be handled by this library? From what I can understand from the COCOEvaluator, I need it to be be of type AnnotationSet and the best way to generate one of those appears to be the following function:

@staticmethod
    def from_yolo_v5(
        folder: PathLike,
        *,
        image_folder: PathLike,
        image_extension=".jpg",
        verbose: bool = False,
    ) -> "AnnotationSet":
        return AnnotationSet._from_yolo(
            folder,
            image_folder=image_folder,
            image_extension=image_extension,
            conf_last=True,
            verbose=verbose,
        )

Now the only hints I get are that I need a PathLike folder, and a PathLike image_folder. I guess the image_folder can be the same as was used to generate the ground truth AnnotationSet, but what about the folder ? What needs to be in there and in what format? How can I take the yolov5 results i.e. <class 'yolov5.models.common.Detections'> in order to generate this folder? Or is there a way to use the <class 'yolov5.models.common.Detections'> object directly while in memory rather than save/load from file?

I tried a couple of different ways, but nothing throws errors, it just looks like nothing is parsed. i.e. evaluator.labels is empty.

[topherbuckley]

Just jotting down my notes for following up on this. If you follow the _from_yolo() you eventually end up globbing all files from the directory here:

        files = list(glob(folder, extension, recursive=recursive))
        return AnnotationSet.from_iter(parser, files, verbose=verbose)

Where extension is set to .txt here

Where parser was defined back in from_txt:

def _get_annotation(file: Path) -> Annotation:
            if image_folder is not None:
                image_path: Path | None = None

                for image_ext in IMAGE_EXTENSIONS:
                    image_id = file.with_suffix(image_ext).name
                    path = image_folder / image_id  # type: ignore

                    if path.is_file():
                        image_path = path
                        break

                assert (
                    image_path is not None
                ), f"Image {file.name} does not exist, unable to read the image size."

                image_id = image_path.name

                try:
                    image_size = get_image_size(image_path)
                except UnknownImageFormat:
                    raise ParsingError(
                        f"Unable to read image size of file {image_path}. "
                        f"The file may be corrupted or the file format not supported."
                    )
            else:
                image_size = None
                image_id = image_id = file.with_suffix(image_extension).name

            return Annotation.from_txt(
                file_path=file,
                image_id=image_id,
                box_format=box_format,
                relative=relative,
                image_size=image_size,
                separator=separator,
                conf_last=conf_last,
            )

this glob of .txt annotation files of unknown format are then passed through here:

        annotations = thread_map(parser, iterable, desc="Parsing", verbose=verbose)

where iterable is the glob, and parse the _get_annotation the parser and thread_map being defined as:

def thread_map(
    fn: Callable[[U], V],
    it: Iterable[U],
    desc: Optional[str] = None,
    total: Optional[int] = None,
    unit: str = "it",
    verbose: bool = False,
) -> "list[V]":
    disable = not verbose
    total = total or length_hint(it)
    results = []
    with tqdm(desc=desc, total=total, unit=unit, disable=disable) as pbar:
        futures = SHARED_THREAD_POOL.map(fn, it)
        for result in futures:
            results.append(result)
            pbar.update()
    return results

So it is applying the _get_annotation function on each annotation file. So coming back to the _get_annotation function, each file here is just a a Path to each .txt file previously globbed.

Eventually it calls Annotation.from_txt here

            return Annotation.from_txt(
                file_path=file,
                image_id=image_id,
                box_format=box_format,
                relative=relative,
                image_size=image_size,
                separator=separator,
                conf_last=conf_last,
            )

box_format=BoxFormat.XYWH and relative=True from here. Reminder to self to confirm this format. I believe it is center XY followed by Width and Height, as opposed to any corner coords. (which I believe is the yolo5 default).

Then the .txt file is split into lines:
lines = path.read_text().splitlines()

Then each line is itterated over to generate the boxes variable via:

            boxes = [
                BoundingBox.from_txt(
                    l,
                    box_format=box_format,
                    relative=relative,
                    image_size=image_size,
                    separator=separator,
                    conf_last=conf_last,
                )
                for l in lines
            ]

separator=None from back here so when the lines are split here

values = string.strip().split(separator) it splits based on any sequential whitespace (So need whitespace not commas, etc.).

Then it checks if there are either 5 or 6 values, else throw ParsingError. It appears those with 5 values are used as ground truth annotations (i.e. confidence=None) thus I also now know I need 6 values.

Continuing on, I know conf_last=True, as per here so now I know I need the following format:

label, *coords, confidence = values

Now is label supposed to be the class ID or the string representation of the class? I assume confidence should be from 0 to 1? Are the coords normalized to 1 or relative to image size?

yolov5 results gives both class (numerical id) and name (text string), so need to confirm which one it needs here.

Confidence should indeed be from 0 to 1 as per this assertion

relative=True from here so coords are not normalized.

Following the return call to BoundingBox.create

So in summary to answer my question folder needs to contain:

1. .txt files one for each image in image_folder with idential names to the images, but minus whatever image extension and with .txt instead
2. Each .txt file should have one or more lines. Each line with exactly 6 whitespace separated values starting with the label (still unsure if this needs to be numerical id or text classname), followed by XYWH coords, then confidence [0,1].

[topherbuckley]

Without a real plan for confirming what label should be, I tried both and it looks like the text string is what is desired as that is what was given for the label in my call to gt = AnnotationSet.from_coco(gt_path) from inspecting gt._labels, but the other explanation is the COCO format dataset json file I have from roboflow is using the wrong values and should be using the numerical values. Either way, it seems using numerical values in the detections leads to evaluator.show_summary() with both numerical and text labels, so guessing this isn't right. Guessing the only thing that matters is that they match (i.e. can be either numerical or text class name, but should match in both ground truth annotations and detection annotations).

Using the text labels, I still see all 0.00% for my dataset, which I can't image is true. So I narrowed it down to a single image for "unit" testing where I know my model infers pretty well. I stopped using the COCO format for this as it was harder to parse the COCO single json file rather than just using a single label and image file for this unit test. In doing so I found the dataset in Yolo5 Pytorch format from Roboflow comes with coords that are relative=True but the coords from the yolov5 model in results are relative=False.

The only workaround I could find was to use the from_txt function so I could override the incorrect values for relative and had to also specify conf_last=True and box_format=BoxFormat.XYWH as I was bypassing the defaults from the yolo specific from_ functions:

predictions = AnnotationSet.from_txt(folder="./detections", image_folder="./dataset/test/images", box_format=BoxFormat.XYWH, relative=False, conf_last=True)

This gave me nonzero results as expected.

Is this a known issue? Guessing its not ideal that this relative value is different for the dataset given by Roboflow compared to the one given by the model itself for detection results.

[laclouis5]

Hi @topherbuckley,

The from_yolo_v5 method is for parsing YOLOv5 annotations stored to disk in .txt files. The folder parameter refers to the directory containing such annotations while the image_folder refers to the directory containing the images corresponding to the annotations (could be the same as folder.

You should not use this method if you want to parse in-memory annotations / predictions such as ones of type yolov5.models.common.Detections. For this purpose I recommend using the BoundingBox.create method. Here is how you should proceed to build an AnnotationSet:

• For each image, create a list of bounding boxes using BoundingBox.create.
• Store the bounding boxes using an Annotation object. Do not forget to specify the image size and image name in the init.
• Finally, create an AnnotationSet object feeding the init with the list of Annotation objects of your predictions.

[topherbuckley]

Thanks for the recommendation. I already implemented it with the AnnotationSet.from_txt function, so might come back and redo it when dealing with larger datasets for some potential speed increase.

[topherbuckley]

@laclouis5 would something like this be desired within globox as a from_yolo_v5_results or the like? Assuming I go back and implement this in the near future, wondering if you wanted it implemented here as I could use it from here rather than track it in our project. Happy to submit a PR later if so.

[laclouis5]

If you want to contribute implementing this functionality, I'd will happily review your PR!

[topherbuckley]

Alrighty. Where would you want the highest level call? From my point of view, I'd ideally like a single call to something like AnnotationSet.from_yolo_v5_results that takes a yolov5.models.common.Detections object as an input and then proceed as you mentioned from BoundingBox.create. Does that align with how you'd want this to be structured?