Optimisation of directory type checking in file list #1381
Conversation
… directory-based image formats
|
I did a test with a directory with 20000 fits image files, and a directory with 20000 casa image directories. The text computer is a local build with a fast SSD. I did purge disk cache for each test case. The test results (in seconds) are summarized in the following table: We see improvements of ~2x for the casa image with the "filter by extension" and the "all files" modes. Is this expected in this improvement and consistent with what you tested and observed? The psrecord plots are attached below as a record. The first plateau is casa image and the second is fits image. old - content new - content old - extension new - extension old - all new - all It appears to me that once we access the context of a file, the CPU usage won't be 100% in my test case. |
|
| @@ -10,6 +10,7 @@ | |||
| #include <fstream> | |||
| #include <regex> | |||
|
|
|||
| #include "Casacore.h" | |||
There was a problem hiding this comment.
Asking because I am looking at a file list bug and might include File.h in Casacore.h.
This is an attempt to improve the performance of generating a file list for directories with lots of subdirectories. The main performance bottlenecks in the current code are:
There are three options for displaying file lists: filtering by content, filtering by file extension, and no filtering. Filtering by content additionally slows down the file list generation because it reads the signature of each ordinary file to determine the type. Guessing the type from the file extension speeds up the processing of ordinary files, but not subdirectories, which are always checked with casacore's
ImageOpener. Turning off filtering entirely has no additional performance impact on the image file list (because detection of directory-based images is still required), but speeds up the region file list (because if no filtering is required, all directories can be shown without additional checks).This PR is an attempt to add a less expensive heuristic for directory-based images, to be used when the option to filter by content is not selected. It performs almost the same checks as
ImageOpener, but only for directories, by looking for files inside the directory withfs::exists, and without distinguishing between different CASA image subtypes.The existing code assumes that there are directory image formats which we do not support, and handles them differently. However, it's clear from the
ImageOpenercode that the GIPSY format is a pair of files, not a directory (soImageOpenerwould never return that type for a directory), and theCAIPSandNEWSTARtypes are obsolete and never returned byImageOpener(at all). So I have removed this option from the code, and not implemented it in the alternative code.The result: the alternative code appears to be slightly faster, but I don't know if it's faster enough for it to make sense to add it as an alternative to the casacore code. If this implementation is sufficient for our purposes (e.g. we don't need to read the
table.infofile because we don't need do distinguish between CASA sub-types here), then perhaps we should replace the casacore check with this (for a modest speed improvement in all cases).Other optimizations we discussed:
I think we're planning to use the last option as our long-term solution, and I would suggest applying that strategy to all files: instead of loading file information up-front when generating the file list, we could initially return just a bare list of files and directories, and then return information for lists of files and directories as the frontend requests them.
Checklist