VoidFinder hanging on small number of cells to search

[kadglass]

Accidentally ran parallel VF on a job with ~300 cells to search. By setting the batch_size to 10, VF was able to complete the job. This indicates that there were processes that weren't getting any work and were therefore hanging the code. The sign for this is a printout of "Processed 0 cells of xxx cells".

[gideonkmc]

Sorry to revive an old issue, but I was wondering if a solution was found for this? I just realized that this may be hanging my code too. I think I will try lowering the batch size, but it would be great to hear if there is a better fix.

[kadglass]

@gideonkmc - Great question! This issue should be resolved in the periodic_fix_issue111 branch. We need to resolve some minor unit test conflicts before merging it into the main branch, which is why the associated pull request is still open. Feel free to use that branch, and let us know if you encounter any issues with it!

[gideonkmc]

@kadglass Thanks for the suggestion! I tried using the periodic_fix_issue111 branch, and it helped a little bit, but my code starts to slow down as more and more cells get processed. If I run the example script SDSS_VoidFinder_dr7.py everything works fine and it runs quickly, but when I modify the code to use my dataset and redshift limits, the hole-growing algorithm seems to get stuck.

The main issue seems to be the change in redshift limits. My data is from a redshift of z = 0 to z = 0.2, but if I change the limits to 0 and 0.1, like in the SDSS example, the algorithm speeds up significantly. Is this an error that is known? And if so, is there any way to extend the redshift limit so that the algorithm runs better above a z=0.1? Happy to open a separate issue on this if it would be helpful!

[QuiteAFoxtrot]

I'm not 100% sure if this is what you're experiencing, but one known issue of slowness with VoidFinder is that if you have large empty regions between where your data is, and the edge of your survey mask, whenever VF goes to grow a hole there it basically grows a giant hole which doesn't terminate until the center of that enormous hole reaches the boundary of the survey mask. One thing you could try to start would be to try and check how tightly your survey mask encapsulates your valid data region. If by some chance you're working with a rectangular sub-section of a survey you can use mask_type='xyz' and the xyz_limits https://github.com/DESI-UR/VAST/blob/master/python/vast/voidfinder/voidfinder.py#L923 input to find_voids instead of mask_type='ra_dec_z' and mask.

[gideonkmc]

@QuiteAFoxtrot Thanks for the reply and apologies for the delay in getting back to you on this! I did some tests with changing the boundary of the mask and limiting the redshift of my survey and it looks like the slowdown was caused by the issue you mentioned. If I limit the redshift so that there are enough galaxies near the border, everything seems to run fine.

Thanks again for your help!