crash when running --all on a very large project

[todcourtney]

ClangBuildAnalysis is a great tool and it has really helped me improve the build time of a large project I am working on.

My project is so large that the code crashes when loading the json files. I believe it is running out of memory while parsing the json dom. my project has 83GB of json files. I have run ClangBuildAnalyzer on all the files and I can analyze all of them if I do it in two groups, with the 51 largest json files in one group and the other 241 files in the other group.

While debugging, I changed the max tasks to 1 in main.cpp, hoping that parsing 1 json at a time would reduce the memory footprint. It may have gotten farther, but ClangBuildAnalyzer still crashed.

I tried using gdb to see the crash, but the app is killed due to out of memory, instead of core dumping, so gdb didn't give me any useful information.

I wondered if there was a memory leak or other bug in the json parsing. I manually copied the latest release of simdjson (0.9.4) into ClangBuildAnalyzer and rebuilt. I still got the same crashes.

I am currently using clang-11. I had more luck with ClangBuildAnalyzer on this project in the past and may have been using clang-10. I wonder if clang-10 writes fewer events? I do not see any warnings about unknown events while running ClangBuildAnalyzer with clang-11.

Could ClangBuildAnalyzer add a batch mode to support large projects? For example, I could run ClangBuildAnalyzer -all <subdirN> on the N subdirs of my build. Then, I would run ClangBuildAnalyzer --analyze with N file names instead of 1. ClangBuildAnalyzer would then read all N binary event files and concatenate all the data together before producing the statistics over the combined set of all events.

I have studied the LoadBuildEvents() method and believe it could be possible to have it append new files to the current event and names lists, but haven't had time to do it yet, especially making the event to name index is updated correctly.