DIANN 1.8.1 exited

[PsuperGirl]

Hi Vadim,
I am using DIANN 1.9 to quantify my 3227 MS files (converted to mzML) from Orbitrap Astral.
However, it failed to generate output tables and reports and exited as in the screenshot below. I restarted PC and tried to change the threads number (from maximal 56 to 32). However, it did not help very much. I am wondering if you have some suggestions for it.

Best,
Juanjuan

[vdemichev]

Hi Juanjuan,

Could it be that some of the .quant files were rewritten by a parallel analysis of the same raw data?
If you monitor RAM usage, is there enough RAM free?
Here I cannot really help with troubleshooting, can only recommend to switch to 1.9.1.

Best,
Vadim

[PsuperGirl]

Hi Juanjuan,

Could it be that some of the .quant files were rewritten by a parallel analysis of the same raw data? If you monitor RAM usage, is there enough RAM free? Here I cannot really help with troubleshooting, can only recommend to switch to 1.9.1.

Best, Vadim

Hi Vadim,
Thank you very much. It was already 1.9. Is there a new version than 1.9? I will check it out. There is only DIANN running on those raw files and no other programs running either on my PC. It was competely free still while DIANN was running.

Best,
JJ

[vdemichev]

Hi JJ,

The screenshot you showed is from 1.8.1

Best,
Vadim

[PsuperGirl]

Hi Vadim,

I have tried it again with v1.9. It has been reading quant.files for 3 days. But nothing is going forward, including the running time. It stays at 572 all the time. There is no indication of the reading progress either. I am wondering if DIANN is stuck.
I am not running it in the command line right now. But I was wondering if there is a way I could check out what DIANN is doing now.

Best,
JJ

[vdemichev]

Hi JJ,

And what's the physical RAM usage at this point?
If it's not a RAM issue, iI would be grateful if you could .zip all the .quant files and upload them somewhere together with the 2024_Library_Phase1.tsv?

Best,
Vadim

[PsuperGirl]

Hi JJ,

And what's the physical RAM usage at this point?

Best, Vadim

Hi Vadim,
It is quite low on the usage of memory. The usage of CPU from DIANN is 0. I am not sure if it is a good sign.
Would you like to test some stuffs out from your end? All my .quant files together would be more than 300GB. It would be too big to share. Additionally, with GDPR rule in Denmark, I am not allowed to share the data with you either.
Best,
JJ

[vdemichev]

Is G a network drive?

[PsuperGirl]

Is G a network drive?

It is a hard drive.
Best,
JJ

[PsuperGirl]

Is G a network drive?

All my data including quant. files are also in one folder on this hard drive.
Best,
JJ

[PsuperGirl]

Is G a network drive?

I have also tried to export the output tables in a local drive. However, it did not work either. DIANNv1.9 did not generate the output tables and exited.

Best,
JJ

[vdemichev]

This is very puzzling, giving that the issue is with two different DIA-NN versions.
If you just select the subset of files in the GUI (since you can keep 'Use existing .quant files' checked, the analysis should take just several minutes), does it still fail (version 1.9)?

[PsuperGirl]

This is very puzzling, giving that the issue is with two different DIA-NN versions. If you just select the subset of files in the GUI (since you can keep 'Use existing .quant files' checked, the analysis should take just several minutes), does it still fail (version 1.9)?

In deed, it is very puzzing. I first tried to run 100 samples with v1.9 and worked out. But with more than 3000 files, it failed to generate output tables (export to the local drive or a hard drive) on another computer with v1.9. Therefore I changed to another version v1.8 and another PC. But it did not help as in the message I sent before. Then I change back to v1.9. It just kept reading files for 3 days now without any progress shown.

I just tried v1.9.1 now while the other DIANN v1.9 is still running "reading quant.files". Suddendly, it shows my quant.files are currupted as below. I am not sure if it is because there is another DIANN is running.

[vdemichev]

'Corrupted' message will appear if the .quant file was produced with a different DIA-NN version

[PsuperGirl]

Owner

Ohh. OK. Then it makes sense. I generated quant.files with v1.9.

[PsuperGirl]

Hi Vadim,
In this case, in my setting now, I also used "low RAM&high speed". Do you think it could be the reason? Shall I stop the current run and start a new run with different settings?
Additionaly, I am wondering if you have an example script file for running in the command line which I could use as a base to modify according to my conditions?
Best,
JJ

[vdemichev]

No, this setting cannot affect things here.
The GUI prints the example command line on top of the log.

[PsuperGirl]

No, this setting cannot affect things here. The GUI prints the example command line on top of the log.

Shall I leave the DIANN running then? Is it normal the timeline shown in the log field is not changing forwards while reading quant.files?

Best,
JJ

[vdemichev]

Well, you can leave it, maybe it finally does something. Does Task Manager show that it's accessing the disk?

[PsuperGirl]

Well, you can leave it, maybe it finally does something. Does Task Manager show that it's accessing the disk?

From the task manager, the usage of the hard drive(G) is 0%.

[vdemichev]

I see it's a USB disk... Possible that for whatever reason it just takes a lot of time to read those files from it? Frankly speaking, disk access being somehow slow is the only thing I can think of. Any software can do one of the following:

• Crash - but then you see this.
• Go into an infinite loop - but then CPU load will be high.
• Deadlock - but at this stage DIA-NN does not use any locks (multithreaded synchronisation mechanisms), this part is enrtirely singlethreaded, deadlocking is not possible.

One thing can help diagnose: https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer.
In this tool, go to View -> Select Columns -> Process I/O -> select Read bytes. Then see if Read bytes column keeps changing for the diann.exe process. Also, View -> Select Columns -> Process Memory -> Private bytes will be increasing if DIA-NN is reading things...

Possible that some firewall/antivirus suspended diann.exe or possible that it was set to low priority in Windows? Can check in Task Manager.

[PsuperGirl]

I see it's a USB disk... Possible that for whatever reason it just takes a lot of time to read those files from it? Frankly speaking, disk access being somehow slow is the only thing I can think of. Any software can do one of the following:

• Crash - but then you see this.
• Go into an infinite loop - but then CPU load will be high.
• Deadlock - but at this stage DIA-NN does not use any locks (multithreaded synchronisation mechanisms), this part is enrtirely singlethreaded, deadlocking is not possible.

One thing can help diagnose: https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer. In this tool, go to View -> Select Columns -> Process I/O -> select Read bytes. Then see if Read bytes column keeps changing for the diann.exe process. Also, View -> Select Columns -> Process Memory -> Private bytes will be increasing if DIA-NN is reading things...

Possible that some firewall/antivirus suspended diann.exe or possible that it was set to low priority in Windows? Can check in Task Manager.

Thank you so so much for the very informative guide. DIANN had a normal priority before. I have set it as high now. After checking out DIANN performance in the tool, Read bytes is not changing and stays at 8.1M. Instead, Private bytes keeps changing.
Does this indicate DIANN is not reading things?
Best,
JJ

[vdemichev]

Well, if private bytes keeps changing then it's allocating memory, so something is going on (that's positive :)).

Best,
Vadim

[PsuperGirl]

Well, if private bytes keeps changing then it's allocating memory, so something is going on (that's positive :)).

Best, Vadim

Thanks a lot. It looks like this now. I was mainly monitoring the DIANN instance with DIANN symbol at the bottom of the screenshot. If I monitor the first one, both read bytes and private bytes keep changing.

Best,
JJ

[vdemichev]

OK, answer to everything is simple :)

Second DIA-NN: 430 Gb of RAM (please terminate it) - most of that should be swapped, means extremely slow processing.

First DIA-NN: still OK, slowed down by Windows constantly swapping RAM to disk and back because of the first one.

Why are there two running diann instances?

[PsuperGirl]

OK, answer to everything is simple :)

Second DIA-NN: 430 Gb of RAM (please terminate it) - most of that should be swapped, means extremely slow processing.

First DIA-NN: still OK, slowed down by Windows constantly swapping RAM to disk and back because of the first one.

Why are there two running diann instances?

The second one was from the previous try with v1.9.1. It was not fully terminated. I have terminated the old run with GUI and started a new run with the command line. Now it is reading files again. Both read bytes and private beads keep changing.
Best,
JJ

[vdemichev]

So all .quant files together 300Gb?
That would explain why takes so much RAM.
We do actually plan on optimising (a lot) RAM usage for such scenarious, but for now it just puts all files in RAM.

[vdemichev]

If not enough RAM, you can always process them 'by batches' (i.e. 'Use existing .quant files' in combination to just selecting a subset of raw files). Makes most sense doing this by splitting the experiment in a way that corresponds to experimental batches. DIA-NN then needs to run in a way that makes quantification consistent across batches, i.e. see the docs on incremental processing of large experiments here: https://github.com/vdemichev/DiaNN?tab=readme-ov-file#frequently-asked-questions.

[PsuperGirl]

If not enough RAM, you can always process them 'by batches' (i.e. 'Use existing .quant files' in combination to just selecting a subset of raw files). Makes most sense doing this by splitting the experiment in a way that corresponds to experimental batches. DIA-NN then needs to run in a way that makes quantification consistent across batches, i.e. see the docs on incremental processing of large experiments here: https://github.com/vdemichev/DiaNN?tab=readme-ov-file#frequently-asked-questions.

Thank you very much. Yes. All quant.files would be 300GB. Indeed the optimized-RAM-usage version will help a lot for large-scale studies. I could try out the suggestions.
Best,
JJ

[vdemichev]

About incremental processing. In addition to what's described in docs, another option with 1.9 and later here is that you can use --export-quant to just get fragment-level non-normalised quantities for every precursor ID. Can then (with the help of some R scripting) aggregate those into precursor and protein quantities.