weights and sphere are Nan

[armand-hoxha25]

Hello and thanks for the toolbox. I have question regarding some of the data i have. I have run the cudaica on a couple of datasets and i receive an error. I traced it down to the point where the weights and sphere matrix from cudaica are all NaN. This does not happen on every dataset, however some of the other datasets are retreive results which appear to be different from pop_runica. Otherwise the cudaica works properly on other datasets with almost identical results to pop_runica (i run pop_runica with 'cudaica' option, and using [wts,sph]=cudaica(EEG.data); ). Is there something wrong with the data i have?

at the end of each Step in ica i get:
Step 100 - epsilon: -nan, lrate 0.001000000, wchange -nan, angledelta -nan deg

[fraimondo]

Hello, CUDAICA is indeed the same algorithm that is there in EEGLab under ‘binica’, ‘infomax’ or ‘infomax-extended’. The only difference is that CUDAICA does not perform PCA before (pca parameter is not valid). If you run infomax or binica with the same parameters, you should get the same result. Although components can be inverted (multiplied by -1) or in a different order. Nevertheless, form my experience, your problem seems to be associated with the data. With this dataset that you have problems, did you have the same problem with 'infomax’? Can you paste the output of cudaica?

…

On 1 Nov 2017 at 20:43 +0100, Armand Hoxha ***@***.***>, wrote: Hello and thanks for the toolbox. I have question regarding some of the data i have. I have run the cudaica on a couple of datasets and i receive an error. I traced it down to the point where the weights and sphere matrix from cudaica are all NaN. This does not happen on every dataset, however some of the other datasets are retreive results which appear to be different from pop_runica. Otherwise the cudaica works properly on other datasets with almost identical results to pop_runica (i run pop_runica with 'cudaica' option, and using [wts,sph]=cudaica(EEG.data); ). Is there something wrong with the data i have? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[armand-hoxha25]

Hello thanks for the reply, I ran the EEGLAB ica and i do not see the same issue, as in i get good decomposition of the data.
I attached the cudaica outputs from my cudaica (weights, and sphere). these are the files that are created by cudaica, i did not put in the fdt file, but if you want i can attach that one too.
cudaica_out.tar.gz

in case its my graphic card here is my cuda ica initialization
List of cuda devices:

=====================

Device: 0
CUDA Device capabilities:
Name: GeForce GTX 780
Global Mem: 3159031808
Mem: 3159031808
Mem per Block: 49152
Regs per Block: 65536
Warp size: 32
Mem pitch: 2147483647
Max Threads per Block: 1024
Max Threads Dim: 1024 x 1024 x 64
Max Grid Size: 2147483647 x 65535 x 65535
Total Const Mem: 65536
Major: 3
Minor: 5
Clock Rate: 1019500
Texture Alignment: 512
Device Overlap: 1
Multiprocessor Count: 12
Kernel Timeout Enabled: 1
Integrated: 0
Can Map host mem: 1
Compute mode: 0
Concurrent kernels: 1
ECC Enabled: 0
PCI Bus ID: 1
PCI Device ID: 0
TCC Driver: 0

Selecting device 0 Success!

In matlab the outcomes of weights and sphere are 64x64 matrix of NaN .

[fraimondo]

You have set the PCA parameter to 60, so you are reducing the dimensionality. CUDAICA does not support that.

…

On 2 Nov 2017 at 13:24 +0100, Armand Hoxha ***@***.***>, wrote: Hello thanks for the reply, I ran the EEGLAB ica and i do not see the same issue, as in i get good decomposition of the data. I attached the cudaica outputs from my cudaica (weights, and sphere). these are the files that are created by cudaica, i did not put in the fdt file, but if you want i can attach that one too. cudaica_out.tar.gz in case its my graphic card here is my cuda ica initialization List of cuda devices: Device: 0 CUDA Device capabilities: Name: GeForce GTX 780 Global Mem: 3159031808 Mem: 3159031808 Mem per Block: 49152 Regs per Block: 65536 Warp size: 32 Mem pitch: 2147483647 Max Threads per Block: 1024 Max Threads Dim: 1024 x 1024 x 64 Max Grid Size: 2147483647 x 65535 x 65535 Total Const Mem: 65536 Major: 3 Minor: 5 Clock Rate: 1019500 Texture Alignment: 512 Device Overlap: 1 Multiprocessor Count: 12 Kernel Timeout Enabled: 1 Integrated: 0 Can Map host mem: 1 Compute mode: 0 Concurrent kernels: 1 ECC Enabled: 0 PCI Bus ID: 1 PCI Device ID: 0 TCC Driver: 0 Selecting device 0 Success! In matlab the outcomes of weights and sphere are 64x64 matrix of NaN . — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[CloudyDory]

Hi,

I also got 64*64 NaN matrix out of cudaica.

Here is the .sc file:

DataFile /home/yunhui/Downloads/Subj10/cudaica1270.fdt
 
chans 64
frames 174375
WeightsOutFile /home/yunhui/Downloads/Subj10/cudaica1270.wts
SphereFile /home/yunhui/Downloads/Subj10/cudaica1270.sph
sphering on
bias on
extended 1
 
 
 
pca 0
lrate 1.0e-4
 
blocksize 0
 
stop 1.0e-7
 
maxsteps 512
posact off
 
annealstep 0.98
 
annealdeg 60
momentum 0
verbose on

And every step I get:

Step xxx - epsilon:    -nan, lrate 0.000100000, wchange    -nan, angledelta -nan deg - Elapsed time: 352203 tics = 0 h 0 m 0 s

Yet ICA runs fine using "runica()" with "'extended', 1" options in the EEGLAB with exactly the same dataset. Fraimondo can you please help me on this problem?

[CloudyDory]

After debugging I found NaNs are produced at line 1010 of infomax.cu:

HANDLE_ERROR(cudaMemcpyFromSymbol(&h_change, SYMBOL(dotResult), sizeof(h_change)));

After this line, the value of h_change changes to NaN.

I am not familiar with CUDA programming, however. Anyone knows what is the cause of this problem?

[fraimondo]

The value of h_change changes to NaN after that line because that is the line when the value is copied from GPU memory to CPU memory.

Can you provide the full output?
If you run binica < cudaica1270.sc, do you have the same problem? The .SC files are compatible with the CPU binary version, and it should produce the same result.

[CloudyDory]

Hi, thanks for the reply. After further debugging, the NaN originates in the following line of whitening.cu:
host_eigd[i] = 0.5 * sqrt(host_eigd[i]);

It may produce NaN because if the data is rank-deficient (like if you rereference the data to average reference, or remove and interpolate channels), some of the eigenvalues are very close to zero, but negative. They produce NaN after taking the square root.

I have used EEGLAB for more than one year now, and it seems strange that Matlab can always produce close-to-zero but positive results when it calculates eigenvalues from double precision, rank-deficient covariance matrix.

[fraimondo]

Hi! CUDAICA is not prepared for that. You should not do average reference nor interpolate channels before doing ICA. See: https://sccn.ucsd.edu/~arno/mypapers/Artoni2018.pdf

…

On 3 May 2018, at 14:08, yhz-1995 ***@***.***> wrote: Hi, thanks for the reply. After further debugging, the NaN originates in the following line of whitening.cu: host_eigd[i] = 0.5 * sqrt(host_eigd[i]); It may produce NaN because if the data is rank-deficient (like if you rereference the data to average reference, or remove and interpolate channels), some of the eigenvalues are very close to zero, but negative. They produce NaN after taking the square root. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AESRg0RkUwhErTZ74pe9Als-e5YCJEQpks5tuvM4gaJpZM4QOt-_>.