Segmentation violation in random matlab jobs

Question

0 Stimmen

When I send a number of Matlab (2015b) jobs to our (Linux) computing cluster, some of them crash because of a 'segmentation violation'. I cannot figure out the cause. Upon re-submission of the crashed jobs, again some of them might crash again, some of them won't. Those that crashed multiple times didn't do it at the same point every time, so the problem is unreproducible.

My analysis is custom using FieldTrip toolbox. In short, I preprocess my data and do frequency analysis by looping over the trials. The crash always occurs at the frequency analysis step, but never at the same trial (I made sure there are no bugs in the code. The crash also occurred in another function with a loop).

The crashes do not occur because of a problem with a computing node. I also made sure I have enough resources (memory and walltime). The only similar problem I found here on the forum had to do with segmentation violation during startup of matlab, which is no problem for me.

This is what the matlab crash report says:

   ------------------------------------------------------------------------
         Segmentation violation detected at Mon Aug 14 21:29:53 2017
  ------------------------------------------------------------------------
Configuration:
  Crash Decoding      : Disabled
  Crash Mode          : continue (default)
  Current Graphics Driver: Unknown software 
  Current Visual      : None
  Default Encoding    : UTF-8
  GNU C Library       : 2.17 stable
  Host Name           : dccn-c019.dccn.nl
  MATLAB Architecture : glnxa64
  MATLAB Root         : /opt/matlab/R2015b
  MATLAB Version      : 8.6.0.267246 (R2015b)
  OpenGL              : software
  Operating System    : Linux 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64
  Processor ID        : x86 Family 111 Model 2 Stepping 0, AuthenticAMD
  Virtual Machine     : Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
  Window System       : No active display
Fault Count: 1
Abnormal termination:
Segmentation violation
Register State (from fault):
  RAX = 0000000000000090  RBX = 00007fb8083bb070
  RCX = 00007fb65094e9d0  RDX = 00b9104000000000
  RSP = 00007fb8150f8cc8  RBP = 00007fb8150f8cf0
  RSI = 00007fb817f1b580  RDI = 00007fb8085b82d0
     R8 = 0000000000000006   R9 = 0000000000000003
    R10 = 00007fb79d20dfa0  R11 = 00007fb8085b82d0
    R12 = 00007fb808323e90  R13 = 00007fb8085b82d0
    R14 = 00007fb8083bb070  R15 = 00007fb8048abcd0
    RIP = 00007fb83372b1e3  EFL = 0000000000010206
     CS = 0033   FS = 0000   GS = 0000
Stack Trace (from fault):
[  0] 0x00007fb83372b1e3                                   <unknown-module>+00000000
[  1] 0x00007fb81d128676       /opt/matlab/R2015b/bin/glnxa64/libmwm_lxe.so+05678710
[  2] 0x00007fb81d137075       /opt/matlab/R2015b/bin/glnxa64/libmwm_lxe.so+05738613
[  3] 0x00007fb81d308033       /opt/matlab/R2015b/bin/glnxa64/libmwm_lxe.so+07643187
[  4] 0x00007fb81d2cfc40       /opt/matlab/R2015b/bin/glnxa64/libmwm_lxe.so+07412800
[  5] 0x00007fb81d2d2078       /opt/matlab/R2015b/bin/glnxa64/libmwm_lxe.so+07422072
[  6] 0x00007fb81d2d2140       /opt/matlab/R2015b/bin/glnxa64/libmwm_lxe.so+07422272
[  7] 0x00007fb81d3496bc       /opt/matlab/R2015b/bin/glnxa64/libmwm_lxe.so+07911100
[  8] 0x00007fb81d349abc       /opt/matlab/R2015b/bin/glnxa64/libmwm_lxe.so+07912124
[  9] 0x00007fb822300d0d /opt/matlab/R2015b/bin/glnxa64/libmwm_interpreter.so+02600205 _Z51inEvalCmdWithLocalReturnInDesiredWSAndPublishEventsRKSbIDsSt11char_traitsIDsESaIDsEEPibbP15inWorkSpace_tag+00000077
[ 10] 0x00007fb823f15a12         /opt/matlab/R2015b/bin/glnxa64/libmwiqm.so+00915986 _ZNK3iqm18InternalEvalPlugin24inEvalCmdWithLocalReturnERKSbIDsSt11char_traitsIDsESaIDsEEP15inWorkSpace_tag+00000098
[ 11] 0x00007fb823f15bd8         /opt/matlab/R2015b/bin/glnxa64/libmwiqm.so+00916440 _ZN3iqm18InternalEvalPlugin7executeEP15inWorkSpace_tagRN5boost10shared_ptrIN14cmddistributor17IIPCompletedEventEEE+00000120
[ 12] 0x00007fb8230bb695         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00677525
[ 13] 0x00007fb823f0f1c6         /opt/matlab/R2015b/bin/glnxa64/libmwiqm.so+00889286
[ 14] 0x00007fb823efc645         /opt/matlab/R2015b/bin/glnxa64/libmwiqm.so+00812613
[ 15] 0x00007fb81ea11bf9      /opt/matlab/R2015b/bin/glnxa64/libmwbridge.so+00146425
[ 16] 0x00007fb81ea121f4      /opt/matlab/R2015b/bin/glnxa64/libmwbridge.so+00147956
[ 17] 0x00007fb81ea176cd      /opt/matlab/R2015b/bin/glnxa64/libmwbridge.so+00169677
[ 18] 0x00007fb81ea177bc      /opt/matlab/R2015b/bin/glnxa64/libmwbridge.so+00169916
[ 19] 0x00007fb81ea17ead      /opt/matlab/R2015b/bin/glnxa64/libmwbridge.so+00171693 _Z8mnParserv+00000749
[ 20] 0x00007fb8230bdb4f         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00686927 _ZN11mcrInstance30mnParser_on_interpreter_threadEv+00000031
[ 21] 0x00007fb8230aa443         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00607299
[ 22] 0x00007fb8230aaa39         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00608825 _ZN5boost6detail11task_objectIvNS_3_bi6bind_tIvPFvRKNS_8functionIFvvEEEENS2_5list1INS2_5valueIS6_EEEEEEE6do_runEv+00000025
[ 23] 0x00007fb8230abf47         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00614215 _ZN5boost6detail9task_baseIvE3runEv+00000071
[ 24] 0x00007fb8230abfa7         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00614311
[ 25] 0x00007fb8230a72fa         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00594682
[ 26] 0x00007fb8239497ab    /opt/matlab/R2015b/bin/glnxa64/libmwservices.so+01947563
[ 27] 0x00007fb8172e06ed         /opt/matlab/R2015b/bin/glnxa64/libmwuix.so+00206573
[ 28] 0x00007fb823a4b2ba    /opt/matlab/R2015b/bin/glnxa64/libmwservices.so+03003066
[ 29] 0x00007fb823a4b5f4    /opt/matlab/R2015b/bin/glnxa64/libmwservices.so+03003892
[ 30] 0x00007fb823a4cd9f    /opt/matlab/R2015b/bin/glnxa64/libmwservices.so+03009951
[ 31] 0x00007fb823a4d84c    /opt/matlab/R2015b/bin/glnxa64/libmwservices.so+03012684 _Z25svWS_ProcessPendingEventsiib+00000092
[ 32] 0x00007fb8230a79b8         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00596408
[ 33] 0x00007fb8230a7cd4         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00597204
[ 34] 0x00007fb823093fed         /opt/matlab/R2015b/bin/glnxa64/libmwmcr.so+00516077
[ 35] 0x00007fb8320eddc5                             /lib64/libpthread.so.0+00032197
[ 36] 0x00007fb831e1421d                                   /lib64/libc.so.6+01008157 clone+00000109
[ 37] 0x0000000000000000                                   <unknown-module>+00000000
If this problem is reproducible, please submit a Service Request via:
    http://www.mathworks.com/support/contact_us/
A technical support engineer might contact you with further information.
Thank you for your help.** This crash report has been saved to disk as /home/electromag/matves/matlab_crash_dump.61818-1 **
MATLAB is exiting because of fatal error
/var/spool/torque/mom_priv/jobs/14085786.dccn-l029.dccn.nl.SC: line 3: 61818 Killed                  /opt/matlab/R2015b/bin/matlab -singleCompThread -nosplash -nodisplay -r "restoredefaultpath;addpath('/project/3011085.02/scripts/fieldtrip/qsub');qsubexec('/project/3011085.02/scripts/erfosc/tfa_subj_7_high_onset_j039');exit"