[yam] multi-core problem

[Tom Eulenfeld]

Hello Weijun,

sorry, your mail got somehow lost by the Mailman instance. I attach it 
below.

Regarding your problem:

1. Did you run yam-runtests? Does it show the same error? Which 
operating system are you using?
2. Is your installation up to date? Check yam --version. The latest 
version is 0.3.0.
3. If you are already on the latest version. Can you try out the 
development version of yam? You can install dev with

pip install https://github.com/trichter/yam/archive/master.zip

Recently, I reworked how things are written to the HDF5 file. In version 
0.3.0 and prior versions an extra process was spanned just for writing 
into HDF5 files to circumvent the concurrent writing problem. In the dev 
version writing is done from the main process which is simpler and less 
error prone.

Best,
Tom



-------- Forwarded Message --------

Hello, Yawar,
When I run yam with multi-core, errors frequently appear as a example 
following.  It should be the problem about concurrent writting to hdf5 
file in commands.py.  I am not familar with hdf5, so don't know whether 
the website( http://docs.h5py.org/en/latest/swmr.html) and 
"Multiprocess concurrent write and read" segment  can help.
Thanks,

-----------------------------------------

$ yam correlate 1b

--------------------error message--------------------------------
  20%|████████▌                                 | 75/366 [02:52<11:08, 
2.30s/it]Traceback (most recent call last):
   File 
"/home/wwj/anaconda3/envs/obspy/lib/python3.6/site-packages/h5py/_hl/files.py", 
line 111, in make_fid
     fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
   File "h5py/h5f.pyx", line 78, in h5py.h5f.open
OSError: Unable to open file (unable to lock file, errno = 11, error 
message = 'Resource temporarily unavailable')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "/home/wwj/anaconda3/envs/obspy/bin/yam", line 11, in <module>
     load_entry_point('yam', 'console_scripts', 'yam')()
   File "/home/wwj/old/gits/obspy/yam/yam/main.py", line 388, in run_cmdline
     run(**args)
   File "/home/wwj/old/gits/obspy/yam/yam/main.py", line 147, in run
     run2(command, **args)
   File "/home/wwj/old/gits/obspy/yam/yam/main.py", line 211, in run2
     yam.commands.start_correlate(io, **args)
   File "/home/wwj/old/gits/obspy/yam/yam/commands.py", line 168, in 
start_correlate
     _write_stream(result)
   File "/home/wwj/old/gits/obspy/yam/yam/commands.py", line 156, in 
_write_stream
     result[key].write(io[key], 'H5', mode='a')
   File 
"/home/wwj/anaconda3/envs/obspy/lib/python3.6/site-packages/obspy/core/stream.py", 
line 1443, in write
     write_format(self, filename, **kwargs)
   File 
"/home/wwj/anaconda3/envs/obspy/lib/python3.6/site-packages/obspyh5.py", 
line 186, in writeh5
     with h5py.File(fname, mode, libver='latest') as f:
   File 
"/home/wwj/anaconda3/envs/obspy/lib/python3.6/site-packages/h5py/_hl/files.py", 
line 269, in __init__
     fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
   File 
"/home/wwj/anaconda3/envs/obspy/lib/python3.6/site-packages/h5py/_hl/files.py", 
line 113, in make_fid
     fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
   File "h5py/h5f.pyx", line 98, in h5py.h5f.create
OSError: Unable to create file (unable to open file: name = 'corr.h5', 
errno = 17, error message = 'File exists', flags = 15, o_flags = c2)


--
Weijun Wang

Institute of Earthquake Forecasting, China Earthquake Administration
Beijing, China