Virtual Datasets Source remapping or autoupdate

1 Ansicht (letzte 30 Tage)
Hoang-Le Tran
Hoang-Le Tran am 19 Dez. 2023
Bearbeitet: Hoang-Le Tran am 1 Feb. 2024
I am using Matlab's low-level hdf5 functions to map multiple chunks of an image stored in .h5 to a VDS. The problem I am having is when the vds file is moved, the mappings remain the same(expected). I was hoping see if there was a way to have the mappings be dependent on the vds file and not static to the folder from which it was mapped. Something along the line of:
Source files:/1/data_folder
VDS: /1/vds.h5
If the folder was changed to /1_new/ the mappings would be reflect based on the new location of the VDS: /1_new/vds.h5
If this isn't possible, is there a way to remap to new locations without creating a new vds or even just convert a VDS to a regular HDF5?
Edit:
Apparently, HDF5 actually has support for relative path prefix where it can be relative to the VDS itself, which might facilitate a VDS that can auto update its source locations. Documentations below. The challenge, however, is that MATLAB does not currently(as of my knowledge) have support for these low level functions, so I have not gotten a chance to try and test. HDF5 has a C library. Does anybody have an idea of how I can utilize this library in MATLAB?

Akzeptierte Antwort

Hoang-Le Tran
Hoang-Le Tran am 1 Feb. 2024
Bearbeitet: Hoang-Le Tran am 1 Feb. 2024
When mapping the VDS, it is possible to use relative pathing when using H5P.set_virtual() without using the prefix functions. As long as the provided path does not start with '/' in unix, the relative path of source files will be prepended with the current directory of the VDS file. Documentation here: H5Pset_virtual in the 'source file resolutions' section.

Weitere Antworten (1)

Ayush Modi
Ayush Modi am 10 Jan. 2024
Hi,
As per my understanding, you are trying to create a dynamic mapping of multiple chunks of data stored in .h5 file to VDS.
However, VDS mappings are static once created; they don't automatically update if the source files are moved to a new location. This means that if you move the source files or the VDS file to a new location, the VDS mappings will still point to the original file locations and will not reflect the new paths.
However, you can convert VDS to a regular HDF5 file. You can read the data through the VDS and write it to a new regular HDF5 file. This way, you lose the virtual mapping, but you create a standalone HDF5 file that contains all the data and does not depend on the source file locations.
Here is an example to demonstrate how you can accomplish this:
% Read the data from the VDS dataset
data = H5D.read(dset_id, 'H5ML_DEFAULT', 'H5S_ALL', 'H5S_ALL', 'H5P_DEFAULT');
% Creating a new HDF5 file and writing the data
new_file_id = H5F.create('new_regular_file.h5', 'H5F_ACC_TRUNC', 'H5P_DEFAULT', 'H5P_DEFAULT');
dataspace_id = H5S.create_simple(ndims(data), fliplr(size(data)), []);
new_dset_id = H5D.create(new_file_id, '/dataset', 'H5T_NATIVE_DOUBLE', dataspace_id, 'H5P_DEFAULT');
H5D.write(new_dset_id, 'H5ML_DEFAULT', 'H5S_ALL', 'H5S_ALL', 'H5P_DEFAULT', data);
Please refer to the following MathWorks documentation for more information on the following functions:
I hope this resolves the issue you were facing.
  1 Kommentar
Hoang-Le Tran
Hoang-Le Tran am 10 Jan. 2024
Hey Ayush,
Thanks for chiming in. I just found today that HDF5 actually has support for relative path prefix where it can be relative to the VDS itself, which might facilitate a VDS that can auto update its source locations. Documentations below. The challenge, however, is that MATLAB does not currently(as of my knowledge) have support for these low level functions, so I have not gotten a chance to try and test. Any input on how I could call the whole HDF5 library within MATLAB? Maybe through a java wrapper or C Mex. I have no idea on how MATLAB supports external libraries.

Melden Sie sich an, um zu kommentieren.

Produkte


Version

R2022a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by