News
- 2007-03-24
Version 0.9.1 is out! The most notable change is improved ease of installation, particularly on Windows platforms. The Windows installer now requires no dependencies. Both the Linux and Windows versions ship with the necessary OpenG components, so they do not need to be installed separately. (However, I still recommend the OpenG libraries to you, as they are quite useful.) The Windows installer will also install the HDF5 DLLs, if desired.
There are other changes, including the addition of a datalogging-type interface to the datasets, improved attribute handling, high level interfacing to LabVIEW tree controls, and more. Please see the README for all changes.
- Designing the format
- Coding the format
- Writing an entire program everytime you want to extract any portion of the data
- Handling modifications when you add that extra piece of data you didn't consider at first
- Documenting the format
- Remembering where you put the documentation for the format
- Automatically create HDF5 datatypes for most LabVIEW data types
- Write datasets comprised of most LabVIEW data types, converting the data to a native HDF5 type.
- Read dataset comprised of any HDF5 type that it can write including variations certain attributes such as endianness and string type
- Read and write attributes
- Appropriately handle HDF5 dataspaces
- Perform the full range of group operations, such as creating groups, listing group contents, creating links, unlinking objects, etc.
- Handle HDF5 object and dataset region references
- Handle certain LabVIEW data types
- Call HDF5 functions which required callback functions
- Most anything efficiently
Overview
The LabVIEW HDF5 library is a VI library for interfacing LabVIEW applications to the NCSA's HDF5 scientific data storage format. It is designed to make the storage of arbitrarily complex LabVIEW data simple. It provides an easy, high-level interface for working with HDF5 files built upon an nearly complete low level interface to the HDF5 library.
Motivation
As a researcher, I often deal with large quantities of data. I found neither of the traditional two choices offered by LabVIEW (and most other programming languages) to be sufficient for storing this data. Delimeted text files are great for simple data, but are difficult to manage if there are more than two dimensions of data, or if there are many ancillary data that must be stored along with the main data set. Writing one's own binary format for each experiment avoids these problems, but introduces the following headaches:
Figuring that I wasn't the first person to encounter this delima, I took to Google and discovered several libraries had indeed been written to handle complex data sets in a consistent fashion. After evaluating a few of them, I decided that NCSA's HDF5 format had all of the features for which I was looking.
Unfortunately, there was no LabVIEW interface for the library. Actually, National Instruments had produced one, but it has several shortcomings. It does not work with the most recent version of HDF5, it only functions on Microsoft platforms, it does not export many of the HDF5 library functions and, most importantly, it did not look sufficiently easy to use. I knew that I was only likely to use HDF5 if I could make it simple to program. In fact, I wanted it to be at least as simple as saving a delimited text file (a very trivial task in LabVIEW), yet still allow for complex data to be stored.
So, I designed my own interface LabVIEW interface to HDF5 that was easy to use.
How easy is it?
Pictured below is a block diagram that demonstrates simple use of the HDF5 library. Admittedly the data in this example could fairly easily be stored in a text file or binary file. That might not be the case if the attribute list were longer, or there were many such datasets that needed to be stored together in an organized fashion.
This example shows how one might save several arrays of heterogeneous data, such as that generated by Gen Data. It is likely that real data would be acquired from different instruments, and therefore would initially be handled as multiple arrays. The data is converted into one array of a cluster using Index & Bundle Cluster Array. This representation is a more natural format for this type of data. Unfortunately, Index & Bundle does not create cluster member labels, so we must do that manually with a type case. The cast-to type is identical to the input other than the member labels.
Once the data is in the desired format, writing it to the HDF5 file is trivial. Open a file, write the dataset (and the attributes), close the file. Don't forget to check for errors.
Here is an image of the resulting data file, as displayed by HDFView, a free HDF viewer/editor available from the NCSA.
What can LVHDF5 do?
What can't LVHDF5 do?