HDF5 File Schema¶

WESTPA stores all of its simulation data in the cross-platform, self-describing HDF5 file format. This file format can be read and written by a variety of languages and toolkits, including C/C++, Fortran, Python, Java, and Matlab so that analysis of weighted ensemble simulations is not tied to using the WESTPA framework. HDF5 files are organized like a filesystem, where arbitrarily-nested groups (i.e. directories) are used to organize datasets (i.e. files). The excellent HDFView program may be used to explore WEST data files.

The canonical file format reference for a given version of the WEST code is described in src/west/data_manager.py.

Overall structure¶

/
    #ibstates/
        index
        naming
            bstate_index
            bstate_pcoord
            istate_index
            istate_pcoord
    #tstates/
        index
    bin_topologies/
        index
        pickles
    iterations/
        iter_XXXXXXXX/\|iter_XXXXXXXX/
            auxdata/
            bin_target_counts
            ibstates/
                bstate_index
                bstate_pcoord
                istate_index
                istate_pcoord
            pcoord
            seg_index
            wtgraph
        ...
    summary

The root group (/)¶

The root of the WEST HDF5 file contains the following entries (where a trailing “/” denotes a group):

Name	Type	Description
ibstates/	Group	Initial and basis states for this simulation
tstates/	Group	Target (recycling) states for this simulation; may be empty
bin_topologies/	Group	Data pertaining to the binning scheme used in each iteration
iterations/	Group	Iteration data
summary	Dataset (1-dimensional, compound)	Summary data by iteration

The iteration summary table (/summary)¶

Field	Description
n_particles	the total number of walkers in this iteration
norm	total probability, for stability monitoring
min_bin_prob	smallest probability contained in a bin
max_bin_prob	largest probability contained in a bin
min_seg_prob	smallest probability carried by a walker
max_seg_prob	largest probability carried by a walker
cputime	total CPU time (in seconds) spent on propagation for this iteration
walltime	total wallclock time (in seconds) spent on this iteration
binhash	a hex string identifying the binning used in this iteration

Per iteration data (/iterations/iter_XXXXXXXX)¶

Data for each iteration is stored in its own group, named according to the iteration number and zero-padded out to 8 digits, as in /iterations/iter_00000001 for iteration 1. This is done solely for convenience in dealing with the data in external utilities that sort output by group name lexicographically. The field width is in fact configurable via the iter_prec configuration entry under data section of the WESTPA configuration file.

The HDF5 group for each iteration contains the following elements:

Name	Type	Description
auxdata/	Group	All user-defined auxiliary data0 sets
bin_target_counts	Dataset (1-dimensional)	The per-bin target count for the iteration
ibstates/	Group	Initial and basis state data for the iteration
pcoord	Dataset (3-dimensional)	Progress coordinate data for the iteration stored as a (num of segments, pcoord_len, pcoord_ndim) array
seg_index	Dataset (1-dimensional, compound)	Summary data for each segment
wtgraph	Dataset (1-dimensional)

The segment summary table (/iterations/iter_XXXXXXXX/seg_index)¶

Field	Description
weight	Segment weight
parent_id	Index of parent
wtg_n_parents
wtg_offset
cputime	Total cpu time required to run the segment
walltime	Total walltime required to run the segment
endpoint_type
status

Bin Topologies group (/bin_topologies)¶

Bin topologies used during a WE simulation are stored as a unique hash identifier and a serialized BinMapper object in python pickle format. This group contains two datasets:

index: Compound array containing the bin hash and pickle length
pickle: The pickled BinMapper objects for each unique mapper stored in a (num unique mappers, max pickled size) array