User's Reference

Import

Category

Import and Export

Function

Reads an external data file.

Syntax

data = Import(name, variable, format, start, end, delta);

Inputs

Name Type Default Description
name string none name of file containing data to be read, or "!command"
variable string or string list format dependent variable to be read
format string file extension or content "dx," "general," "netcdf," "CDF," "hdf," "cm"
start integer first frame first data frame to be imported
end integer last frame last data frame to be imported
delta integer 1 increment between frames

Name	Type	Default	Description
`name`	string	none	name of file containing data to be read, or "!command"
`variable`	string or string list	format dependent	variable to be read
`format`	string	file extension or content	"dx," "general," "netcdf," "CDF," "hdf," "cm"
`start`	integer	first frame	first data frame to be imported
`end`	integer	last frame	last data frame to be imported
`delta`	integer	1	increment between frames

Outputs

Name Type Description
data object object containing requested variables

Name	Type	Description
`data`	object	object containing requested variables

Functional Details

From an external data file this modules creates Data Explorer objects that can be processed by other modules.

name
is the name of the data file being imported. If the parameter specifies an absolute path name, the system attempts to open the file. Otherwise, it first searches the current directory (i.e., the directory from which Data Explorer was invoked) and then, if necessary, the directories specified by the environment variable DXDATA (see C.1 , "Environment Variables" in IBM Visualization Data Explorer User's Guide).
Note: This parameter can also specify an external filter (see External filters).
If name contains a series, the parameters start, end, and delta can be used to import a portion of the data (see parameter descriptions below).
variable
specifies the variable(s) to be imported.
format
specifies the format of the data to be imported. Valid format names are: "dx," "general," "netcdf," "CDF," "hdf," and "cm." These keywords can also be used as extensions on file names.
start and end
specify the first and last data frame to be imported from a data file containing a series.
delta
specifies the increment in counting the data frames in the range from start to end. For example, if the first and last frames are 10 and 20 respectively, and delta = 2, the output data is a series group with six members (frames 10, 12, 14,...).

For Future Reference

If the data set being imported is changed (e.g., by editing) during a Data Explorer session, and if the cache is enabled (the default condition), it may be necessary to reinitialize the Data Explorer executive to access the new data. To do so, select Reset Server in the Connections pull-down menu of the VPE window.
Resetting the server flushes the executive cache. The next time the visual program is invoked, the entire network executes (not just the portions affected by changes) and Import will reaccess the data set.
Specifying that the module's output not be cached has the same effect. Select the appropriate option in:

the "Cache" option menu of the module's configuration dialog box, or
the "Set Output Cacheability" option menu in the Edit pull-down menu.
Note that it may be necessary to apply the same restriction to any module downstream from Import.
To specify that no outputs are to be cached, use the -cache off option when starting Data Explorer.

For Future Reference
If the data set being imported is changed (e.g., by editing) during a Data Explorer session, and if the cache is enabled (the default condition), it may be necessary to reinitialize the Data Explorer executive to access the new data. To do so, select `Reset Server` in the `Connections` pull-down menu of the VPE window. Resetting the server flushes the executive cache. The next time the visual program is invoked, the entire network executes (not just the portions affected by changes) and Import will reaccess the data set. Specifying that the module's output not be cached has the same effect. Select the appropriate option in: the "Cache" option menu of the module's configuration dialog box, or the "Set Output Cacheability" option menu in the `Edit` pull-down menu. Note that it may be necessary to apply the same restriction to any module downstream from Import. To specify that no outputs are to be cached, use the `-cache off` option when starting Data Explorer.

Data Explorer format files. A Data Explorer data file consists of one or more header and data sections that describe the structure and values of user data. The header section is a text description of one or more Data Explorer objects, and the data section is either a text or binary representation of the data values. Non-binary data in data files is limited to a 4K line limit.

If variable specifies more than one object, the module creates a group and each object is added to the group by name. If variable is not specified, the default object is imported. This object can be specified with the default keyword in the Data Explorer file format (see B.2 , "Data Explorer Native Files" in IBM Visualization Data Explorer User's Guide). If it is not specified, the default object is the last object defined in the data file.

Any Data Explorer object in a Data Explorer data file can be specified for import, including Lights, Cameras, and Transforms, as well as more common objects such as Series, Groups, and Fields. The data can be in a separate file from the header, and header and data sections can be interspersed. And the data can be specified in a variety of formats (see see B.2 , "Data Explorer Native Files" in IBM Visualization Data Explorer User's Guide).

General array importer files. You can use the general format described in 5.1 , "General Array Importer" in IBM Visualization Data Explorer QuickStart Guide to import data from various file formats and convert the data to objects. This format allows you to describe the structure of your data so that Data Explorer can create Data Explorer objects from it. If you do not specify a variable, then all variables are imported.

Normally, the name parameter in this case is the general array header file. However, the name parameter can be the data file if the extended form of the format parameter includes the header file as a template. The format parameter can also include any set of keyword-value pairs as a comma-separated list. The specified values are used instead of those in the header file. This format is useful for data files with similar header files where only the size of the data changes. An example for the grid keyword is:

format ="general, template=headerfile, grid=num_x x num_y x num_z ..."

An example parameter for the points keyword is:

format = "general, template=headerfile, points=n"

You may also omit template=headerfile if all the necessary information is specified by the keyword-value pairs.

netCDF files. When the netCDF file is opened, variables matching the variable parameter are read in as field objects. If you give no field name, all fields are read in and placed as separate fields in a group. Each group member is named using the name of the field in the netCDF file. If more than one variable has the same field name, a composite field is created.

You can import both regular and irregular data. If the data are regular, nonzero origins and non-unit spacing can be handled. You can also import scalar, vector, and tensor data. For irregular data, "positions" and "connections" are determined from information in the netCDF variable attributes associated with the field. Additional components can also be read in and added to the field, based on netCDF attribute information.

For a detailed description of the attributes required in a netCDF file, and an example of the correct format, see B.4 , "netCDF Files" in IBM Visualization Data Explorer User's Guide.

CDF files. When the CDF is opened, variables matching the "variable" parameter are read in as fields. If "variable" is not specified, then all variables are imported and placed as fields in a group. Each group member is named using the name of the field (CDF variable) in the CDF. If the CDF contains records, then variable(s) are imported as a series. Some CDF variables become the "positions" component of the field, while others become the "data" component of the field. For a series, the values of the record-varying variable become the "series positions" attribute(s). Variable and global attributes present in the CDF are imported as object attributes. Only CDF r-variables are supported. See IBM Visualization Data Explorer User's Guide for more information on importing data from a CDF.

HDF files. Scientific DataSets are read in as fields. If there are more than one DataSet in the HDF file, you can specify the variable as a number corresponding to the position of the data set (0 corresponds to the first file). If no variable is specified, all fields are read in and placed as separate fields in a group. Each group member is named using the label (if it exists) from the HDF file.

If scales are present, they are interpreted as "positions" with regular "connections." Otherwise, the positions are a regular grid with regular connections. For more information on HDF, see B.6 , "HDF Files" in IBM Visualization Data Explorer User's Guide.

CM files. Import will import saved color-map files. (To save a color map explicitly as a separate .cm file, choose Save As... in the File menu of the ColorMap Editor.)

The imported file will be a group containing the color map as the first field and the opacity map as the second field. (Alternatively, you can import just one of these maps by specifying the variable parameter to Import as "colormap" or "opacity" respectively.)

The color map is a field with a 1-dimensional "positions" component (the data values) and a 3-dimensional "data" component (the colors). Similarly the opacity map is a field with a 1-dimensional "positions" component (the data values) and a 1-dimensional "data" component (the opacities).

You can pass the imported color and opacity maps to (1) the color and opacity tabs of the Color module or (2) the color-map and opacity parameters of the Colormap tool.

When a .cm file is imported, the result is not only information describing the color and opacity maps themselves, but also information specifically intended for the Colormap Editor regarding control points. Users are not expected to create their own .cm files (other than by writing them using the Save As command in the Colormap Editor), as the content of this file is not documented. However users can import any field which has the appropriate color or opacity map structure and use it as input to either the Color or the Colormap tools. For more information on the structure of color and opacity maps, see Color.

External filters. If the first character of the name parameter is "!" (e.g., "!ext2dx mydata.ext mydata.dx"), the rest of the string following the exclamation point is interpreted as a shell command to be executed. The command should be the name of an external filter program with any required arguments. The filter program can be any user-supplied program that reads data from other file formats or generates data, but it must output "dx" or "general array" format as standard output. The Import module waits for the program to execute, reads the output of the program, and imports the objects with the same options as if reading directly from a file.

Example Visual Programs

Nearly every example visual program uses the Import module. Most import Data Explorer format files. Two example programs that import general array format files are:

GeneralImport1.net
GeneralImport2.net

An example program that uses the external filter option is:

ImportExternalFilter.net

An example program that uses the extended form of the format parameter is:

MRI_2.net

[ OpenDX Home at IBM | OpenDX.org ]

`name`	is the name of the data file being imported. If the parameter specifies an absolute path name, the system attempts to open the file. Otherwise, it first searches the current directory (i.e., the directory from which Data Explorer was invoked) and then, if necessary, the directories specified by the environment variable DXDATA (see C.1 , "Environment Variables" in IBM Visualization Data Explorer User's Guide). Note: This parameter can also specify an external filter (see External filters). If `name` contains a series, the parameters `start`, `end`, and `delta` can be used to import a portion of the data (see parameter descriptions below).
`variable`	specifies the variable(s) to be imported.
`format`	specifies the format of the data to be imported. Valid format names are: "dx," "general," "netcdf," "CDF," "hdf," and "cm." These keywords can also be used as extensions on file names.
`start` and `end`	specify the first and last data frame to be imported from a data file containing a series.
`delta`	specifies the increment in counting the data frames in the range from `start` to `end`. For example, if the first and last frames are 10 and 20 respectively, and `delta` = 2, the output `data` is a series group with six members (frames 10, 12, 14,...).