QuickStart Guide - Importing Data: Header File Examples

5.2 Importing Data: Header File Examples

The examples in this section are divided into three groups: single variable (a simplified case of record style), record style, and spreadsheet style. A review of these examples will provide a good grounding in the use of the Data Prompter and the creation of header files for importing data with the General Array Importer.

The examples in the first group are generally more detailed than those in the second and third groups. Since examples often build on previous examples, it is recommended that you start at the beginning of a group.

The instructional sequence in each example begins with the initial dialog box of the Data Prompter. Most examples use the Data Prompter to create a header file and each example shows the header file produced. (For the syntax of keyword statements in a header file, see 5.3 , "Header File Syntax: Keyword Statements".) The command that invokes the Data Prompter and generates the initial dialog box is:

dx -prompter

Record Style: Single-Variable Data

It is recommended that you treat the first four examples as a unit and review them in sequence.

Example 1. Scalar Data on a Regular Grid

This example illustrates how a simple floating-point scalar field, on a regular grid, might be imported from a text file named "record_scalar". The origin of the grid is [1 3 2], with deltas of 0.5, 1, and 0.75 in the x, y, and z directions respectively.

In the Data Prompter initial dialog box, choose Grid or Scattered File; then select the leftmost button in the row labeled Grid type (regular grid).
Confirm that Number of variables is set to "1" and that the Single time step toggle button is activated.
For Data organization, confirm that the Block (i.e., record) toggle button is activated.
Click on Describe Data to bring up the "simplified" prompter.
Enter the path name of the data file in the Data file field:
```
/usr/local/dx/samples/data/record_scalar
```
Browse the data file if you like by choosing Browser from the ellipses to the right of the Data Field field.
Enter the values "5," "8," and "6," in that order, in the first three Grid size fields. The number of data values is 240 (i.e., 5 × 8 × 6).
Confirm that Data format is set to ASCII (text).
Confirm that Data order is set to Row.
Finally, set the first three origin-delta pairs (in the Grid positions section of the prompter) to: 1.0, 0.5; 3.0, 1.0; and 2.0, 0.75, in that order.
Since the data order is Row (i.e., last index varies fastest), the first six data values are associated with positions [1 3 2], [1 3 2.75], [1 3 3.5 ], [1 3 4.25], [1 3 5], and [1 3 5.75 ]. (If the data are stored so that "last index varies slowest," Data order should be set to Column.)
To save the header file:
1. Select Save As... from the File pull-down menu.
2. When the Save a Data Prompter Header File... dialog box appears, position the cursor in the Selection field (at the point indicated by the carat).
3. Enter the name of the file (record_scalar).
4. Click on OK.
The file is saved in your current directory with the extension ".general" and should look like:

file = /usr/local/dx/samples/data/record_scalar
grid = 5 x 8 x 6
format = text
interleaving = record
majority = row
field = field0
structure = scalar
type = float
dependency = positions
positions = regular, regular, regular, 1.0, 0.5, 3.0, 1.0, 2.0, .75
end

Note the information that you have supplied directly (lines 1, 2, and 10). You can visualize the data file using the Visualize Data button in the initial Data Prompter window.

Example 2. Cell-centered Data

This example involves modifying the header file created in Example 1. The important difference is that the data here is cell-centered (connection dependent): instead of 240 data values (one for each of the 5 x 8 x 6 positions), there are 140 values (one for each of the 4 x 7 x 5 connections). The format is binary.

In the Data Prompter initial dialog box, click on Grid or Scattered File, then Describe Data, to bring up the simplified prompter.
Select Open from the File pull-down menu. A new dialog box appears called "Open a Data Prompter Header...."
In the Directories column, highlight the directory in which you saved the record_scalar header file (if it is not already highlighted).
Highlight record_scalar.general in the File column, and then click on OK. The simplified prompter now displays the information for the record_scalar header file.

Change the path name in the Data file field to:

/usr/local/dx/samples/data/record_depconnections

Select Full prompter from the Options pull-down menu.
When the Full prompter dialog box appears, change Dependency from "positions" to "connections." To confirm this change, click on Modify at the bottom of the dialog box (note the instruction there).
Repeat Example 1, Step 10 (see above), to save the header file, which should look like:

file = /usr/local/dx/samples/data/record_depconnections
grid = 5 x 8 x 6

format = text

interleaving = record
majority = row
field = field0
structure = scalar
type = float
dependency = connections
positions = regular, regular, regular, 1, .5, 3, 1.0, 2.0, .75
end

Note the information that you have supplied directly or changed (lines 1, 2, 3, and 9).

Example 3. Data with Header information

A data file may contain descriptive information in addition to the data to be imported. To import only the data, therefore, it is necessary to "skip" such information when the file is read. The header keyword statement enables you to do just that, by specifying a number of bytes or lines to be skipped or a string to be searched for. For example, suppose the scalar data field of Example 1 had 3 lines of descriptive text preceding the data.

As in Example 2 (Steps 2 through 4), open the record_scalar header file.

Change the path name in the Data file field to:

/usr/local/dx/samples/data/record_withheader

Activate the Header toggle button and then click on the option button just to the right of it.
From the list that appears, select # of lines and enter the value "3" in the associated text field.
Repeat Example 1, Step 10 (see To save the header file), to save the header file, which should look like:

file = /usr/local/dx/samples/data/record_withheader
grid = 5 x 8 x 6
format = text
interleaving = record
majority = row
header = lines 3
field = field0
structure = scalar
type = float
dependency = positions
positions = regular, regular, regular, 1, .5, 3, 1.0, 2.0, .75
end

Note the addition of a header keyword statement (line 6).

Example 4. Naming a Field

By default, the Data Prompter names data fields in numerical order: field0, field1, and so on. But a data field can be named with a field keyword statement.

Once the data are imported into Data Explorer, you can, for example, extract the name (using the Attribute module) and include it in a caption (using the Caption module). So if there are two types of data (e.g., temperature and pressure), each can be automatically and appropriately labeled with an identifying name, thereby "tagging" the associated data for future reference. As a result, it is also possible to import a field by name when there is more than one field.

For this example suppose that the data in Example 1 are temperature values (see To save the header file).

Open the record_scalar header file, as in Example 2.
In the right-hand panel of the prompter, change Field name from "Field0" to "Temperature." To confirm this change, click on Modify at the bottom of the dialog box (note the instruction there).
Repeat Example 1, Step 10 (see To save the header file), to save the header file, which should look like:

file = /usr/local/dx/samples/data/record_scalar
grid = 5 x 8 x 6
format = text
interleaving = record
majority = row
field = Temperature
structure = scalar
type = float
dependency = positions
positions = regular, regular, regular, 1, .5, 3, 1.0, 2.0, .75
end

Note the change in the field keyword statement (line 6).

Example 5. Deriving Grid Information from a Data File

Being able to derive grid information directly from a data file is particularly useful if you import data with a standard format but with grid dimensions that vary from data set to data set. For example, if the first line of the data file is:

      dimensions 100 300

you can use any of the following grid keyword statements to obtain the grid dimensions from the data file.

grid = lines 0, 11, 3, 1, 3
This statement says
1. Skip 0 (zero) lines of the file.
2. Skip 11 characters (the word "dimensions" and one space).
3. Read the first dimension from 3 characters.
4. Skip 1 character.
5. Read the second dimension from 3 characters.
grid = bytes 11
This statement says to skip 11 characters and begin reading.
grid = marker "dimensions"
This statement says to start reading after the string marker "dimensions."

See "grid". See also B.1 , "General Array Importer: Keyword Information from Data Files" in IBM Visualization Data Explorer User's Guide.

Note: This derivation feature is not available with the Data Prompter.

Examples 6 and 7. Vector Data

The General Array Importer supports three representations, or "styles," of vector data: record, record-vector, and series-vector. The first two are illustrated here. For the third, see "interleaving".

Which representation matches the data depends on a characteristic called interleaving. In record interleaving, the data for each vector component are stored together in individual blocks (e.g., X₀, X₁,..., X_n, Y₀, Y₁,..., Y_n). In record-vector interleaving, the components of each vector are stored consecutively (e.g., X₀Y₀, X₁Y₁,..., X_nY_n).

The following pair of examples illustrates the differences between the two representations and between the header files used to import them. The header files are identical in that they both specify a unit 2-vector that parallels the x-axis and is defined on a 5 x 4 regular grid. That is, the data consists of 20 instances of the vector [1 0].

In Example 7, the interleaving style of the data file is record:

1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
1 1 1 1
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0

In the Data Prompter simplified dialog box, select New in the File pull-down menu (this selection clears the dialog box of any information from a previous file).
Enter the path name of the data file in the Data file field:
```
/usr/local/dx/samples/data/record_vectordata1.
```
Enter the values "5" and "4," in that order, in the first two Grid size fields. (Note that the origin-delta values default to [0 1].)
Add a field to the Field list by typing a name (e.g. field0) in the Field name text field, and then pressing the Add button.
Change Type (right-hand panel) to "int."
Note: To implement this change, you must click on Modify at the bottom of the dialog box. However, you can delay implementation to Step 5, and implement both steps at the same time.
Change Structure (right-hand panel) to "2-vector." To implement this change, click on Modify at the bottom of the dialog box (note the instruction there).
Step 5 activates the Vector interleaving button. Select X₀,X₁,...,X_n,Y₀,Y₁,.. .,Y_n (the notation used for record style).
Repeat Example 1, Step 10 (see To save the header file), to save the header file, which should look like:

file = /usr/local/dx/samples/data/record_vectordata1
grid = 5 x 4
format = text
interleaving = record
majority = row
field = field0
structure = 2-vector
type = int
dependency = positions
positions = regular, regular, 0, 1, 0, 1
end

In Example 7, the interleaving style of the data file is record-vector:

1 0 1 0 1 0 1 0 1 0
1 0 1 0 1 0 1 0 1 0
1 0 1 0 1 0 1 0 1 0
1 0 1 0 1 0 1 0 1 0

If you have not closed the simplified dialog box from Example 5, all you need do is:
1. Change the path name for the data file (in the Data file field) to:
```
/usr/local/dx/samples/data/record_vectordata2.
```
2. For Vector interleaving, select X₀Y₀,X₁Y₁,...,X_nY_n (the notation used for record-vector style).
Otherwise you can repeat Example 6 (or open the header file record_vectordata1) and make the appropriate changes.
Repeat Example 1, Step 10 (see To save the header file), to save the new header file, which should look like the one shown at the top of the next page.

file = /usr/local/dx/samples/data/record_vectordata2
grid = 5 x 4
format = text
interleaving = record-vector
majority = row
field = field0
structure = 2-vector
type = int
dependency = positions
positions = regular, regular, 0, 1, 0, 1
end

Note: If the interleaving is not specified, the default is record-vector.

Example 8. Series Data

This example illustrates how a 7-step time series of a single scalar field might be imported. The field is on a regular 5 × 5 grid, the data are connections dependent, and the style is record.

# Time-Series Data
Time Step 1
12  9 14  1 10 16  7 20
19  6 11 15 18  8 13 17
Time Step 2
12  9  1 21 10 16  7  1
19  6 11 15 18  8 13 17
Time Step 3
12  1 14 21 10 16  1 20
19  6 11  1 18  8 13 17
Time Step 4
 1  9 14 21 16  1  7 20
19  6  1 15 18  8 13  1
Time Step 5
12  9 14 21  1 16  7 20
 1  6 11 15 18  1 13 17
Time Step 6
12  9 14 21 10 16  7 20
 1  6 11 15 18  1 13 17
Time Step 7
12  9 14 21 10 16  7 20
19  6 11 15  1  8 13 17

In the Data Prompter initial dialog box, select Grid or Scattered File, select the leftmost button in the Grid type row (regular grid) and deactivate the Single time step toggle button.
Confirm that the Block toggle button is activated and click on Describe Data. The Full prompter dialog box appears. (Because this example requires the full prompter, the simplified dialog box is not accessible.)
Enter the path name of the data file in the Data file field:
```
/usr/local/dx/samples/data/record_series
```
Activate the Header toggle button.
Step 4 also enables the # of bytes button to the right. Click on this button and select string marker.
Enter "Time Step 1 \n" in the associated text field to specify that the data file is to be read starting with the line after "Time Step 1" (see Note). Alternatively, selecting # of lines and specifying the value "2" in the text field would produce the same result.
Note: The new-line character "\n" must be included in the specification, and the spacing between it and the marker must match that in the data file (e.g., if "Time Step 1" and "\n" are separated by three spaces in the file, they must be separated by three spaces in the specification). This spacing is easily determined in the Data Browser by placing the cursor at each point and reading the corresponding offset value (see Figure 18).
Enter the value "5" in each of the first two Grid size fields.
Activate the Series toggle button and specify the number of series members by entering the value "7" in the associated n field. (Leave the start and delta fields unchanged.)
Activate the Series separator toggle button, select # of lines, and enter the value "1" in the associated text field. (When the Import module reads the data file, it will skip the lines "Time Step 2," "Time Step 3," and so on.)
Change Dependency from "positions" to "connections." To confirm this change, click on Modify at the bottom of the dialog box (note the instruction there).
Repeat Example 1, Step 10 (see To save the header file), to save the header file, which should look like:

file = /usr/local/dx/samples/data/record_series
grid = 5 x 5
format = text
interleaving = record
majority = row
header = marker "Time Step 1 \n"
series = 7, 1, 1, separator = lines 1
field = field0
structure = scalar
type = float
dependency = connections
positions = regular, regular, 0, 1, 0, 1
end

Note: For scalar data, as in this example, the interleaving keyword is not required (it defaults to record). However, when series data include vectors, this keyword must be included and the appropriate value specified. For more information, see "interleaving".

Example 9. Data and Header in the Same File

The header and end keywords make it possible to combine header information and data in the same file.

Note: Although the General Array Importer is designed to process files that contain both header information and data, the Data Prompter is not. It cannot create them or read them in. Such files, like the one in this example, must be created with an editor.

# The Importer disregards this line, since it is a comment line.
grid = 5 x 5
dependency = connections
type = byte
structure = scalar
format = ascii
header = marker "Start data \n"
end
# There may be comments about the data here (e.g., who created it and
# when).  These will be passed over because of the marker specified
# in the header keyword statement.
Start data
17811 41218 3 9
1021913 1 71420

The end keyword marks the end of the header section. The header keyword statement specifies "Start data" as the search string and the next line as the start of the actual data. Note that if the data starts on the same line, the new-line character (\n) is not required as part of the marker (see also Step 6 of Example 8 in Enter "Time Step 1 \n").

The "positions" keyword, omitted in this example, defaults to an origin of [0 0] and deltas of [1 1].

Record Style: Multivariable Data

To import record-style data, you must set the interleaving keyword to record, record-vector, or series-vector. (When using the Data Prompter, select Block for Field interleaving.) If the data includes vectors, select the appropriate vector interleaving, as discussed in "Examples 6 and 7. Vector Data"; see also "interleaving".

Example 1. Multiple Scalar Fields

This example illustrates the importation of multiple scalar fields. The grid is 4 × 2 × 3, with an origin of [0 0 0] and deltas of [1 1 1]. The three data variables are scalar. The data file looks like:

Energy
    2.158719      1.45419     1.566509     1.551361     2.215095     1.726923
    2.080461     1.418617     1.373206     2.231642     1.316575     1.445211
    1.673182     1.445737     1.820333     2.167849     1.721611     1.554906
    1.604594     2.061092     1.398391     2.062042     1.996196      1.50964
Pressure
    34.81398     18.81529     29.65139       42.499     22.96053     31.41604
    19.92936     27.79935     26.34873     28.91081     21.17855     28.89354
    6.320079      43.9068     6.597938     20.41342     14.83351     43.53309
    16.36901     18.19812     4.628566     43.64742     44.99699     26.32183
Temperature
    295.3329     302.5431      301.835     296.0127     297.8344     295.5451
    301.6786     298.4496     302.0944     296.7458     296.3459     296.4179
    303.1223     300.3094     297.9714     300.0774     299.1322     296.9368
     302.096     294.8137      300.662     299.5744     304.1986     302.4216

The header file to import this data should look like:

file = /usr/local/dx/samples/data/record_multiscalar
grid = 4 x 2 x 3
format = text
interleaving = record
majority = row
header = lines 1
field = Energy, Pressure, Temperature
recordseparator = lines 1
end

Example 2. Cell-Centered Data

This example is identical to the preceding one except that each of the data variables is dependent on the connections between data points rather than on their positions. Thus there are only six data values per field (3 × 1 × 2). The data file looks like:

Energy
    2.158719      1.45419     1.566509     1.551361     2.215095     1.726923
Pressure
    34.81398     18.81529     29.65139       42.499     22.96053     31.41604
Temperature
    295.3329     302.5431      301.835     296.0127     297.8344     295.5451

The header file to import this data should look like:

file = /usr/local/dx/samples/data/record_multiscalardepconn
grid = 4 x 2 x 3
format = test
interleaving = record
majority = row
header = lines 1
field = Energy, Pressure, Temperature
recordseparator = lines 1
dependency = connections
end

Example 3. Multiple Scalars with Mixed Dependencies

This example differs from the preceding one in that Energy and Temperature are dependent on the positions of the grid, while Pressure is dependent on the grid elements (connection dependent). The data file looks like:

Energy
    2.158719      1.45419     1.566509     1.551361     2.215095     1.726923
    2.080461     1.418617     1.373206     2.231642     1.316575     1.445211
    1.673182     1.445737     1.820333     2.167849     1.721611     1.554906
    1.604594     2.061092     1.398391     2.062042     1.996196      1.50964
Pressure
    34.81398     18.81529     29.65139       42.499     22.96053     31.41604
Temperature
    295.3329     302.5431      301.835     296.0127     297.8344     295.5451
    301.6786     298.4496     302.0944     296.7458     296.3459     296.4179
    303.1223     300.3094     297.9714     300.0774     299.1322     296.9368
     302.096     294.8137      300.662     299.5744     304.1986     302.4216

The header file looks like the one shown at the top of the next page.

file = /usr/local/dx/samples/data/record_multiscalarmixed
grid = 4 x 2 x 3
format = text
interleaving = record
majority = row
header = lines 1
field = Energy, Pressure, Temperature
dependency = positions, connections, positions
recordseparator = lines 1
end

Examples 4 and 5. Scalar and Vector Data

This example uses the same grid as the previous 3, but here the second data field (velocity) consists of 2-vectors. In Example 4, all the x-components of the 2-vectors are listed first, followed by all the y-components. For example, the x- and y-components of the first 2-vector are 34.81398 and 2.158719, respectively.

Energy
    2.158719      1.45419     1.566509     1.551361     2.215095     1.726923
    2.080461     1.418617     1.373206     2.231642     1.316575     1.445211
    1.673182     1.445737     1.820333     2.167849     1.721611     1.554906
    1.604594     2.061092     1.398391     2.062042     1.996196     1.509640
Velocity
    34.81398     18.81529     29.65139       42.499     22.96053     31.41604
    19.92936     27.79935     26.34873     28.91081     21.17855     28.89354
    6.320079      43.9068     6.597938     20.41342     14.83351     43.53309
    16.36901     18.19812     4.628566     43.64742     44.99699     26.32183
    2.158719      1.45419     1.566509     1.551361     2.215095     1.726923
    2.080461     1.418617     1.373206     2.231642     1.316575     1.445211
    1.673182     1.445737     1.820333     2.167849     1.721611     1.554906
    1.604594     2.061092     1.398391     2.062042     1.996196     1.509640
Temperature
    295.3329     302.5431      301.835     296.0127     297.8344     295.5451
    301.6786     298.4496     302.0944     296.7458     296.3459     296.4179
    303.1223     300.3094     297.9714     300.0774     299.1322     296.9368
    302.0960     294.8137      300.662     299.5744     304.1986     302.4216

The header file should look like:

file = /usr/local/dx/samples/data/record_scalarvector1
grid = 4 x 2 x 3
format = text
interleaving = record
majority = row
header = lines 1
field = Energy, Velocity, Temperature
structure = scalar, 2-vector, scalar
recordseparator = lines 1, lines 0, lines 1
end

Note that the interleaving specified for the vectors (line 4) is record (see "interleaving") and that the record separator (line 9) specifies: one (1) line separating the Energy and Velocity data; no lines separating the records containing the components of the Velocity data; and one (1) line separating the Velocity and the Temperature data (see "recordseparator").

The data values in Example 5 are the same as those in Example 4, but the components of each vector in the Velocity field appear together (e.g., 34.813980 is followed by 2.158719 in the same row):

Energy
   2.158719   1.454190     1.566509   1.551361     2.215095   1.726923
   2.080461   1.418617     1.373206   2.231642     1.316575   1.445211
   1.673182   1.445737     1.820333   2.167849     1.721611   1.554906
   1.604594   2.061092     1.398391   2.062042     1.996196   1.509640
Velocity
  34.813980   2.158719    18.815290   1.454190    29.651390   1.566509
  42.499001   1.551361    22.960529   2.215095    31.416040   1.726923
  19.929359   2.080461    27.799351   1.418617    26.348730   1.373206
  28.910810   2.231642    21.178551   1.316575    28.893539   1.445211

   6.320079   1.673182    43.906799   1.445737     6.597938   1.820333
  20.413420   2.167849    14.833510   1.721611    43.533089   1.554906
  16.369011   1.604594    18.198120   2.061092     4.628566   1.398391
  43.647419   2.062042    44.996990   1.996196    26.321831   1.509640
Temperature
 295.332886  302.543091  301.834991  296.012695  297.834412  295.545105
 301.678589  298.449585  302.094391  296.745789  296.345886  296.417908
 303.122314  300.309387  297.971405  300.077393  299.132202  296.936798
 302.096008  294.813690  300.661987  299.574402  304.198608  302.421600

The header file should look like:

file = /usr/local/dx/samples/data/record_scalarvector2
grid = 4 x 2 x 3
format = text
interleaving = record-vector
majority = row
header = lines 1
structure = scalar, 2-vector, scalar
field = Energy, Velocity, Temperature
recordseparator = lines 1
end

Note that the interleaving specified for the vectors (line 4) has been changed to record-vector and that the record separator (line 9) specifies one (1) line separating successive records.

Example 6. Deformed (Warped) Regular Grid

A deformed regular grid (sometimes referred to as a warped grid) is one in which the positions are irregular but the connections are regular. In this example the grid is 5 × 4. The data consists of three records, the first two of which contain scalar data defined on the grid. The third contains 2-vector values defining the grid positions. The Data Prompter uses the reserved word locations as a field name for the x,y values of the grid positions. The data file contains no descriptive information.

In the Data Prompter initial dialog box, click on Grid or Scattered File, then on the button representing deformed data (third from the left) in the row labeled Grid type.
Set Number of variables to "2."
Step 1 automatically activates the Positions in data file toggle button (the positions of warped regular data are assumed to be listed in the data file) and displays a Dimension stepper button. Set the Dimension value to "2" for 2-dimensional (x,y) data. .
For Data organization, confirm that the Block (i.e., record) toggle button is activated.
Click on Data Prompter to bring up the simplified data prompter.
Enter the path name of the data file in the Data file field:
```
/usr/local/dx/samples/data/record_deformed
```
Enter the values "5," and "4," in that order, in the first two Grid size fields.
Confirm that Data format, Data order, and Vector interleaving are set respectively to Text, Row, and record-vector (X₀Y₀, X₁Y₁, ...).
In the Field list, click on "locations" and use the "Move field" stepper arrows to position it after "field1" (i.e., at the bottom of the list). This change is necessary to reflect the actual data file, where the two scalar fields precede the x,y positions. (By default, the Data Prompter lists "locations" as the first field.)
Click on "field0" in the list and then change the name to "rainfall" in the Field name field. To confirm this change, click on Modify at the bottom of the dialog box (note the instruction there).
Now change "field1" to "temperature" and the Type option from "float" to "int" (the temperature values are integers). These changes can be confirmed together by clicking on Modify at the bottom of the panel.
Save the header file (Step 10 of Example 1 in To save the header file), which should look like:

file = /usr/local/dx/samples/data/record_deformed
grid = 5 x 4
format = text
interleaving = record-vector
majority = row
field = rainfall, temperature, locations
structure = scalar, scalar, 2-vector
type = float, int, float
dependency = positions, positions, positions
end

Example 7. Scattered Data

This example illustrating the importation of scattered data differs from Example 6 in only a few details, mainly in specifying the number of data points instead of the dimensions of a data grid.

In the Data Prompter initial dialog box, select Grid or Scattered File, then click on the rightmost button (scattered data) in the row labeled Grid type.
Set Number of variables to "2."
Activate the Positions in data file toggle button and set the Dimension value to "2".
Repeat Steps 5 through 11 of Example 6, with the following exception: in Step 7, enter the value "20" in the # of points field (this change from Example 6 is a result of the original selection for scattered data in the initial dialog box).

The header file should look like:

file = /usr/local/dx/samples/data/record_deformed
points = 20
format = text
interleaving = record-vector
field = rainfall, temperature, locations
structure = scalar, scalar, 2-vector
type = float, int, float
dependency = positions, positions, positions
end

Example 8. Using the Block Keyword

The block keyword is used with record-style, fixed-format ASCII data to skip information in a block of data. For example, consider the following data file:

row 1 temperature 39 29 33 56 32
row 2 temperature 32 33 25 33 22
row 3 temperature 31 23 41 53 19
row 4 temperature 43 59 43 21 28
row 5 temperature 23 19 35 46 32

In the Data Prompter initial dialog box, select Grid or Scattered File, then ensure that the leftmost button in the row labeled Grid type is selected.
Click on Data Prompter to bring up the simplified prompter.
Enter the path name of the data file in the data file field:
```
/usr/local/dx/samples/data/block_example.data
```
Activate the header toggle button, reset # of bytes to # of lines, and type 1 in the associated field.
Set Grid size to 5 × 5.
Now click on Full prompter in the Options pull-down menu.
Activate the Block toggle button (right-hand panel).
Set skip to 17, # elem to 5, and width to 3.
Repeat Example 1, Step 10 (To save the header file), to save the header file, which should look like:

file = /usr/local/dx/samples/data/block_example.data
grid = 5 x 5
format = text
interleaving = record
majority = row
header = lines 1
field = field0
structure = scalar
type = int
dependency = positions
block = 17, 5, 3
positions = regular, regular, 0, 1, 0, 1
end

The block statement instructs the importer to skip 17 characters and read 5 (temperature) values (per line in this case), reading each value from a field of three characters.

Columnar Style

Importing columnar-style data requires setting the interleaving keyword to "field": Activate the Columnar toggle button in the Data Prompter initial dialog box or select "Field" for the Field interleaving option in the full prompter.

Example 1. Scalar and Vector Data on a Regular Grid.

This example illustrates the importation of a data file that contains two variables (pressure and velocity) in spreadsheet style. The data are in row majority order (last index varies fastest) and organized in four columns: the first contains the pressure data; the other three, the velocity components. The grid is 5 × 8 × 6.

In the Data Prompter initial dialog box, select Grid or Scattered File, then select the leftmost button in the row labeled Grid type (regular grid).
Set Number of variables to "2."
For Data organization, activate the Columnar toggle button.
Click on Describe Data to bring up the simplified prompter.
Enter the path name of the data file in the Data file field:
```
/usr/local/dx/samples/data/spreadsheet_2var
```
Enter the values "5," "8," and "6," in that order, in the first three Grid size fields.
In the Field list, change the name "field0" to "pressure" and confirm the change by clicking on the Modify button at the bottom of the panel.
Change the name of "field1" to "velocity" and its Structure to "3-vector." (See Steps 9 through 11 of Example 6 in the preceding section, "Example 6. Deformed (Warped) Regular Grid", for procedure.)
Save the header file (Step 10 of Example 1 in To save the header file), which should look like:

file = /usr/local/dx/samples/data/spreadsheet_2var
grid = 5 x 8 x 6
format = text
interleaving = field
majority = row
field = temperature, velocity
structure = scalar, 3-vector
type = float, float
dependency = positions, positions
positions = regular, regular, regular, 0, 1, 0, 1, 0, 1
end

Example 2. Deformed (Warped) Regular Grid

This example differs from Example 6 in the preceding section ("Example 6. Deformed (Warped) Regular Grid") in its data style (spreadsheet), smaller data grid (5 × 4), and number of variables (1). Follow the first 7 steps of that example, except for the following:

In Step 2, set Number of variables to "1."
In Step 4, activate the Columnar toggle button.

In Step 6, use the path name:

/usr/local/dx/samples/data/spreadsheet_deformed.

In Step 7, use the values "3" and "4."

The header file should look like:

file = /usr/local/dx/samples/data/spreadsheet_deformed
grid = 3 x 4
format = text
interleaving = field
majority = row
field = locations, field0
structure = 2-vector, scalar
type = float, float
end

Example 3. Scattered Scalar Data

This example uses the same data set as Example 2 but treats the values as scattered data points. The data file contains an x,y position followed by a data value. There are no implied connections for these data.

In the Data Prompter initial dialog box:
1. Select Grid or Scattered File, then select for scattered data (rightmost grid button).
2. Activate the Position toggle button.
3. Set Dimension to "2."
4. Activate the Spreadsheet toggle button.
5. Click on Describe Data to bring up the simplified data prompter.
Enter the path name used in Example 2.
Enter "12" in the # of points field.
Save the header file (Step 10 of Example 1 in To save the header file), which should look like:

file = /usr/local/dx/samples/data/spreadsheet_deformed
points = 12
format = text
interleaving = field
field = locations, field0
structure = 2-vector, scalar
type = float, float
end

Example 4. Handling Interspersed Text

The layout keyword is used to specify which locations in a data file are to be read, thereby avoiding interspersed text. In the example data file shown here, there are no implied connections between data values.

In the Data Prompter initial dialog box:
1. Select Grid or Scattered File, then select for scattered data (rightmost grid button).
2. Activate the Position toggle button.
3. Set Number of variables to "1."
4. The data positions (latitude and longitude) are in the data file and are 2-dimensional: set Dimension to "2."
5. Activate the Columnar toggle button.
6. Click on Describe Data to bring up the simplified data prompter.

In the Data file field enter the path name:

/usr/local/dx/samples/data/CO2fragment.lis

Activate the Header toggle button, select the "String marker" option, and enter "CO2_CONC \n" in the associated text field. (Note that the marker text is the heading for the third column of data and that the reading of data will start after the new-line character, at the point marked by "*" on the following line. See also Step 6 of Example 7 in Enter "Time Step 1 \n".)

Note: The asterisk (*) at the beginning of the first data line and the interval scale following the data are for reference purposes only and do not appear in the actual file (see Steps 3 and 5f-j in this example).

 VARIABLES AND SPECIFIED RANGES
_________________________________________________________________________
 EPOCH                01-Jul-1983 00:00:00.000 31-Dec-1987 00:00:00.000
 LATITUDE                               -90.00                    90.00
 LONGITUD                              -180.00                   180.00
 CO2_CONC                             -10000.0                  10000.0
         EPOCH                  LATITUDE      LONGITUD      CO2_CONC
*01-Jul-1983 00:00:00.000        -37.95         77.53         341.4
 01-Jul-1983 00:00:00.000        -89.98        -24.80         341.0
 01-Jul-1983 00:00:00.000         -7.92        -14.42         343.4
 01-Jul-1983 00:00:00.000        -40.68        144.68        -100.0
 01-Jul-1983 00:00:00.000         19.52       -154.82         341.9
 01-Jul-1983 00:00:00.000        -14.25       -170.57         342.0
 01-Jul-1983 00:00:00.000          2.00       -157.30        -100.0
 01-Jul-1983 00:00:00.000         55.20       -162.72         335.3
 01-Jul-1983 00:00:00.000        -75.67        -27.00         341.7
 01-Jul-1983 00:00:00.000        -43.83       -172.63         341.3
 01-Jul-1983 00:00:00.000         25.67        -80.17         343.8
 01-Jul-1983 00:00:00.000         -4.67         55.17         339.1
 01-Jul-1983 00:00:00.000         13.43        144.78         344.0
 01-Jul-1983 00:00:00.000         19.53       -155.58         343.5
 01-Jul-1983 00:00:00.000         76.23       -119.33         339.8
 01-Jul-1983 00:00:00.000         40.05       -105.63         339.5
 01-Jul-1983 00:00:00.000         66.00          2.00         338.7
 01-Jul-1983 00:00:00.000        -64.92        -64.00         341.4
 01-Jul-1983 00:00:00.000         71.32       -156.60         340.1
 01-Jul-1983 00:00:00.000         17.75        -64.77         342.3
 01-Jul-1983 00:00:00.000         38.75        -27.08         341.1


|-----------skip 33-------------| width  12 | width  12 | width 10 |

Enter "21" in the # of points field.
Use the layout option to "skip" interspersed text:
1. Select Full prompter from the Options menu.
2. Select "locations" in the Field list of the Data Prompter.
3. Activate the Layout toggle button (in the right-hand panel of the prompter).
4. Bring up the Browser for the data file by clicking on the ellipsis button (...) next to the Data file field.
5. Click on Browser... to view the data file.
6. In the data file, position the cursor at the beginning of the first data value in the first data line (-37.95) and note the Byte Offsets value (counting from the start of the line). Enter this number (33) in the skip field of the Layout option.
7. Enter the value "12" in the width field of the Layout option. Since the latitude-longitude pairs are 2-vectors, the Data Prompter will read the specified width twice in succession, once for each component. (Thus, the Data Prompter skips 33 characters, reads 12 characters, and then reads 12 more, as specified by the first layout settings. See the marked intervals at the bottom of the data file in skip 33.)
8. To confirm these changes, click on Modify at the bottom of the panel.
9. Select "field0" in the Field list and rename it "CO2_concentration."
10. Enter the value "0" (zero) in the skip field and "10" in the width field. (Now the Data Prompter skips zero characters and then reads 10 characters. See the marked intervals at the bottom of the data file in skip 33.)
11. To confirm these changes, click on Modify at the bottom of the panel.
Save the header file (Step 10 of Example 1 in To save the header file), which should look like:

file = /usr/local/dx/samples/data/CO2fragment.lis
points = 21
format = text
interleaving = field
header = marker "CO2_CONC \n"
field = locations, CO2_concentration
structure = 2-vector, scalar
type = float, float
layout = 33, 12, 0, 10
end

Example 5. Time Series with Interspersed Text

The 21 lines of data in the preceding example represent a portion of a larger file (/usr/local/dx/samples/data/CO2.lis) containing a time series with 53 members.

In the Data Prompter initial dialog box, select Grid or Scattered File, then click on Describe Data to bring up the simplified Data Prompter.
Select Open from the File menu.
In the Open a Data Prompter Header dialog box, select CO2fragment.lis.general in the Files list and then click on OK.
Change the path name to:
```
/usr/local/dx/samples/data/CO2.lis
```
Activate the Series toggle button and change the value in the n field from "1" to "53." Do not change the start or delta field.
Save the header file (Step 10 of Example 1 in To save the header file), which should look like:

file = /usr/local/dx/samples/data/CO2.lis
points = 21
format = text
interleaving = field
header = marker "CO2_CONC \n"
series = 53, 1, 1
field = locations, CO2_concentration
structure = 2-vector, scalar
type = float, float
dependency = positions, positions
layout = 33, 12, 0, 10
end

Example 6. Column Majority Data

The General Array Importer assumes that the order of the data it imports is row majority (last index varies fastest). That is, on a 2-dimensional n × m grid, the order of data is:

f(X₀,Y₀), f(X₀,Y₁), ..., f(X₀,Y_m), f(X₁,Y₀), f(X₁,Y₁), ...

If the order of data is column majority (first index varies fastest), the order of data is:

f(X₀,Y₀), f(X₁,Y₀), ..., f(X_n,Y₀), f(X₀,Y₁), f(X₁, Y₁), ...

The General Array Importer will accept column-majority data if you select "Column" for the Data order option in the Data Prompter.

The file /usr/local/dx/samples/data/temp_wind.lis. contains data in column majority order. A header file that imports this data is:

file = temp_wind.lis
grid = 144 x 73
format = text
interleaving = field
majority = column
header = lines 9
field = temperature, wind_velocity
structure = scalar, 2-vector
type = float, float
dependency = positions, positions
layout = 39, 14, 0, 14
positions = regular, regular, -178.75, 2.5, 90.0, -2.5
end

[ OpenDX Home at IBM | OpenDX.org ]