-
Notifications
You must be signed in to change notification settings - Fork 1
CSV Transform Specification
The instructions for taking a file and dividing it up into fixed-length records and creating a comma-separated values (CSV) file out of them are simply listed in a plain text file used as a transform in a Transformenator invocation. Data from the original file can be interpreted as ASCII text, as EBCDIC text, or as hexadecimal values.
The keywords in this section define an interpretation of a file as records. Note that only NEXTREC is required, as it defines the uniform length of each record. The case of the keywords is not significant. All numerical values are given in undecorated hexadecimal.
Any line starting with a semicolon will not be considered. Example:
; This text will not be considered
Example: FIRSTREC=0E00
Example: NEXTREC=100 ; One record every 0x100 bytes
When present, defines the criteria that must be true in order to consider a record for output. If SELECTIF is missing, all records will be unconditionally exported, one per NEXTREC bytes. This specification first lists the offset within the record, followed by a comma, followed by one or more hex bytes that must match in order to be exported. Multiple SELECTIF clauses may be used to produce logical OR conditions. Examples:
SELECTIF=00,40 ; First byte must be 0x40 to be valid
SELECTIF=01,20 ; Or, second byte can be 0x20
Defines field layout based on positional and length specifications as demonstrated directly by the existence of repeating characters. See the Automatic Layout Processing section below. Example:
; Define field layout by example
LAYOUT="mmddyy..nnnnnnxxxxxxxpppppp"
The keywords below define export instructions for a single field when there is no LAYOUT definition present. Each field definition must contain one each of the four keywords below. Multiple sets of these specifications typically make up a full record to export. The case of the keywords is not significant. Fields can overlap and skip positions as desired. All numerical values are given in undecorated hexadecimal.
Field name to be exported on header row of output CSV file. Example: NAME="Field 1"
Example: ORIGIN=01
Example: LENGTH=0a
Text will be interpreted and encoded in the output CSV file as ASCII text, EBCDIC text, or hexadecimal values (ASCII, EBCDIC, and HEX, respectively) with surrounding quotes. With InterpLiteral, additional syntax and quoting will be added to the output CSV file that convinces spreadsheets not to re-interpret numerical values. INTERP and INTERPLITERAL can be mixed and matched with ASCII, EBCDIC, and HEX specifications as desired. Examples:
INTERP=ASCII
INTERP=HEX
INTERPLITERAL=EBCDIC
As an alternative to specifying each field "by hand," you can let the computer do the computing by specifying textually how the fields are laid out. Consider a LAYOUT specification like this:
LAYOUT="aaaaa2222rrrffftttt"
This specifies that there are going to be five fields (each character transition triggers a new field definition) and they have the positional and length specifications as demonstrated directly by the existence of repeating characters. It is not significant what the characters actually are. It is not possible to skip over bytes, and it is not possible to overlap fields when using LAYOUT. When a CSV extraction is performed with a transform containing a LAYOUT specification, the computed field values will be printed to the console so they can be used to refine the specification if things like overlapping fields are required.
When there are both LAYOUT and full field (ORIGIN and LENGTH) specifications present, they will all be merged. That is, the "manual" field specifications will be respected for their field names and interpretation types, but the position attributes will be overwritten by the LAYOUT specification. Additional fields specified by LAYOUT but missing accompanying manual definitions will be given default names and interpretation types, and will be given position specifications from the LAYOUT pattern.
When exploring an otherwise unknown disk's files, it is often useful to first dump the entire set of records with a textual representation of the entire thing on one line, then manually inspect those results to ascertain where the field boundaries are based on common patterns in the data. Using the LAYOUT keyword here is an easy way to line up the data with positionally-dependent fields visually.
Below is an example transform file that interprets a disk image taken from an IBM System/36 floppy disk that was used for record storage:
;
; Custom IBM System/36 to CSV transform
;
FIRSTREC=0E00 ; First record starts at 0xE00
NEXTREC=100 ; Records continue every 0x100 bytes
SELECTIF=00,40 ; The zeroeth byte needs to be an EBCDIC space
;
NAME="Field 1" ; First field definition
ORIGIN=01 ; Starts at byte 0x01 (zero-indexed)
LENGTH=09 ; Runs for a length of 0x09 bytes
INTERP=EBCDIC ; Interpret data as EBCDIC
;
NAME="Field 2" ; Second field definition
ORIGIN=0A ; Starts at byte 0x0a (zero-indexed)
LENGTH=1E ; Runs for a length of 0x1e bytes
INTERP=EBCDIC ; Interpret data as EBCDIC
;
NAME="Field 3" ; Third field definition
ORIGIN=28 ; Starts at byte 0x28 (zero-indexed)
LENGTH=05 ; Runs for a length of 0x05 bytes
INTERP=EBCDIC ; Interpret data as EBCDIC
;
NAME="Field 4" ; Fourth field definition
ORIGIN=2d ; Starts at byte 0x2d (zero-indexed)
LENGTH=02 ; Runs for a length of 0x02 bytes
INTERP=EBCDIC ; Interpret data as EBCDIC
;
NAME="Field 5" ; Fifth field definition
ORIGIN=2f ; Starts at byte 0x2f (zero-indexed)
LENGTH=3d ; Runs for a length of 0x3d bytes
INTERP=HEX ; Interpret data as hexadecimal bytes
Here is a similar example using the LAYOUT specification:
;
; Custom IBM System/36 to CSV transform
;
FIRSTREC=0E00 ; First record starts at 0xE00
NEXTREC=100 ; Records continue every 0x100 bytes
SELECTIF=00,40 ; The zeroeth byte needs to be an EBCDIC space
LAYOUT="011111111122222222222222222222222222222233333445555555555555555555555555555555555555555555555555555555555555"
;
NAME="Ignore" ; Ignore the first byte
INTERP=EBCDIC ; Even though we want to skip it, we need to account for it
;
NAME="Field 1" ; First field definition
INTERP=EBCDIC ; Interpret data as EBCDIC
;
NAME="Field 2" ; Second field definition
INTERP=EBCDIC ; Interpret data as EBCDIC
;
NAME="Field 3" ; Third field definition
INTERP=EBCDIC ; Interpret data as EBCDIC
;
NAME="Field 4" ; Fourth field definition
INTERP=EBCDIC ; Interpret data as EBCDIC
;
NAME="Field 5" ; Fifth field definition
INTERP=HEX ; Interpret data as hexadecimal bytes