DelphiDabbler CodeSnip
File Format Documentation

# Backup Files

## Introduction

The CodeSnip backup file format is a general format that can store all the files from a single directory, excluding sub-directories. It preserves file dates and contents.

CodeSnip stores backups of both the main and user databases using this format.

A backup file is a single file containing a concatenation of all the backed up files, along with some header and housekeeping information. Quite how the data is stored and organised has varied over time. There have been four versions of the file format to date:

Version 1
  This first version of the file was used only to back up the user database in a single text file. All housekeeping information and meta data was stored as encoded text. The file size was limited to 32Kb. Only text files could be stored in this format.
Version 2
  Similar to version 1 except that the 32Kb size limit was lifted (now 2Gb). Used only for backing up the user database.
Version 3
  Similar to Versions 1 and 2. Now used for backing up both the user database and the main database. A file ID field was added to specify which database was backed up. The file ID generalises to any file.
Version 4
  Major changes in this version. The contents of the file are now binary rather than text. Any file type can now be included in the backup. Maintains the FileID from Version 3. Also has an ASCII text watermark to enable earlier readers to detect it.

## Encoding

The following encodings have been used for backup files:

Versions 1 and 2
  The entire backup file is in the System Default ANSI code page. All backed up files are expected to be text files with the same encoding. Meta data and housekeeping data is stored as hex representation of integers. Note that hex characters are the same in this code page as they are in the ASCII character set.
Version 3
  Up to and including CodeSnip 3.5.0 backup files used the same System Default ANSI code page as versions 1 and 2. From CodeSnip 3.5.1 the code page was changed to Windows-1252. Again hex characters are the same as they are in the ASCII character set.
Version 4
  This file format uses pure binary raw data. The watermark part of the file header is written in ASCII. File names are stored as UTF-8. All other information is stored as raw binary.

File formats v1 to v3 required that hex characters were written using a single byte, ruling out the use of Unicode formats such as UTF-16.

## File Formats

### Version 1 Format

`FileCount`
  Number of files in backup file. SmallInt encoded as four hex digits.

Followed by `FileCount` records of:

`Name`
  Name of file without path information. AnsiString preceded by its size as a SmallInt encoded as four hex digits.
`FileDate`:
  File's modification date as a DOS file stamp. LongInt encoded as eight hex digits.
`Checksum`
  MD5 checksum of backed up file. 16 Bytes, each encoded as two hex digits.
`Content`
  File contents. AnsiString preceded by its size as a SmallInt encoded as four hex digits.

### Version 2 Format

`'FFFF'`
  Indicates this is a post-v1 file format. SmallInt($FFFF) encoded as four hex digits.
`'0002'`
  Indicates this is the version 2 file format. SmallInt(2) encoded as four hex digits.
`FileCount`
  Number of files in backup file. SmallInt encoded as four hex digits.

Followed by `FileCount` records of:

`Name`
  Name of file without path information. AnsiString preceded by its size as a SmallInt encoded as four hex digits.
`FileDate`
  File's modification date as a DOS file stamp. LongInt encoded as eight hex digits.
`Checksum`
  MD5 checksum of backed up file. 16 Bytes, each encoded as two hex digits.
`Content`
  File contents. AnsiString preceded by its size as a LongInt encoded as eight hex digits.

### Version 3 Format

`'FFFF'`
  Indicates this is a post-v1 file format. SmallInt($FFFF) encoded as four hex digits.
`'0003'`
  Indicates this is the version 3 file format. SmallInt(3) encoded as four hex digits.
`FileID`
  File type indicator. SmallInt encoded as four hex digits.
  - `'CBAC'` for main database backup.
  - `'DBAC'` for user database backup.
`FileCount`
  Number of files in backup file. SmallInt encoded as four hex digits.

Followed by `FileCount` records of:

`Name`
  Name of file without path information. AnsiString preceded by its size as a SmallInt encoded as four hex digits.
`FileDate`
  File's modification date as a DOS file stamp. LongInt encoded as eight hex digits.
`Checksum`
  MD5 checksum of backed up file. 16 Bytes, each encoded as two hex digits.
`Content`
  File contents. AnsiString preceded by its size as a LongInt encoded as eight hex digits.

### Version 4 Format

`'FFFF'`
  Indicates this is a post-v1 file format. SmallInt($FFFF) encoded as four hex digits.
`'0004'`
  Indicates this is the version 4 file format. SmallInt(4) encoded as four hex digits.
`'00000000'`
  Padding field. LongInt(0) encoded as eight hex digits. The first four characters of this field are where v2 file readers look for the file count. The second four characters are where v3 file reader for the same information. Both readers interpret the appropriate value as meaning their are no files in the backup and will not try to read any further in the file.
`FileID`
  File type indicator. SmallInt in binary.
  - `'CBAC'` for main database backup.
  - `'DBAC'` for user database backup.
`FileCount`
  Number of files in backup file. SmallInt in binary.

Followed by `FileCount` records of:

`Name`
  Name of file without path information. Bytes of a UTF-8 string preceded by number of bytes as a SmallInt in binary.
`FileDate`
  File's modification date as a DOS file stamp. LongInt in binary.
`Checksum`
  MD5 checksum of backed up file. 16 bytes in binary.
`Content`
  File contents. Bytes of literal copy of file contents preceded by the number of bytes as a LongInt in binary.

## File Reading & Writing Notes

Code that writes backup files should only ever write the latest available file format at the time the code was written. It musr never attempt to write earlier versions.

Code that restores backup files must be able to interpret all earlier formats in addition to the current one, because the program may need to restore data from an old backup file. If the code detects a later version than it was designed for it should display an error message and terminate.

### Problems with Earlier Backup Restoration Code

Some earlier versions of the backup restoration code have faults in that they will attempt to read later backup file versions than those they were designed for. By good fortune, factors in the design of later file formats mean that the reading code always abandons reading later, albeit with no error message. These problems are:

File readers designed to work with file format v1
  At this time there was no concept there would be different versions of the backup file format, so the code does not attempt to detect the file version.
  The reading code expects backup files to begin with a file count as a text representation of a SmallInt in hex. In file version 2 and later the first four ANSI chars of the file are `'FFFF'`, which decodes as `-1`. The reader interprets this value as meaning there are no files in the backup and does nothing.
File readers designed to work with file format v2
  These readers interpret the v1 and v2 file formats correctly.
  In v3 files the file ID is recorded in the same location and same format as the v2 file format expects to find the file count. Luckily the only two values in use as File IDs are `'CBAC'` and `'DBAC'`, both of which decode as negative integers expressed in hex. The reader software interprets this as meaning there are no files in the backup.
  The v4 file format moved the file ID value to a new location and replaced it with a padding value of `'0000'` in the v2 file count location, again indicating there are no files in the backup to v2 file format readers.
File readers designed to work with file format v3
  These readers interpret the v1, v2 and v3 file formats correctly.
  Up to, and including, CodeSnip v3.8.5, file reader software designed to work with file format v3 will also try to read later file versions. It looks for its file count in a hex encoded SmallInt value at position 12 in the backup file. File format v4 was designed to have a padding value of `'0000'` in this location. This fools the v3 reader software into thinking there are no files in the backup.
  From CodeSnip v3.8.6 the file reader works correctly and exits with an error when an unsupported backup file version is encountered.

The file reader designed to work with the v4 file format does recognise newer versions of the file format and does not try to read them. Instead it raises an exception.

In summary, all future backup files should begin with these ANSI / ASCII characters: `'FFFFxxxx00000000'` where xxxx is the file version number as a SmallInt encoded as four hex characters. For example for file format v4 we use `'FFFF000400000000'`.

### Identification of file format versions

Each file version can be considered to have a unique "watermark" at the start of the file if we consider an empty watermark to be valid. This is fine as long as only one file version has the empty watermark. Here are the current watermarks as strings of ASCII characters:

File format version 1
  Watermark: `''` (empty)
File format version 2
  Watermark: `'FFFF0002'`
File format version 3
  Watermark: `'FFFF0003'`
File format version 4
  Watermark: `'FFFF000400000000'`

It can be seen that the version number at byte offset 4 is not necessary for identification if the watermark is made longer. However, it should always be used because earlier versions of the file readers may use it to determine which format version is to be read.