**************************
*** Catalog Persistent ***
**************************
(last modified: 11/28/2005)

Catalog now stored in BDB files as opposed to constructing them via hard-coded values in the code. Currently there are two BDB tables - CstoreSysTblProjection and CstoreSysTblColumn, in the catalog. As the name suggested, one table describes the proejction, another, column. Those tables are stored in RuntimeData folder(or RUNTIME_DATA specified). Catalog now preloaded with projections and columns for part of TPCH queries. For adding or modifyiing projection/column, please see catalog maintainence below.

**********************
*** Data Structure ***
**********************

Projection has four fields, described as follow:

#define MAX_NAME_LENGTH  128
typedef struct projectionRec {
  char          projectionName[MAX_NAME_LENGTH];                  /* projection name */
  unsigned int  projectionID;                                     /* projection ID */
  unsigned int  tableID;                                          /* belongs to this table*/
  unsigned int  primaryColumnID;                                  /* ID primary Column */
} PROJECTION_REC;

Column has six fields, described as follow:

typedef struct columnRec {
  char          columnName[MAX_NAME_LENGTH];                      /* column name */
  unsigned int  columnID;                                         /* column ID */
  unsigned int  projectionID;                                     /* belongs to this projection */
  unsigned int  dataType;                                         /* data type */
  unsigned int  compressionType;                                  /* compression type */
  unsigned int  isStorageKey;                                     /* is this a storage key column  1=true */
} COLUMN_REC;


data type of a column:
1 = INTEGER
2 = FLOAT
3 = STRING
4 = STAR

Compression type of a column (from the C-Store paper):
1 = TYPE1
2 = TYPE1A
3 = COMPRESSION_TYPE2
4 = COMPRESSION_TYPE3 
5 = COMPRESSION_TYPE3A
6 = COMPRESSION_TYPE4


System tables(catalog) file names(defined in cstore/common/Constants.h):
#define CSTORE_SYSTABLE_PROJECTION  string("CstoreSysTblProjection.bdb")  //projection primary 
#define CSTORE_SYSTABLE_COLUMN      string("CstoreSysTblColumn.bdb")      //column primary
#define CSTORE_SYSTABLE_COLUMN_AK   string("CstoreSysTblColumnAK.bdb")    //column secondary (alt. key = projectinID)

****************************
*** catalog maintainence ***
****************************
We do not have gui or command-line interface as for now. The interface is through editing text files and use the loader program to generate BDB catalog tables. There are two text files resides in RuntimeData folder, again, one for projection(CstoreSysTblProjection.txt), another for column(CstoreSysTblColumn.txt). Loader will process these two files and generate catalog. The text file contains the "values" of the fields defined in the above section. BE AWARE, there is a delimeter '#' between each field

Here is a projection file should look like:

lineitem#1#0#0
orders#2#0#0
customers#3#0#0
...

the first line defines a projection named lineitem, has id number 1. Currently we don't use table id and primary column id.

Now the column file: (this example defines five coulums of lineitem projection)
l_shipdate#1#1#1#1#1
l_orderkey#2#1#1#6#0
l_returnflag#3#1#3#6#0
l_extendedprice#4#1#1#6#0
l_suppkey#5#1#1#6#0
...

The first line defines a column named l_shipdate, has column id number 1. It belongs to projection ID 1, lineitem. The data type is integer. And it is a TYPE1 column.

After done editing the source file, now we type in

./cstoreqptest 0 createCatalog.cnf global.cnf

to process the text file and generate catalog.

(PLEASE NOTE: the actual data files will still need to be generated. Usually by running
./cstoreqptest 0 createData.cnf global.cnf) 

**********************
*** Related files ****
**********************

/cstore/src/AM/CatalogInstance.cpp               main C++ API
/cstore/src/AM/CatalogInstanceBDB.h              define key extractor for storing alternate key of Projection table.
/cstore/src/Util/BDBFil.cpp                      utility to interface with BDB 
/cstore/src/Util/BDBFil.h                     
/cstore/RuntimeData/CstoreSysTblProjection.txt   text file contains records of projection value
/cstore/RuntimeData/CstoreSysTblColumn.txt       text file contains records of column values
/cstore/RuntimeData/CstoreSysTblProjection.bdb   proejction BDB table
/cstore/RuntimeData/CstoreSysTblColumn.bdb       column BDB table

