= Chronicle Queue
Peter Lawrey

//removed the link as web page is not available
//image::http://chronicle.software/wp-content/uploads/2014/07/ChronicleQueue_200px.png[]

== How Chronicle Works
Chronicle is a Java project focused on building a persisted low latency messaging framework for high performance and critical applications.

== What is Different?
This project can be seen as yet another queue implementation but it has major design choices that should be emphasized.

Using non-heap storage options (`RandomAccessFile`) Chronicle provides a processing environment where applications do not suffer from garbage collection (GC).

While implementing high performance, and memory-intensive, applications in Java, one of the biggest problems is garbage collection. Garbage collection may slow down your critical operations non-deterministically at any time. In order to avoid non-determinism, and escape from GC delays off-heap memory solutions are addressed. The main idea is to manage your memory manually so it does not suffer from GC. Chronicle behaves like a management interface over off-heap memory, so that you can build your own solutions over it.

Chronicle uses `RandomAccessFiles` while managing memory, and this choice brings a great deal of possibilities. Random access files permit non-sequential, or random, access to a files contents. To access a file randomly, you open the file, seek a particular location, and then read from, or write to, that file. `RandomAccessFiles` can be seen as large C-type byte arrays that allows you to access any random index directly using pointers. File portions can be used as `ByteBuffers` if the portion is mapped into memory.

=== What is the effect of page faults when we have a huge Chronicle and not enough RAM ?
Pages are swapped in and out by the operating system on demand.  Writes are performed asynchronously, and under moderate loads do not impact the latency of writes.  Sequential reads are also read using look-ahead, that is, before you ask for them.  Random reads perform best when the data accessed is already in memory

The size of data stored can exceed the amount of memory you have. Provided the amount you use is less than the main memory size, you see little impact.  If you exceed main memory size, you will see measurable performance degradation, but it is dramatically more graceful than using too much heap.  Using the same amount of heap can cause the machine to fail.

== Building Blocks

Chronicle Queue has two main conceptual components:

 - Tailer - used for both sequential reads and random reads
 - Appender - sequential writes

Chronicle Queue is the main interface for management and can be seen as the collection class of the Chronicle environment.

Chronicle Queue allows you to put huge numbers of objects of different sizes onto a single queue. For each data record, Chronicle builds it indexes ( used for random access reads ) automatically when ever messages are appended to the queue. 

An appender is used to write/add data to the queue.

== Getting Started
In the following example we will:

- Create a Chronicle Queue
- Put a record to the queue
- Read the record from queue.

[source, Java]
----
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueue;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;
----
[source, Java]
----
// will write the .cq4 file to working directory
try (SingleChronicleQueue queue = SingleChronicleQueueBuilder.builder().path(".").build();
     ExcerptAppender appender = queue.createAppender();
     ExcerptTailer tailer = queue.createTailer()) {

  // write 'hello world' to the queue
  appender.writeText("hello word");

  // read 'hello world' from the queue
  System.out.println(tailer.readText());
}

----

Create a tmp directory for the Chronicle Queue giving the name starting `Java_temp_directory/SimpleChronicle` as the base folder.

[source,java]
----
File tmpPath = Files.createTempDirectory("SimpleChronicle").toFile();
SingleChronicleQueue queue = SingleChronicleQueueBuilder.builder().path(tmpPath).build();
----

Create appender and reader:

[source,java]
----
ExcerptAppender appender = queue.acquireAppender();
ExcerptTailer tailer = queue.createTailer();
----

The `write` and `read` an object 

[source, java]
----
package net.openhft.chronicle.queue;

import net.openhft.chronicle.queue.impl.single.SingleChronicleQueue;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;
import net.openhft.chronicle.wire.Marshallable;

import java.io.IOException;
import java.nio.file.Files;

public class Example {

    static class MyObject implements Marshallable {
        String name;
        int age;

        @Override
        public String toString() {
            return Marshallable.$toString(this);
        }
    }

    public static void main(String[] args) throws IOException {
        // will write the .cq4 file to working directory
        try (SingleChronicleQueue queue = SingleChronicleQueueBuilder.builder()
                .path(Files.createTempDirectory("queue").toFile()).build();
             ExcerptAppender appender = queue.createAppender();
             ExcerptTailer tailer = queue.createTailer()) {

            MyObject me = new MyObject();
            me.name = "rob";
            me.age = 40;

            // write 'MyObject' to the queue
            appender.writeDocument(me);

            // read 'MyObject' from the queue
            MyObject result = new MyObject();
            tailer.readDocument(result);

           // System.out.println(result);
        }
    }
}
----

outputs the following: 

----
!net.openhft.chronicle.queue.Example$MyObject {
  name: rob,
  age: 40
}
----

== Tiered Indexing
Chronicle Queue uses tiered multi-level indexing to provide a fast and efficient method for searching for messages in a large queue.

=== Primary index
[source, YAML]
----
# position: 385, header: -1 #  <1>
--- !!meta-data #binary
index2index: [ #  <2>
  # length: 32768, used: 1
  262568, #  <3>
  0, 0, 0, 0, 0,
]
----
<1> `position` specifies the starting address of the data that follows, within the queue (*.cq4) file.
<2> `index2Index` defines a pointer to the next lower level of index.
<3> specifies the point to the next level index.

NOTE: In this way, indexes can be tiered (primary, secondary, tertiary, etc.) as required to enhance indexing speed and efficiency.

=== Final level index

----
# position: 262568, header: -1 #  <1>
--- !!meta-data #binary
index: [ #  <2>
  # length: 32768, used: 1
  524744, #  <3>
  0, 0, 0, 0, 0, 0, 0, 0,
]
----
<1> `position` specifies the point in the queue.

<2> `index` defines a pointer to the data.

<3> specifies the point in the queue where the data begins.

=== Data location

----
# position: 524744, header: 0 #  <1>
--- !!data #binary #  <2>
"": some more text
...
# 785952 bytes remaining

----
<1> `position` specifies the point in the queue where the data begins.

<2> `data` defines the information that follows is data (`--- !!data #binary #`), rather than than meta data (`--- !!meta-data #binary`).

== File rolling

A Chronicle Queue is a logical view of a directory on the file-system.
The queue data itself is split across multiple files, each of which contains
data belonging to a single _cycle_. The length of the cycle is determined by
the _rollCycle_ parameter passed to the queue builder.

Example configuration of _RollCycle_:

   * `RollCycles.DAILY` events stored in the queue will be grouped into 24-hour periods
   * `RollCycles.HOURLY` every hour, a new queue file will be created for written events

As new files are created to accommodate events being written to the queue, a persisted
data-structure (`directory-listing.cq4t`) is updated with the lowest and highest
_cycle_ numbers present in the directory.

Maintaining this table allows an `ExcerptTailer` to busy-spin waiting for new data to
be appended to the queue, without the need to make costly calls to the file-system to
check for the existence of new queue files.

=== Read-only mode

When opening a queue in read-only mode, it is not possible to utilise the structure
described above (since the structure needs to be written to). In this case,
Chronicle Queue will fallback to inspecting the file-system to determine when
new queue files are created.

For this reason, significant garbage will be generated when using an `ExcerptTailer`
in a busy-spin loop in read-only mode.

=== Queue File Reference Counting (Enterprise Edition Feature)

As a queue is written to, it will create a new file for each _roll-cycle_.

Over time, it may become necessary to automatically delete or archive
old files. An automated process needs to ensure that there are not active
file-handles open on a queue file before attempting to delete.

To facilitate this operation, Enterprise Chronicle Queue tracks
references to its _roll-cycle_ files internally.

The suggested approach is to perform the maintenance operation from
a separate JVM to the application, in the following manner:

[source, java]
....
public void removeOldQueueFiles() throws IOException {
    final Path queuePath = Paths.get("/path/to/queue");
    try (final SingleChronicleQueue queue = SingleChronicleQueueBuilder.
            binary(queuePath).build()) {

        try (final Stream<Path> queueFiles = Files.list(queuePath).
                filter(p -> p.toString().endsWith(SingleChronicleQueue.SUFFIX))) {

            queueFiles.filter(p -> isReadyForDelete(p)).map(Path::toFile).
                    filter(f -> queue.numberOfReferences(f) == 0).
                    forEach(File::delete);

        }
    }
}
....

Use the `ChronicleQueue.numberOfReferences()` method to ensure that there are
zero references to a given file before attempting to delete it.

== Configuration

Chronicle Queue (CQ) can be configured via a number of methods on the `SingleChronicleQueueBuilder` class.

=== RollCycle

One such piece of configuration is the `RollCycle` that determines the rate at which CQ will roll the underlying queue files.
For instance, using the following code snippet will result in the queue files being rolled (i.e. a new file created) every hour:

[source,java]
----
ChronicleQueue.singleBuilder(queuePath).rollCycle(RollCycles.HOURLY).build()
----

Once a queue's roll-cycle has been set, it cannot be changed at a later date.
More formally, after the first append has been made to a Chronicle Queue, any further instances of `SingleChronicleQueue`
configured to use the same path _must_ be configured to use the same roll-cycle.

This check is enforced by `SingleChronicleQueueBuilder`, so the following code causes an exception to be thrown:

[source,java]
----
final Path queueDir = Paths.get("/data/queue/my-queue");
try (ChronicleQueue queue = ChronicleQueue.singleBuilder(queueDir).rollCycle(SECONDLY).build()) {
    // this append call will 'lock' the queue to use the SECONDLY roll-cycle
    try (DocumentContext documentContext = queue.acquireAppender().writingDocument()) {
        documentContext.wire().write("somekey").text("somevalue");
    }
}

// this call will fail since we are trying to create a new queue,
// at the same location, with a different roll-cycle
try (ChronicleQueue recreated = ChronicleQueue.singleBuilder(queueDir).rollCycle(HOURLY).build()) {

}
----

In the case where a Chronicle Queue instance is created _before_ any appends have been made, and there is
a subsequent append operation with a _different_ roll-cycle, then the roll-cycle will be updated to match the
persisted roll-cycle. In this case, a warning log message will be printed in order to notify the library user
of the situation:

[source,java]
----
// creates a queue with roll-cycle MINUTELY
try (ChronicleQueue minuteRollCycleQueue = ChronicleQueue.singleBuilder(queueDir).rollCycle(MINUTELY).build()) {
    // creates a queue with roll-cycle HOURLY - valid since no appends have yet been made
    try (ChronicleQueue hourlyRollCycleQueue = ChronicleQueue.singleBuilder(queueDir).rollCycle(HOURLY).build()) {
        // append using the HOURLY roll-cycle
        try (DocumentContext documentContext = hourlyRollCycleQueue.acquireAppender().writingDocument()) {
            documentContext.wire().write("somekey").text("somevalue");
        }
    }
    // now try to append using the queue configured with roll-cycle MINUTELY
    try (DocumentContext documentContext2 = minuteRollCycleQueue.acquireAppender().writingDocument()) {
        documentContext2.wire().write("otherkey").text("othervalue");
    }
}
----
console output:
[source]
----
[main] WARN SingleChronicleQueueBuilder - Overriding roll cycle from HOURLY to MINUTELY.
----
