# Scheduler

Each worker thread runs a scheduler.  Which does the following:

* check the resumable queue for "too old" packets.
** once per second, and then check check all packets older than 1s.
** ideally printing out where the request is blocked (module, instance, name)
** See yield / resume, below
* service the event list
** FD events first
** followed by timer events
** so that we read packets from sockets before they time out...
* service the incoming queue from network threads
** add packets to the "decode" list, which is run at a higher priority than other requests.
** the idea here is to quickly clean up the messages between network threads and worker threads
** push decoded packets onto the "runnable" priority heap, which is ordered by time.
* grab a request from the "runnable" heap.
* check `fr_status_continue()`
** if socket is closed, drop the request
** if conflicting packet has come in, drop the request
** process the packet until it's done, or until it yields
* keep looping

## Signaling

See the channel page for more details. And
signaling page.

The main problem we have with network / worker threads is signaling.
If the packets are widely spaced, the network thread can signal the
worker thread for every packet, and vice-versa for every reply.
However, if the packets arrive quickly, each end should switch to
busy-polling.

How to do this is non-trivial.

## Scheduling

Each network thread runs its own scheduler over the worker threads.
Not that the network thread may be away of only a subset of worker
threads.  Splitting the scheduler like this means there is minimal
contention, and ideal scaling.

The worker threads are weighted by total CPU time spent processing
requests.  They are put into a priority heap, ordered by CPU time.
This CPU time is not only the historical CPU time spent processing
requests, but also the predicted CPU time based on current "live"
requests.  This information is passed from the worker thread to the
network thread on every reply.

The network thread puts the workers into a priority heap.  When a
packet comes in, the first element is popped off of the heap, the
worker thread CPU time is updated (based on the predicted time spent
processing the new packet), and the thread is inserted back into the
priority heap.

As the heap will generally be small, the overhead of this work will be
small.

## Statistics

We need delay statistics for slow modules (Hi, SQL!).  The best
solution is the following.

Times are recorded on `unlang_yield()` and `unlang_resume()`.  The
delta is the time spent waiting on events for the module to do
something.  In addition, we need to track the module name, the
instance name, and a name of this yield point (e.g. user lookup,
versus group lookup).

We do this via dictionary attributes.  Which there is no *requirement*
to do so, doing it this way enables policies to be based on these
times, and allows for simple tracking / stats.

We create a new list `stats` for each request.  This is a list of
statistics attributes that is (ideally) read-only.  Only the server
core can write to it.

The `modules.c` code then creates a series of TLVs, all under one
parent TLV.  8 bit sub-TLVs should be good enough here.
----
Module-Statistics
  Module-Stats-SQL
    Module-Stats-SQL-SQL1
      Module-Stats-SQL-SQL1-operation1
        Module-Stats-SQL-SQL1-operation1-Delay
        Module-Stats-SQL-SQL1-operation1-Total-Requests
        Module-Stats-SQL-SQL1-operation1-Delay-1us
        Module-Stats-SQL-SQL1-operation1-Delay-10us
        Module-Stats-SQL-SQL1-operation1-Delay-100us
        Module-Stats-SQL-SQL1-operation1-Delay-1ms
        Module-Stats-SQL-SQL1-operation1-Delay-10ms
        Module-Stats-SQL-SQL1-operation1-Delay-100ms
        Module-Stats-SQL-SQL1-operation1-Delay-1s
        Module-Stats-SQL-SQL1-operation1-Delay-10s
        Module-Stats-SQL-SQL1-operation1-Delay-100s
----

All are TLVs, except for the lowest layer, which are `integer64`.

The `modules.c` code creates the first few layers of each TLV.

The `modules` header file defines a few macros:

  DEFINE_YIELD_POINT(name)
  USE_YEILD_POINT(inst, name)
  CACHE_DA_YIELD_POINT(name)
  CREATE_DA_YIELD_POINT(inst, name)

Which (in turn):

* defines a named yeild point, and updates various other internal macros to be used by `CACHE_DA` and `CREATE_DA`

* uses the pre-defined `da` inside of an `unlang_yield()` call.  The
  `da` should be cached inside of the module instance struct, so that
  we don't have to do dictionary lookups at run time.

* creates a `fr_dict_attr_t const *da_yeild_point_NAME;` inside of the module instance structure.

* in the module `bootstrap()` function, creates all of the relevant
  `da`s and caches them in the module instance.

We should then also have a `module_stats` module, which looks at the
per-request stats, and auto-creates the bins for each yeild point.
And probably also aggregates the stats up the TLV tree.

Then, once per 1K requests, or every second (whatever is shorter),
grabs a mutex lock, and aggregates the binned data to global counters.

That way the stats API can just query that module, and the module
returns total stats for all delays in the server.

## Yield / Resume

The thread needs to know which requests are yielded, so it can issue
warnings about blocked modules.  The best way here is for the
`REQUEST` to have a pointer to thread-specific data (yielded linked
list), and adds itself to the tail of the list.  When the request
resumes, it removes itself from the list.

In the current code, when a request is waiting for a timeout or a
socket event, the request is "lost", and buried inside of the event
loop.  The thread has no way of knowing where the request is.

With this solution, the thread has a linked list (oldest to newest) of
yielded requests.

This design goes along with the philosophy of the rest of the server
of "track as little as possible, and do work asynchronously where
possible".

!Diagram of yield / resume timing
