# yada yada yada

## Malcolm Sparks

Created: 2016-12-10 Sat 08:40

## The printing press: 1440

Around 1440 Gutenberg assembled the printing press.

## The Web: 1990

From a historical perspective

DNA? The age of the universe? HTTP?

- URIs - universal address system for information
- HTTP - universal information retrieval system
- HTML - retrieval and navigation for humans

nil

## HTTP: the choice of a new generation

Now the web has become truly industrialized, and mechanized - literally - we are seeing machines talk to other machines using HTTP.

A few weeks ago I purchased one of these Amazon things and in a couple of hours was able to program it to add some new tricks. We did it as a family programming exercise. Of course, it was simply HTTP POSTS

## Web frameworks

- Still far too complex!
- So much for the developer to do
- RFC 7230, 7231, 7232, 7233, 7234, 7235…
- Ambivalent about standards
- We need a better way!

Here's the request, you figure out the response!

## WebMachine - a way forward

nil

## `yada`

## `yada` is declarative

#

    (yada/resource
     {:methods
      {:get
       {:produces "text/html"
        :response
        (fn [_] "Hello World!")}
       :put
       {:consumes "application/json"
        :parameters {:body {:greeting String}}
        :response (fn [ctx] ...)}}})

It took a year for this simple idea to dawn on me. The first iteration of yada was actually based on Clojure's records and protocols.

But what could be simpler than Clojure's maps?

This is one of the reasons Clojure is so nice.

If you haven't used Clojure's literal syntax, it's a bit like JSON.

But I find Clojure's syntax much nicer to type than JSON. JSON forces you to use speech-marks around keys, add commas in the right place, but worst of all, stops you writing comments!

Clojure's syntax has all the benefits of JSON - you can generate it, manipulate it with your own programs (swagger), you can serialize it so you can send it over the network.

From this simple declaration we can tell a few things: it produces static HTML.

There's a lot more you can add to a resource declaration, but there's also some schema behind this, so if you make a mistake and put it something that yada doesn't recognise, it will let you know by failing quickly.

## `yada` is standards-compliant

- Response status codes and headers
- Content negotiation
- Conditional requests
- Security
- Much more, and more to come

## `yada` is linear

available? → known-method? → uri-too-long? → TRACE → method-allowed? → parse-parameters → capture-proxy-headers → authenticate → get-properties → authorize → process-content-encoding → process-request-body → check-modification-time → select-representation → if-match → if-none-match → invoke-method → get-new-properties → compute-etag → access-control-headers → security-headers → create-response → logging → return

- State monad (state is known as request context)
- By default, all requests for all routes go through the same chain
- Every interceptor has access to request, whole resource model and any information collected by previous interceptors

## Interceptor

    (defn known-method?
      [ctx]
      (if-not (:method-wrapper ctx)
        (d/error-deferred
          (ex-info ""
            {:status 501
             ::method (:method ctx)}))
        ctx))

## `yada` is asynchronous

available? → known-method? → uri-too-long? → TRACE → method-allowed? → parse-parameters → capture-proxy-headers → authenticate → get-properties → authorize → process-content-encoding → process-request-body → check-modification-time → select-representation → if-match → if-none-match → invoke-method → get-new-properties → compute-etag → access-control-headers → security-headers → create-response → logging → return

- Any interceptor can defer its return value (futures, promises, core.async channels)

## `yada` is extensible

- Methods
- Media-types
- Interceptors

## Adding methods

    (extend-protocol Method
      BrewMethod
      (keyword-binding [_] :brew)
      (safe? [_] false)
      (idempotent? [_] false)
      (request [_ ctx]
        ;; Method semantics here
        ))

## Adding media-types

    (defmethod parse-stream
      "application/transit+msgpack"
      [_ stream]
      (-> (bs/to-input-stream stream)
          (transit/reader :msgpack)
          (transit/read)
          (with-400-maybe)))

## `yada` is scaleable

- Built upon aleph and netty

To many tens of thousands, even hundreds of thousands of simultaneous connections

## Content Negotiation

available? → known-method? → uri-too-long? → TRACE → method-allowed? → parse-parameters → capture-proxy-headers → authenticate → get-properties → authorize → process-content-encoding → process-request-body → check-modification-time → select-representation → if-match → if-none-match → invoke-method → get-new-properties → compute-etag → access-control-headers → security-headers → create-response → logging → return

## 'Data Macros'

    {:produces "text/html"}

becomes

    {:produces
     [{:media-type "text/html"}]}

- See https://juxt.pro/blog/posts/data-macros.html

## but you can do

    {:produces
     [{:media-type "text/html"
       :charset "UTF-8"
       :language #{"en" "zh-ch;q=0.9"}}
      {:media-type "text/html"
       :charset "Shift_JIS"
       :encoding "gzip"
       :language "zh-ch;q=0.9"}
      {:media-type "text/plain"}]}

## Parameter validation

available? → known-method? → uri-too-long? → TRACE → method-allowed? → parse-parameters → capture-proxy-headers → authenticate → get-properties → authorize → process-content-encoding → process-request-body → check-modification-time → select-representation → if-match → if-none-match → invoke-method → get-new-properties → compute-etag → access-control-headers → security-headers → create-response → logging → return

- Attempt to coerce parameters to required types
- If parameters aren't valid, return with a 400

## Parameters

    (yada/resource
     {:methods
      {:get
       {:parameters
        {:query {:foo s/Inst}
         :body {:request {:intent {:name s/Str}}}}}}})

## Conditional Requests

available? → known-method? → uri-too-long? → TRACE → method-allowed? → parse-parameters → capture-proxy-headers → authenticate → get-properties → authorize → process-content-encoding → process-request-body → check-modification-time → select-representation → if-match → if-none-match → invoke-method → get-new-properties → compute-etag → access-control-headers → security-headers → create-response → logging → return

## Properties

    (extend-protocol yada.resource/ResourceCoercion
      String
      (as-resource [s]
        (resource
         {:properties {:last-modified (to-date (now))
                       :version s}
          :methods
          {:get
           {:produces
            [{:media-type "text/plain"
              :charset charset/platform-charsets}]
            :response (fn [ctx] s)}}})))

## Security

available? → known-method? → uri-too-long? → TRACE → method-allowed? → parse-parameters → capture-proxy-headers → authenticate → get-properties → authorize → process-content-encoding → process-request-body → check-modification-time → select-representation → if-match → if-none-match → invoke-method → get-new-properties → compute-etag → access-control-headers → security-headers → create-response → logging → return

## Security

    (yada/resource
     {:id ::basic-example
      :access-control
      {:scheme "Basic"
       :verify
       (fn [[user password]]
         (when (= [user password]
                  ["scott" "tiger"])
           {:user "scott"
            :roles #{"secret/view"}}))
       :authorization
       {:methods
        {:get "secret/view"}}}})

## Server sent events

    (yada/resource
     {:methods
      {:get {:produces "text/event-stream"
             :response (chan)}}})

## Uploads

available? → known-method? → uri-too-long? → TRACE → method-allowed? → parse-parameters → capture-proxy-headers → authenticate → get-properties → authorize → process-content-encoding → process-request-body → check-modification-time → select-representation → if-match → if-none-match → invoke-method → get-new-properties → compute-etag → access-control-headers → security-headers → create-response → logging → return

- Fully asynchronous
- Support for form-data/multipart (~200Mb/s)
- Boyer-Moore-Horspool algorithm (modified for streaming)
- Pluggable handlers

## Error handling

- Separate (pluggable) error interceptor chain

## Status responses

    (yada/resource
     {:methods
      {:get
       {:response (fn [ctx] nil)}}
      :responses
      {404 {:response "Not found"}}})

## Routing

- Not supported!

> Routes and Resources should be independent – Phillip Meier

About 3 years ago, Billy and I were at EuroClojure in Berlin in 2013 and I about whether he would be adding routing to Liberator, without even hesitating he said no!

I asked why. He explained that resources and their names are different concepts and shouldn't be mixed. I slept on this for a couple of months, a process which we call 'Hammock Driven Development'. It worked, over Christmas I had a sudden moment of inspiration, grabbed the nearest computer and typed out the first version of what became bidi.

## bidi

- A URI router, nothing more
- Recursive pattern matching
- Bidirectional

## bidi example

    ["/phonebook" [
                   ["" :index]
                   [["/" :id] :entry]
                  ]]

    /phonebook -> :index
    /phonebook/123 -> :entry

    :index -> /phonebook
    :entry 123 -> /phonebook/123

## `yada` is evolving

- clojure.spec
- Instrumentation and debugging
- Environment profiles
- Range requests & partial content
- Rate limiting
- WebDAV
- TLS and HTTP/2 (server push)
- ClojureScript?

## Programming the Amazon Echo

## Getting started with bidi and `yada`

- Documentation: https://juxt.pro/yada
- Sample application: https://github.com/juxt/edge