Introduction to yada

Malcolm Sparks

Created: 2016-08-24 Wed 19:21

Slides

https://juxt.pro/yada.pdf

Opening

HTTP

scale
longevity

HTTP services in Clojure: Ring

Ring loves Data!

(def request
  {:server-port 443,
   :server-name "www.google.com",
   :remote-addr "218.29.32.10",
   :uri "/",
   :query-string "q=yada",
   :scheme :https,
   :request-method :get,
   :headers {"host" "www.google.com"}})

(def response
  {:status 200
   :headers {"content-type" "text/plain;charset=utf-8"}
   :body "Hello World!"})

But… Ring middleware?

(def api
  (-> routes
      wrap-keyword-params
      wrap-params
      wrap-head
      wrap-not-modified
      wrap-json-response))

Is Ring middleware suitable for building HTTP services?

You're doing it wrong

Example A: wrap-head

4.3.2. HEAD

"This method can be used for obtaining metadata about the selected representation without transferring the representation data and is often used for testing hypertext links for validity, accessibility, and recent modification."

― RFC 7231

Example A: wrap-head

(defn head-request
  "Turns a HEAD request into a GET."
  {:added "1.2"}
  [request]
  (if (= :head (:request-method request))
    (assoc request :request-method :get)
    request))

(defn head-response
  "Returns a nil body if original request was a HEAD."
  {:added "1.2"}
  [response request]
  (if (and response (= :head (:request-method request)))
    (assoc response :body nil)
    response))

Example B: Compojure routing

(routes
  (GET "/hello" [] (fn [req] "Hello World!"))
  (POST "/hello" [] (fn [req] (launch-missiles!))))

Q. Which HTTP status do we get from this Ring handler if we send a PUT request to /hello?
Q. Which HTTP status should we get?

Example C: wrap-modified

5.2. Conditionals

The HTTP conditional request header fields [RFC7232] allow a client to place a precondition on the state of the target resource, so that the action corresponding to the method semantics will not be applied if the precondition evaluates to false.

― RFC 7231

Example C: wrap-modified

(defn wrap-not-modified
  "Middleware that returns a 304 Not Modified from the wrapped handler
  if the handler response has an ETag or Last-Modified header, and the
  request has a If-None-Match or If-Modified-Since header that matches
  the response."
  {:added "1.2"}
  [handler]
  (fn [request]
    (-> (handler request) ; WAT?
        (not-modified-response request))))

Functions vs. Data…

Arthur Dent: What happens if I press this button?

Ford Prefect: I wouldn't-

Arthur Dent: Oh.

Ford Prefect: What happened?

Arthur Dent: A sign lit up, saying 'Please do not press this button again.'

― Douglas Adams, The Original Hitchhiker Radio Scripts

Pick & Mix

Knowing what middleware to add to a Ring application, and in what order, can be difficult and prone to error. ― https://github.com/ring-clojure/ring-defaults

So is there an alternative?

(Hint: follow the data)

yada

All of HTTP (mostly)

correctness
completeness
performance

DON'T PANIC!

yada is easy

just a function

that returns a Ring handler

`(yada "Hello World!")`

=> result

`(yada (atom "Hello World!"))`

=> result

`(yada (fn [ctx] "Hello World!") {:allowed-methods #{:get}})`

and so on…

["/talks/" (yada (clojure.java.io/file "talks"))]

(yada ["A" "B" "C"])

(yada {:do "a deer, a female deer"
       :re "(let's stop this now)"}

(yada (new-template-resource "page.html" {:title "yada"}))

(yada (map->PostgresTable {:table "ACCOUNTS"}))

(-> "Let's go meta!" yada yada yada)

Features

Parameter coercions

(properties [_]
 {:parameters
  {:get {:path {"dept" String}
         :query {"order" #{:asc :desc}}}
   :post {:path {"dept" String}
          :form {"id" Long
                 "name" String
                 "dob" Date}
          :header {"X-Tag" java.util.UUID}}}})

Content Negotiation

media types
charsets
content encodings
languages

(properties
 [_]
 {:representations
  [{:media-type #{"text/html" "text/plain"}
    :charset #{"UTF-8" "US-ASCII"}
    :language #{"en" "fi"}}

   {:media-type "text/html"
    :charset #{"UTF-8" "Shift_JIS;q=0.9"}
    :language "zh-ch"
    :encoding "gzip"}]})

(properties
 [_]
 {:representations
  [{:media-type #{"application/json"
                  "application/json;pretty=true"
                  ;; Just add some more
                  "application/edn"
                  "application/edn;pretty=true"}}]})

Conditional Requests

If-Match
If-None-Match
If-Modified-Since
If-Unmodified-Since

Routing

Describe your whole API in data

(def api
  ["/" {"hello" (yada "Hello World!" {:id :hello})
        "hello-atom" (yada (atom "Hello World!"))}])

Built-in Swagger!

(def api
  ["/hello-api"

   (yada/swaggered
    {:info {:title "Hello World!" :version "1.0"
            :description "Demonstrating yada + swagger"}}

    ["/" {"hello" (yada "Hello World!")
          "hello-atom" (yada (atom "Hello World!"))}])])

`(yada (atom "Hello World!"))`

Swagger (behind the curtain)

(defn swaggered [info route]
  (let [spec (merge info
                    {:paths (->> route bidi/route-seq …)})]
    (->Swaggered (yada (->SwaggerSpec spec (now))) route)))

Swaggered record

(defrecord Swaggered [spec route]

  bidi.bidi/Matched
  (resolve-handler [this m]
    (if (= (:remainder m) "/swagger.json")
      (succeed this m) ; match!
      (resolve-handler [route] m))) ; keep traversing!

  bidi.ring/Ring
  (request [_ req match-context] (spec req)))

SwaggerSpec record

(require '[ring.swagger.swagger2 :as rs])

(defrecord SwaggerSpec [spec created-at]
  p/Properties
  (properties [_]
    {:representations
     [{:media-type #{"application/json"
                     "application/json;pretty=true"}
       :charset #{"UTF-8" "UTF-16;q=0.9" "UTF-32;q=0.9"}}]

     ::swagger-json (rs/swagger-json spec)})

  (properties [_ ctx]
    {:last-modified created-at :version spec})
  Get
  (GET [_ ctx] (-> ctx :properties ::swagger-json)))

Finally, we mentioned the Swagger spec was wrapped as a yada resource. Here's the code.

Notice how the resource makes some claims via the properties function. It says that it renders JSON, preferrably in UTF-8 but will provide UTF-16 or UTF-32 if necessary.

We also generate the swagger JSON at this point, which is one line of code thanks to the ring-swagger library.

Notice how we're returning data again, which is fed into the model.

There's a second form of this properties function that is called on every request. These help yada implement conditional requests. The ETag is derived from the hash of the Swagger spec. If the spec changes, so will the ETag.

For the actual GET method we pull out the swagger spec, it's still a Clojure map at this point, and return it. Yada's body coercion mechanics handles the conversion to json. I've stripped down the code for the slides, but the real code does HTML and EDN too.

Data transformation

(def api
  ["/"
   [["hello" (yada "Hello World!" {:id ::hello})]
    ["hello-atom" (yada (atom "Hello World!"))]]])

(defn add-security [api]
  (clojure.walk/postwalk
   (fn [handler]
     (if (instance? Handler handler)
       (assoc handler :authorization my-auth)
       handler))
   api))

And much more…

Basic/custom authentication
OPTIONS, HEAD, TRACE
Interceptor chains
Custom methods
Prefer, Vary, Allow
CORS
Service availability, Retry-After
Request validation
Journaling, cljs debug console
…

Async

Built on https://github.com/ztellman/manifold

Return a future

;; 'Hello World!' is now stored in a file
(yada (fn [ctx] (read "greeting.txt")))

;; We're about to do some IO, let's return a future
(yada (fn [ctx] (future (read "greeting.txt"))))

Or a promise

;; Asynchronous GET request with callback
(let [p (promise)]
  (http-kit/request
   {:url "www.google.com"}
   (fn [response] (deliver p response)))
  p ; Return the promise!
  )

Or any other deferred value

Here's a full search engine implementation!

(defrecord ClojureSearchEngine []
  Properties
  (properties [_]
    {:parameters {:get {:query {"q" String}}}})
  Get
  (GET [_ ctx]
    (aleph.http/get
     (str "https://www.google.com/q=clojure+"
          (get-in ctx [:parameters "q"])))))

;; Bidi!
["/search" (yada (->SearchEngine))]

Or you can return anything that manifold terms a 'deferred' value. If you need to make client requests to another HTTP service, and you use Aleph, it already returns deferred values so you don't have to do anything!

Last night in the sauna we were discussing the fact that many of Clojure library names, such as 'system' or 'boot' are totally ungoogleable. (apologies to Daniel and Micha, I didn't want to mention your names!)

Anyway, I said I'd try to develop a new search engine that would solve this problem. And here it is!

Let me explain what's going on. Our GET method calls into Aleph's client with a GET method. Aleph immediately returns a deferred value to yada, which in turn passes that back to the Aleph server it is running on. Later on, when Google starts returning a response, it is handled completely by Aleph. At no point are we blocking on I/O.

This is all done without core.async channels, go blocks or callbacks.

Async everywhere!

Server Sent Events

(defn hello-sse [ch]
  (go-loop [t 0]
    (when (>! ch (format "Hello World! (%d)" t))
      (<! (timeout 100))
      (recur (inc t))))
  (yada ch))

["/hello-sse" (hello-sse (chan 10))]

Full documentation

yada.juxt.pro

History & Roadmap

Started 2014-12-22
0.1.0 Release 2015-03-01
0.2.0 Release 2015-03-20
0.3.0 Release 2015-04-26
0.4.0 Release 2015-05-12
1.0-SNAPSHOT beta cycle started 2015-08-03
1.0 final slated for early 2016

yada

https://github.com/juxt/yada