= Programming at the REPL: Enhancing your REPL workflow
Valentin Waeselynck
2018-02-10
:type: repl
:toc: macro
:icons: font
:navlinktext: Enhancing your REPL workflow
:prevpagehref: navigating_namespaces
:prevpagetitle: Navigating Namespaces
:nextpagehref: guidelines_for_repl_aided_development
:nextpagetitle: Guidelines for REPL-Aided Development

ifdef::env-github,env-browser[:outfilesuffix: .adoc]

toc::[]

At this point, you know enough to understand how the REPL works; we'll now focus on giving
you a more streamlined development experience with the REPL. There are a variety of things
you may want to improve:

[quote]
____
**_Switching between my editor and the REPL is tedious._**
____

Most Clojure programmers don't use the terminal-based REPL for everyday development:
instead, they use a REPL integration in their editor, which enables them to write expressions
directly in their editor buffer and have them evaluated in the REPL with one hotkey.
See the <<editor-integrations,Editor integrations>> section below for more details.

[quote]
____
**_I want to do small experiments in Clojure, but it's painful to write code in the default CLI tool._**
____

As mentioned above, one solution could be to use an <<editor-integrations,editor integration>>.
Note that some editors such as **https://sekao.net/nightcode/[Nightcode]** are designed specifically to
provide a 'batteries-included' experience for Clojure.

However, if setting up an editor is too heavy for your taste, there also exist more ergonomic terminal REPL clients:

* **https://github.com/bhauman/rebel-readline[rebel-readline]** is a terminal readline library made by Bruce Hauman.
If you have already installed the <<xref/../../deps_and_cli#,Clojure CLI tools>>, you can launch it in https://github.com/bhauman/rebel-readline#quick-try[one line at the terminal]
without any additional installation steps.
* **https://github.com/Unrepl/unravel[Unravel]** by Paulus Esterhazy is another option, based on the https://github.com/Unrepl/unrepl[unrepl] communication protocol.

[quote]
____
**_I need to debug a program I'm running from the REPL._**
____

The REPL can definitely help you do that: see the <<debugging-tools-and-techniques,Debugging Tools and Techniques>> section below.

[quote]
____
**_I find myself repeating a lot of manual steps at the REPL for running my development environment._**
____

Consider creating a 'dev' namespace in your project (e.g `myproject.dev`) in which you define functions
for automating common development tasks (for example: starting a local web server, running a database query,
turning on/off email sending, etc.)

[quote]
____
**_When I make changes to my code, it's often difficult to reflect that change in my running program:
I have to do a lot of manual work in the REPL to make it happen.._**
____

Depending on the choices you make when writing your programs, interacting with them at the REPL will
become more or less practical. See the <<writing-repl-friendly-programs,Writing REPL-friendly programs>> section below.

[quote]
____
**_I would like to save my REPL sessions in a 'notebook' format._**
____

http://gorilla-repl.org/[Gorilla REPL] was made for this very purpose.

[quote]
____
**_I want better data visualization than what the REPL provides._**
____

You may get improved visualization features from a specialized Clojure editor:
 see the <<editor-integrations,Editor Integrations>> section below.

Having said that, keep in mind that the REPL is a full-feature execution environment: in particular, you can use it
to launch special-purpose visualization tools (including ones that you develop yourself).
For instance:

* **https://vlaaad.github.io/reveal/[Reveal]** and **https://docs.datomic.com/cloud/other-tools/REBL.html[Cognitect REBL]** are graphical tools for navigating and visualizing Clojure data, supporting two-way interaction with Clojure REPLs.
* **https://github.com/metasoarous/oz[oz]** is a Clojure library for displaying numerical charts
* **https://github.com/eggsyntax/datawalk[datawalk]** is a Clojure library for interactively exploring complex
Clojure data structures
* **https://github.com/walmartlabs/system-viz[system-viz]** is a Clojure library for visualizing the components of a running
Clojure system

[quote]
____
**_I want to customize my REPL._**
____

You can usually customize how your REPL reads, evaluates and prints, but the method to do it
depends on your toolchain. For instance:

* when using a REPL started from <<xref/../../../reference/repl_and_main#,clojure.main>> (as is the case when using the `clj` tool)
you can customize the REPL by launching a 'sub-REPL': see https://clojure.github.io/clojure/clojure.main-api.html#clojure.main/repl[clojure.main/repl].
* when using https://nrepl.org[nREPL].footnote:[
At the time of writing (March 2018), https://nrepl.org[nREPL] is the most popular toolchain
for REPL-editor integration], this can be done by writing custom https://nrepl.org/nrepl/design/middleware.html[middleware].

[quote]
____
**_I want to use the REPL to connect to a live production system._**
____

The Clojure <<xref/../../../reference/repl_and_main#launching_a_socket_server,socket server>> feature can be used
to that end. Tools like **https://nrepl.org[nREPL]** and **https://github.com/Unrepl/unrepl[unrepl]**
can be used to provide a richer experience.

[WARNING]
====
**_NOTE: You may not need all of this!_**

Depending on your projects and personal taste, you will most likely use only a fraction of the tools and techniques presented in this section.
It's important to know that these options exist, but don't try to adopt all of them at once!
====

[#editor-integrations]
== Editor integrations

All <<xref/../../../community/tools#_editors,major Clojure editors>> support a way of evaluating code in the REPL without leaving
the current code buffer, which reduces the amount of context switching a programmer has to do.
Here's what it looks like (the editor used in this example is https://cursive-ide.com/userguide/repl.html[Cursive]):

image:/images/content/guides/repl/cursive-repl-demo.gif["Editor REPL integration"]

[TIP]
====
**TIP:** You can wrap some expressions in a `(https://clojure.github.io/clojure/clojure.core-api.html#clojure.core/comment[comment] ...)`
block so that they don't get evaluated by accident when the file is loaded:

[source,clojure]
----
;; you would NOT want this function to get called by accident.
(defn transfer-money!
  [from-accnt to-accnt amount]
  ...)

(comment
  (transfer-money! "accnt243251" "accnt324222" 12000)
  )
----
====

=== What to expect from an in-editor REPL integration?

Here are some common editor commands provided by REPL integrations. All major Clojure
editors support a majority of them:

* **Send form before caret to REPL:** evaluate the expression before the cursor in the REPL,
in the namespace of the current file. Useful for experimenting in the context of the current namespace.
* **Send top-level form to REPL:** evaluate the biggest expression in which the cursor is currently contained
-typically a `(defn ...)` or `(def ...)` expression-in the namespace of the current file.
Useful for defining or re-defining Vars in a namespace.
* **Load the current file in the REPL.** Useful to avoid <<navigating_namespaces#_working_with_libs,loading libs manually>>.
* **Switch the REPL's namespace to current file:** useful to avoid typing `(in-ns '...)`.
* **Show evaluation inline:** displays the evaluation of the current expression next to it.
* **Replace expression with its evaluation:** replaces the current expression in the editor with its evaluation (as printed by the REPL).

[#debugging-tools-and-techniques]
== Debugging tools and techniques

While traditional debuggers can be used with Clojure, the REPL itself is a powerful debugging environment,
because it lets you inspect and alter the flow of a running program. In this section, we'll study some
tools and techniques to leverage the REPL for debugging.

=== Printing in-flight values with `prn`

`(https://clojuredocs.org/clojure.core/prn[prn] ...)` expressions can be added in strategic places in your code to print intermediary values:

[source, clojure-repl]
----
(defn average
  "a buggy function for computing the average of some numbers."
  [numbers]
  (let [sum (first numbers)
        n (count numbers)]
    (prn sum) ;; HERE printing an intermediary value
    (/ sum n)))
#'user/average
user=> (average [12 14])
12 ## HERE
6
----

[TIP]
====
**TIP:** you can combine prn with the `(https://clojuredocs.org/clojure.core/doto[doto] ...)` macro, i.e `(doto MY-EXPR prn)`,
to make adding `prn` calls less invasive:

[source,clojure]
----
(defn average
  "a buggy function for computing the average of some numbers."
  [numbers]
  (let [sum (first numbers)
        n (count numbers)]
    (/
      (doto sum prn) ;; HERE
      n)))
----
====

==== Going further: 'spying' macros

Some Clojure libraries provide 'enhanced' versions of `prn` that are more informative, by also printing information
about the wrapped expression. For example:

* the **https://github.com/clojure/tools.logging[tools.logging]** logging library
provides a http://clojure.github.io/tools.logging/#clojure.tools.logging/spy[spy] macro to log an expression's code along
with its value
* the **https://github.com/dgrnbrg/spyscope[spyscope]** library lets you to insert these printing calls with very
lightweight syntax.

==== Going further: tracing libraries

_Tracing_ libraries such as **https://github.com/clojure/tools.trace[tools.trace]** and **https://github.com/clojure-emacs/sayid[Sayid]**
can help you instrument larger portions of your code, for example by automatically printing all the function calls in a
given namespace, or all intermediary values in a given expression.

=== Intercepting and saving values on-the-fly

Sometimes you want to do more with intermediary values than just print them:
you want to save them to conduct further experiments on them at the REPL.
This can be done by inserting a `(def ...)` call inside the expression where the value appears:

[source,clojure]
----
(defn average
  [numbers]
  (let [sum (apply + numbers)
        n (count numbers)]
    (def n n) ;; FIXME remove when you're done debugging
    (/ sum n)))
----
[source,clojure-repl]
----
user=> (average [1 2 3])
2
user=> n
3
----

This 'inline-def' technique is described in more depth in https://blog.michielborkent.nl/2017/05/25/inline-def-debugging/[this blog post by Michiel Borkent].

=== Reproducing the context of an expression

When debugging at the REPL, we often want to reproduce manually something that our program did automatically,
that is evaluating some expressions inside a function body. To do that, we need to recreate the context
of the expressions of interest: one trick to achieve that is to define Vars (using `def`) with the same names
and values as the locals used by the expressions. The 'physics' example below illustrates this approach:

[source,clojure]
----
(def G 6.67408e-11)
(def earth-radius 6.371e6)
(def earth-mass 5.972e24)

(defn earth-gravitational-force
  "Computes (an approximation of) the gravitational force between Earth and an object
  of mass `m`, at distance `r` of Earth's center."
  [m r]
  (/
    (*
      G
      m
      (if (>= r earth-radius)
        earth-mass
        (*
          earth-mass
          (Math/pow (/ r earth-radius) 3.0))))
    (* r r)))

;;;; calling our function for an object of 80kg at distance 5000km.
(earth-gravitational-force 80 5e6) ; => 616.5217226636292

;;;; recreating the context of our call
(def m 80)
(def r 5e6)
;; note: the same effect could be achieved using the 'inline-def' technique described in the previous section.

;;;; we can now directly evaluate any expression in the function body:
(* r r) ; => 2.5E13
(>= r earth-radius) ; => false
(Math/pow (/ r earth-radius) 3.0) ; => 0.48337835316173317
----

This technique is described in more depth in Stuart Halloway's article
http://blog.cognitect.com/blog/2017/6/5/repl-debugging-no-stacktrace-required[REPL Debugging: No Stacktrace Required].
The **https://github.com/vvvvalvalval/scope-capture[scope-capture]** library was made to automate
the manual task of saving and re-creating the context of an expression.

=== Community resources about REPL debugging

* https://www.clojure-toolbox.com/[The Clojure Toolbox] provides a list a Clojure libraries for debugging.
* https://cambium.consulting/articles/2018/2/8/the-power-of-clojure-debugging[The Power of Clojure: debugging]
is an article by Cambium Consulting which provides a list of techniques for debugging at the REPL.
* _Clojure From the Ground Up_ by Aphyr contains a https://aphyr.com/posts/319-clojure-from-the-ground-up-debugging[chapter about debugging],
presenting techniques for debugging Clojure in particular and a principled approach to debugging in general.
* In his article http://blog.cognitect.com/blog/2017/6/5/repl-debugging-no-stacktrace-required[REPL Debugging: No Stacktrace Required],
Stuart Halloway demonstrates how the quick feedback loop at the REPL can be used to narrow down the cause of a bug
without using error information at all.
* Eli Bendersky has written some https://eli.thegreenplace.net/2017/notes-on-debugging-clojure-code/#id3[Notes on debugging Clojure code].
* https://www.youtube.com/watch?v=FihU5JxmnBg[Debugging with the Scientific Method] is a conference talk by Stuart Halloway
promoting a scientific approach to debugging in general.

[#writing-repl-friendly-programs]
== Writing REPL-friendly programs

While interactive development at the REPL gives a lot of power to programmers,
it also adds new challenges: programs must be designed so that they lend themselves
well to REPL interaction, which is a new constraint to be vigilant of when writing code.footnote:[
A similar phenomenon happens with the well-known technique of https://en.wikipedia.org/wiki/Software_testing[automated testing]:
while testing can bring a lot of value to programmers, it requires extra care to write code that is 'testable'.
Just like tests, the REPL should not be an afterthought when writing Clojure code.]

Covering this topic extensively would take us too far for the scope of this guide,
so we will merely provide some tips and resources to guide your own research and problem-solving.

**_REPL-friendly code can be re-defined._** Code is more easily redefined when it is called via a Var
(defined e.g via `(def ...)` or `(defn ...)`), because a Var can be redefined without touching the code that calls it.
This is illustrated in the following example, which prints some numbers at a regular time interval:

[source,clojure]
----
;; Each of these 4 code examples start a loop in another thread
;; which prints numbers at a regular time interval.

;;;; 1. NOT REPL-friendly
;; We won't be able to change the way numbers are printed without restarting the REPL.
(future
  (run!
    (fn [i]
      (println i "green bottles, standing on the wall. ♫")
      (Thread/sleep 1000))
    (range)))

;;;; 2. REPL-friendly
;; We can easily change the way numbers are printed by re-defining print-number-and-wait.
;; We can even stop the loop by having print-number-and-wait throw an Exception.
(defn print-number-and-wait
  [i]
  (println i "green bottles, standing on the wall. ♫")
  (Thread/sleep 1000))

(future
  (run!
    (fn [i] (print-number-and-wait i))
    (range)))

;;;; 3. NOT REPL-friendly
;; Unlike the above example, the loop can't be altered by re-defining print-number-and-wait,
;; because the loop uses the value of print-number-and-wait, not the #'print-number-and-wait Var.
(defn print-number-and-wait
  [i]
  (println i "green bottles, standing on the wall. ♫")
  (Thread/sleep 1000))

(future
  (run!
    print-number-and-wait
    (range)))

;;;; 4. REPL-friendly
;; The following works because a Clojure Var is (conveniently) also a function,
;; which consist of looking up its value (presumably a function) and calling it.
(defn print-number-and-wait
  [i]
  (println i "green bottles, standing on the wall. ♫")
  (Thread/sleep 1000))

(future
  (run!
    #'print-number-and-wait ;; mind the #' - the expression evaluates to the #'print-number-and-wait Var, not its value.
    (range)))
----

**_Beware of derived Vars._** If Var `b` is defined in terms of the value of Var `a`,
then you will need to re-define `b` each time you re-define `a`; it may be better to define
`b` as a 0-arity function which uses `a`. Example:

[source,clojure]
----
;;; NOT REPL-friendly
;; if you re-define `solar-system-planets`, you have to think of re-defining `n-planets` too.
(def solar-system-planets
  "The set of planets which orbit the Sun."
  #{"Mercury" "Venus" "Earth" "Mars" "Jupiter" "Saturn" "Uranus" "Neptune"})

(def n-planets
  "The number of planets in the solar system"
  (count solar-system-planets))


;;;; REPL-friendly
;; if you re-define `solar-system-planets`, the behaviour of `n-planets` will change accordingly.
(def solar-system-planets
  "The set of planets which orbit the Sun."
  #{"Mercury" "Venus" "Earth" "Mars" "Jupiter" "Saturn" "Uranus" "Neptune"})

(defn n-planets
  "The number of planets in the solar system"
  []
  (count solar-system-planets))
----

That being said, the problem of derived Vars becoming obsolete might be satisfactorily mitigated:

. either by making sure that Vars are not derived across different files, and by taking care to reload entire files when changes are made;
. or by using utilities like https://github.com/clojure/tools.namespace#clojuretoolsnamespace[clojure.tools.namespace], which let you keep track of changed files and reload them in order.


**_REPL-friendly code can be reloaded._** Make sure that reloading a namespace will not alter the
behaviour of the running program. If a Var needs to be defined exactly once (which should be very rare),
consider defining it with `https://clojuredocs.org/clojure.core/defonce[defonce]`.

When dealing with a codebase with many namespaces, reloading the appropriate namespaces in the correct
order can become difficult: the **https://github.com/clojure/tools.namespace[tools.namespace]** library
was made to assist the programmer in this task.

**_Program state and source code should be kept in sync._** You usually want to make sure
that your program state reflects your source code and vice-versa, but this is not automatic.
Reloading the code is often not enough: you also need to transform the program state accordingly.
Alessandra Sierra has expounded on this problem in her article http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-reloaded[My Clojure Workflow, Reloaded]
and her talk https://www.youtube.com/watch?v=13cmHf_kt-Q[Components Just Enough Structure].

This has motivated the creation of **_state management libraries:_**

* **https://github.com/stuartsierra/component[Component]**, which promotes a representation of program state
as a managed map of Clojure records called a *system*.
* **https://github.com/danielsz/system[System]** is a library on top of https://github.com/stuartsierra/component[Component]
which provides a set of ready-made components.
* **https://github.com/tolitius/mount[Mount]** takes a radically different approach as Component, choosing to
use Vars and namespaces as the supporting infrastructure for state.footnote:[At the time of writing,
there is controversy in the Clojure community regarding the relative merits of both approaches.]
* **https://github.com/weavejester/integrant[Integrant]** is a more recent library which shares Component's approach
while addressing some of its perceived limitations.
