Session & Settings Data
=======================

Decision
--------

Centralise access to settings inside a standalone module offering a
dictalike-interface. The settings can be loaded from and saved to files.
This currently uses JSON (as we have historically) but https://github.com/mu-editor/mu/issues/1203
is tracking the possibility of using TOML or some other format.

Settings objects have defaults which are overridden by values loaded from file
or set programmatically. When the settings are saved, only values overriding
the defaults are saved.

The ``load`` method can be called several times for the same settings; values in
each one override any corresponding existing values. The last loaded filename
is the file which the settings will be saved to. Both load and save attempt to
be robust, carrying on with warnings in the log if files can't be found, open,
read etc.

The existing files (session.json, settings.json) are implemented as singletons
in the settings module, and settings.json is autosaved. New settings to support
venv functionality -- in particular, baseline packages -- is also added.

At its simplest https://github.com/mu-editor/mu/pull/1200 does no more than
implement this set of functionality. The few places in existing code where
settings were used or altered have been updated to use the new objects and
functionality.

Not Implemented / Hooks
-----------------------

During the design and/or based on previous discussions, several ideas were
floated which are at least supported by the new implementation.

* Safe mode / Readonly mode / Reset mode

  As described below, there are situations where teachers or admins would
  like to reset settings for use in a club or classroom setting. The new
  implementation supports this idea via the ``reset`` method and ``readonly``
  flags without actually implementing it as such.

  Such functionality might, in the future, be managed by means of command-line
  switches or some other flag.

* File format: JSON, YAML, TOML...

  The implementation tries to be agnostic as to file format. At present it
  uses the historically-implemented JSON format. But the choice of serialiser
  is centralised towards the top of the module and shouldn't be hard to change,
  especially for any serialiser which uses the conventional ``.dumps``, ``.loads``
  API.

  cf https://github.com/mu-editor/mu/issues/1203

* One file / Two files?

  The new settings implementation facilitates any number of files each of
  which can have an arbitrary hierarchy. Whether we end up with one settings
  file containing, eg, session settings and board settings, or several files
  each specific to an area can be decided later. Nothing in this implementation
  precludes either approach.

* Interpolation

  Because it is easy to implement and doesn't seem risky, this implementation
  applies ``os.path.expandvars`` to any values retrieved. This will do
  platform-sensitive env var expansion so admins can specify, eg, a workspace
  directory of %USERPROFILE%\mu_code or $HOME/.mu/mu_code.

  Value interpolation (where one settings value can rely on another) has *not*
  been implemented. It's potentially quite an involved piece of work, and the
  benefit is not so clear.

* Indicating failure to users

  This is obviously a wider issue, but while this implementation tries to be
  robust when loading / saving settings, it only writes to the standard logs
  and then fails quietly. The problem here is that we're possibly not operating
  within the UI. At the least, we don't have a good overall story for a UI
  which isn't part of the central editor itself.

Background
----------

Mu maintains two files, automatically saved on exit, to hold user settings
and session data. The former contains critical parameters without which the
editor probably won't function. The latter contains more or less cosmetic
items which can be cleared (eg by a "Reset" button) without losing functionality.

Historically access to these files has been somewhat scattered around the
codebase, making it difficult for modules to access them coherently. The
first aim of the re-implementation is to create globally-accessible singletons,
much as is conventional for logging. Those "settings" objects would offer
a dictionary-like interface so that code could easily do::

    import settings

    def set_new_theme(theme):
        ...
        settings.session['theme'] = theme.name

The second aim is, possibly, to reconsider the use of the settings, or their
structures, or which / how many files they are and where they're situated.
Any such refactoring or restructuring should be a lot easier with a newer
implementation.


Discussion and Implementation
-----------------------------

Open questions:

* How many / which files do we need?
* Should we combine both settings / sessions into one file? Is there a meaningful difference which we want to maintain? [+1]
* Should we register exit handlers so the files are always saved on closedown? [+1]
* Should we write files to disc as soon as they are updated? [-0]
* Should we re-read files to allow users to update them mid-session? [-1]
* Should we implement read-only mode (ie the existing file is loaded but not written back)? [+0]
* Should we implement safe mode (ie the file is neither loaded nor written back)? [+1]
* Should we implement reset mode (ie the file is not loaded but is written back)? [+0]
* Should we break out the virtual environment settings (venv location, baseline packages) into its own file? [+1]
* Could we add a boards.json file to allow users to add new/variant configurations? [+0]
* What levels of config do we need? Defaults? One/multiple settings files? Override at instance level?
* Do we still need to look in the application directory as well as the data directory? [-0]
* What format should the files use? [cf https://github.com/mu-editor/mu/issues/1203]
* Should we save everything every time? [-0.5]
* Do we need interpolation of other settings? (eg ROOT_DIR = abc; WORK_DIR = %(ROOT_DIR)/xyz)
* Do we need interpolation of env vars? (eg ROOT_DIR = %USERPROFILE%\mu_code) [+0.5]
* Should we merge ``settings.py`` into ``config.py`` [+0]
* Should settings (as opposed to sessions) be read-only? [+1]

Exit Handlers
~~~~~~~~~~~~~

Registered exit handlers to ensure that files are saved when Mu exits. (This
could probably be alternatively achieved within the Qt app). The advantage of
this is that the save is automatic; the disadvantage is that it's a little
hidden.

Not currently writing to disc as soon as updated: having an exit handler ensures
the settings will be written, even in the event of an unhandled exception.
And it's not clear what advantage an "autosave" would offer.

Levels of Config
~~~~~~~~~~~~~~~~

Allowing three levels of data: the defaults for each setting type, held in
a class dictionary; possible overrides at class instantiation [I'm not clear
where this would be used; it can probably go]; and the .json files.

The ``load`` function merges into the existing settings. Most commonly this means
it'll be preceded by a call to ``reset``. But it could be used to implement a
cascade of settings, eg where an admin sets site-wide settings which are then
overridden by user settings.

Amnesia / Read-only / Reset modes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To support the possible "modes" above -- amnesia, read-only etc. there is a
``readonly`` flag on each settings object, preventing it from being written to
disc; and a ``reset`` method which will return to default settings. This last
can be used either to "forget" any loaded or set settings; or before reloading
from a different file.

So *Safe mode* is implemented by calling ``reset`` without ``load`` and setting ``readonly``.
*Read-only mode* is implemented by calling ``reset`` followed by ``load`` and setting ``readonly``
And *Reset mode* is implemented by calling ``reset`` without ``load`` and *not* setting ``readonly``

The use cases here would be mostly for admins or leaders who needed, eg,
to ensure that new sessions were started for every user, or who needed to debug
or recover from a corrupt settings file.

Failure modes
~~~~~~~~~~~~~

It's critical that we should recover well from not being able to read or to
write settings files, whether that's a file system failure or invalid JSON.
Regardless of the approach we should definitely log any exception, or log a
warning where there's no exception as such but, say, a missing file.

Reading
+++++++

* A failure to find/open a settings file is considered usual: it's expected
  that, the first time around, a user settings file won't exist to be read.
  The loader will log a warning and carry on as though it had found it empty
* A failure to read the JSON from a settings file is more complicated. For
  pragmatic purposes, the intention is here is: log a warning; quarantine the
  file; and carry on as though it had been found empty. That way the editor
  continues to work, albeit in "reset" mode, and the failing file is available
  for debugging.

  Not quite clear: should we automatically enter read-only mode in this situation?

Writing
+++++++

* A failure to open a settings file to write to is more problematic, and there's
  not very much we can do. Log the exception (eg AccessDenied or whatever).
  Perhaps -- given that the text won't be great -- pushign the JSON output to
  the logs as debug might give some manual fallback.
* A failure to *write* JSON is less probable -- although it does happen during
  testing where the JSON lib attempts to serialise a Mock object. Here, we can't
  really do more than log the exception and fail gracefully.

Levels of Config & Defaults
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The thrust of this proposal expects the `Settings` subclass to hold a dictionary
of defaults at class level. These are applied first before any file is loaded.
Any information from a loaded file is overlaid, so the file data "wins". Any
values not present in the file remain per the default.

Although not implemented in any way at present, the mechanism allows for several
files to be loaded in succession, typically for a site-wide file, set up by
an administrator, followed by a user-specific file. In this scenario, the data
would be read: Defaults < Site settings < User settings with later data
replacing earlier data.

The presence of the defaults in the `Settings` subclass should also make for
a more consistent use of defaults across the codebase. Eg if in general device
timeouts should be 2 seconds but can be changed, one piece of code might do::

    timeout_s = settings.user.get('timeout_s', 2)

while another piece elsewhere might do::

    timeout_s = settings.user.get('timeout_s', 3)

If the defaults are present in the class, the `.get` method could be implemented
so the default, instead of `None` as conventional, returns the class default::

    timeout_s = settings.user.get('timeout_s')
    # with no explicit timeout_s setting, timeout_s is now the default value

Taking this further, it's not clear that we even need to load the defaults as
such; we could always just fall back to them in the event of a .get KeyError
or even a __getitem__ KeyError. Taking that approach would also means we wouldn't
need the "dirty data" mechanism because anything in the ``_Settings`` object's own
``_dict`` should be saved out at the end.

Saving Everything?
~~~~~~~~~~~~~~~~~~

Implicit in the new design is the idea that settings are saved out to file(s) at
the end of every session.

Originally, the effect of the defaults was that, say, a workspace directory would
inherit the default which will then be written out to the settings file at the
end of the session. Even if that file had not originally had a settings for the
workspace directory.

On reflection, I've re-implemented for now a "dirty" setting for each attribute.
Only "dirty" attributes are written out to file. Anything loaded from a file
is considered "dirty" even if it remains unchanged for the duration of the
session. Anything updated during the session -- and this will typically be
user-configurable items like Zoom level, Theme &c. -- is also tagged as "dirty"
and will be written out to file.


Implemented via:
~~~~~~~~~~~~~~~~

* https://github.com/mu-editor/mu/pull/1200

Discussion in:
~~~~~~~~~~~~~~

* https://github.com/mu-editor/mu/issues/1184
* https://github.com/mu-editor/mu/issues/1203
