Smith manual: building, testing and releasing fonts

A file called wscript needs to be created to control the build process. The convention is that this file is placed at the root of your font project source tree. This file is in fact a small python program, but the way it is run is designed to hide that as much as possible from the unsuspecting user.

Smith is really a larger toolchain with many dependencies that you install from a Docker registry which means you don’t have to install any of its fairly large number of components manually. In this manual, we assume you are using that approach as described at SIL Font Development Guide. Remember that smith is only the director and a wide range of utilities do the actual work. Installing just smith by itself (the python program) will not get you very far.

Besides taking inspiration from the way smith is used in public projects, like the ones available on github.com/silnrsi for example, you can use the following command to populate a brand new project with basic templates and standard folders:

Then you can start adjusting the various files to the specifics of your font project.

The heart of the build system is the wscript file that controls the build process. This is done by the python program creating a set of component objects. It then takes these objects and allows the user to run various build commands on them.

Waf, on which smith is built, works by creating a build directory into which all the results are stored. This is by design and a useful feature as it leaves the source directories pristine and makes for easy clearing up. The build directory is created using the command:

This process creates the new results directory, checks that all the tools that smith needs to achieve the build as described in wscript are available, and sets up various internal environment variables. Thus, if any changes are made to the wscript that indicate what extra tools are needed, then the configure command needs to be rerun.

After configuration you can now launch a build. This is done using:

This creates development artifacts of the various components configured to be built. But it does not create any publishable releases - or packages that you can share with someone else - these need another command:

This creates a zip of all the generated files and the corresponding documentation. Since this zip is targeted at Windows, text files have their line endings changed to CR LF. This is tagged with development version numbers.

This does the same work as the zip command except it uses LF Linux/macOS line endings and creates a .tar.xz, a compressed tarball. This is tagged with development version numbers.

This command does two things. First, it builds the various components, but marks the build for release. So things like font version strings no longer contain any development information in the form of git revision numbers, etc. Secondly, it builds the various release packages (zip and tarball). It also provides checksums, cryptographic signatures to allow comparison against the zip and tarball. This separate checksum file will allow to verify that what is distributed is really what has been produced.

This creates a subdirectory called graide/ that contains one .cfg file per font. This config file will be read by Graide, the Graphite IDE to allow easier testing and development of smart font behaviours. If the font has no graphite smarts, no configuration file is created.

This creates font tests output by chaining all the available tests. The Tests section will go into more details.

This removes the various files created by smith build in the build directory.

This removes the build directory completely, including any temporary files at the root of the project folder.

The wscript file is a Python program, but the internal environment is set up to minimise the amount of actual programming that needs to be done. There is no setup needed in the file, and object registration is automatic. It is possible to add waf specific extensions to the file, see details in the waf manual.

The basic steps needed to describe an entire build process is to create writing system component objects. These objects are font() or designspace and package(). Specific details on what information each of these objects requires is given in the corresponding sections of this document. Likewise examples are given in the sections.

We start with a simple, single font.

In line 1, we create a new font object that we want the system to build. We specify the target filename. This file will be created in the build tree (results or what is set in the out variable). Line 2 specifies where to find the source file. Notice that the target file is a .ttf file while the source is a .ufo file. Smith will use the necessary commands to build from one format to the other.

With this as our wscript file, we can build our font:

This is the first step in building any project. This command tells smith to set up the build environment and search out all the programs that it may need for the various tasks we may ask of smith. If a necessary program is missing smith will stop at that point and indicate an error. Some programs are not strictly necessary and smith can run with reduced functionality without them. Such missing programs are listed in orange. All other programs that smith searches for and finds are listed, along with their locations, in green. So you can see exactly which program smith will use for any particular task. This is hepful especially in cases where you may have a locally installed self-compiled version: you can more easily see if smith has found the version in /usr/local/ (or other local paths) instead of the stock packaged version.

This command tells smith to go and build all the objects the wscript says to be built. In this case just the simple Example-Regular.ttf which will appear in results. Not very exciting, but a good start.

Most font packages consist of more than one font file and this project is no exception. Can we scale our project to handle more than one file?

This example shows the power of integrating a description with a full programming language. wscript files are python programs, albeit very enhanced ones. So we can use any python-type constructs we might need. Usually the need is slight, and we show a typical example here.

Line 1 is the start of a loop. The lines below that are indented within the loop will be repeated for each value in the list. The first value is -Regular and the second is -Bold. Each time around the loop, the variable weight is set to the appropriate string. We will then use that variable to help set the appropriate values in the two font objects we are creating.

Each time around the loop, we create a new font object. In line 2 we create a new font object whose target font filename is dependent on the weight variable which is set to the various strings from the list at the start of the loop. So we will end up creating two fonts. One called Example-Regular.ttf as before, and one called Example-Bold.ttf. Line 3 gives the source files for each of these fonts.

It may seem easier just to expand out the loop and have two font() object commands but, as the complexity of this font() grows, we will see the value of using a loop. The advantage of adding the loop early is that we can make appropriate use of weight.

Now when we come to build this project, we will get two fonts:

It’s good that we can create multiple fonts, but what do we do with them then? There are two typical products that people want from a font project: a .zip file containing the fonts and corresponding files (with Windows line-endings CR LF) and a tarball (with Linux/macOS line-endings LF). Smith can create these two products from a wscript, but it needs just a little more information to do so:

Line 1 gives the base name of the products that will be created and line 2 gives the version of that product. Notice that the version variable is a string and does not have to be numeric. Case is important here, these are, in effect, magic variables we are setting that smith looks up.

To build this project, we do the same as before, but we can also use two extra commands:

smith zip will create Example-0.0.1-dev-(git-commit-id).zip in the releases folder inside the results folder. (The git-commit-id part, e.g. 66d16eM, will be the first 7 characters taken from the git revision id. We assume you are working from within a git repository.)

smith tarball will create Example-0.0.1-dev-(git-commit-id).tar.xz in the releases folder inside the results folder. (The git-commit-id part, e.g. 66d16eM, will be the first 7 characters taken from the git revision id. We assume you are working from within a git repository.)

Notice the -dev- suffix to indicate that these are development versions, which means they have not been tagged as a stable and tested release.

This zip, or tarball file, contains the four target fonts the build created, since we have now added Bold and Bold-Italic as extra weights in the loop.

We also added extra text files at the root of the project folder: README.txt, README.md, FONTLOG.txt, OFL-FAQ.txt. These are just text files and more documentation than font sources, but they are nice to have and will help users and other developers of your font. We will go into more detail on packaging in the dedicated section.

Before our example gains smart font support and grows in complexity, there is one area of control that is worth examining. For the most part, when creating a wscript one fills in the various 'forms' that create the objects, and smith knows what needs to happen to make things turn out right. But while this makes for pretty tutorials, real world projects have unique quirks that require the ability to add commands into the processing or to create things dynamically. In this exercise we will add a process to the source font:

The interest lies in line 8. Here we use a process() function to tell smith that we want it to run a command over the source font before converting it to a .ttf. A process() function takes a file which already exists (either in the source tree or one that is generated by another process) and then runs the list of cmd() function results over it in order. In this case, the command is to run a simple script that removes glyphs from the background layer in a UFO. The command string takes some study. The program takes two command line parameters, an input font file and an output font file. We represent these in the command string by ${DEP} (the dependent file) as the input and ${TGT} as the output file. smith will fill these in appropriately when it comes to run the command. In addition, note the initial '../' at the start of the command string. This is because all commands in smith are run from the results directory and so we have to go up one level to get back to the project root where the wscript file is and then from there we can navigate to the actual script.

The rest of the new lines in this exercise are simply extra variables being used to make the file easier to read, otherwise some of the lines would become excessively long and confusing. Notice that all the magic variables in a wscript that smith considers are all caps. That is if you use a variable name with a lowercase letter in it, you are sure to be safe from smith assuming some special meaning to that variable.

For the most part, we are not very interested in precisely what smith is doing to get the results we want, but sometimes it helps to know, and all that cryptic output streaming by isn’t much help. But there is a way to get something more helpful. First we need to get back to a completely pristine source tree:

Now we can configure and run in a way that has smith tell us what it is doing:

Thankfully, on a modern terminal, the colourising helps makes more sense of the voluminous information. But it is helpful once you learn to read it. The timestamped runner lines give the precise command-lines that are run at each stage of the build.

Clearly, the key to getting this output is in the command-line options to smith build. The -v says to output the extra information, (v stands for verbosity). But since smith tries to use multiple processors if you have them, to speed up the build (for example without the -j1, my build runs in 0.479s), it means the output can get interleaved. It is therefore wise to restrict smith to a single process (j stands for job) while outputting this information, and this is done using -j1.

In this exercise we grow our description to add OpenType and Graphite tables and also add some tests.

Line 12 tells smith how the OpenType tables will be generated for this font. It is possible to compile in VOLT tables, or, as here to use the internal description already in the font. The internal() does very little, but it does indicate to smith that the font has OpenType tables and that they should be tested.

Line 13 tells smith how the Graphite tables are to be added. There is currently only one form for Graphite source, and that is GDL. The gdl() object tells smith how to generate and bring together the various files that typically make up a Graphite description. A typical Graphite project has an autogenerated component (which is the first parameter to the gdl() (variable.gdl) and a common core .gdl file that is hand authored (the master parameter). Smith then does the work to generate the files and compile them into the font.

Line 15 talks about an attachment point database. This file holds information about glyphs in the font that cannot be held by TTF. Most importantly this file holds the positions of anchor points on the glyphs, and these positions are used when autogenerating smart code, either for volt() or gdl() or whatever. Depending on the source file format, the file may be autogenerated or be required as part of the source files.

These three lines are all it takes to add a sophisticated smart font build system to the font creation step. The rest of this section will look at basic font testing.

The basic principle of font testing in smith is that there is a directory containing test data, by default tests. This data is then applied to the various fonts and results are generated in the build tree. Test data can be of various formats, but the easiest to work with is simple .txt files that are treated as one paragraph per line files.

Line 4 gives the directory where the test files may be found. Since it is outside the project tree rooted in the directory containing the wscript, we have to specify where in the buildtree we want the test results to be put. Line 8 specifies that subdirectory.

Smith allows for user defined tests, but there are some defaults built-in, which we will examine here.

This tells smith to generate pdfs of each test file for each font for each smart font technology. That’s quite a few for each, but it means that you can look at any particular font and its smart rendering technology for each test. The files end up in results/tests based on the value of TESTRESULTSDIR.

So for example, the test file Short.txt will generate 4 pdf files: for the regular font: Short_Example_ot.pdf, Short_Example_gr.pdf and for the bold font: Short_Example-bold_ot.pdf and Short_Example-bold_gr.pdf. The _ot extension is used for OpenType rendered texts and _gr for the Graphite rendered texts. The texts are rendered using XeTeX.

The other file in the tests directory is tests.htxt. The h in htxt tells smith to preprocess the file to convert strings of the form \uxxxx into the corresponding Unicode character before rendering. This makes it easier to create test files.

Line 16 is an important line for OpenType testing since it specifies which script to use when running the OpenType shaping engine.

Another aspect of testing is regression testing. Can we find out what has changed between this font and a known good version? The way this works is that we store known good versions of the fonts and then have smith run tests against both fonts and compare the results. The default directory to keep the font files in is references/.

The results end up in the TESTRESULTSDIR/tests directory as .html files. If there are no differences, the files are 0 length.

A further target that is useful is the ability to create font reports that show all the glyphs in a font. We set this as a font product rather than a kind of test, in line 17. The default target filename is the same as the target .ttf file but with a .pdf extension instead. The file is built as part of smith build.

There are two other targets that this wscript enables:

Line 9 specifies a string that will be used in creating the waterfall files and also the cross font summary files. smith waterfall creates one file per font and technology and stores it in the waterfall sub directory of the TESTRESULTSDIR, prefixing each font and technology with waterfall. smith xfont creates one file per technology in the TESTRESULTSDIR called Crossfont_ot.pdf or Crossfont_gr.pdf that contains the test string output with the font name, one per line.

Another feature of smith is its ability to integrate with graide. Graide is a graphically based IDE for developing GDL Graphite source code. It also incorporates a Graphite debugger to help font developers see how their code executes.

Running smith graide causes smith to create graide configuration files in a graide/ subdirectory. This is one of the few commands that creates files outside of the results/ tree. The user can then run graide referencing one of these configuration files. One file is made per font.

A word of advice. Since, most often, smith does not generate .gdx files when it runs grcompiler (.gdx files are grcompiler debug files), it is best to recompile the font on loading into graide.

The configuration is designed to restrict graide to just editing GDL. If you want to use graide to adjust attachment points or add them, then you will need to enable writing to the AP.xml, in the graide configuration, and you are then responsible for propagating those changes back from the AP.xml to your source font.

So far we have concerned ourselves with the mechanics of font creation. But in order to release a font package we also need to concern ourselves with the metadata that is involved in producing a font release.

We have used an existing font: Andika Mtihani as our base font. The font has been released under the OFL (Open Font License) and so we are free to modify and develop our own derivative under that license. To avoid any confusion it is better to change the name to something different for our derivative, for example: Foobar. We do this in a number of places in the wscript: Line 19 changes both the name of the font file generated (and all derived products), but it also processes that font file to change the internal name to "Foobar", using a process() and a name() function that acts like a cmd() that is suited to font renaming.

We also set the internal version of the font using a version parameter.

The Web Open Font Format (WOFF) is designed particularly for distribution of webfonts and smith can generate such files from the target .ttf font file. The default parameters for this object, take the font target filename as the basis of the woff filename, which is sufficient for our needs. Both in v1 and v2 of the format.

This section is for those interested in doing more advanced types of testing. For most projects there is no need to go to this level of complexity and many users never need to use these capabilities. So this exercise has been placed after the exercise that pretty much completes font creation. We also try to introduce as many advanced techniques as we can, even if the results end up being a little contrived.

Font testing is not limited to just the inbuilt test types. Smith supports the integration of other test programs as you the user desires, so long as they are command line based, non-interactive and report generators.

The interesting section is in lines 17-21. These lines create a fonttest object that is then referenced within the font at line 34. A fonttest object adds new smith commands. This example adds the three smith commands: pdfs, test and report. Notice that the smith pdfs command is actually implemented using a fonttest() object. The targets parameter to fonttest uses a python data structure called a dictionary. This is indicated by the { at the start (and } at the end). Dictionary elements consist of a string before a : and a value after it. The value before the : is known as the key and the value after as the value. So a dictionary is set of key, value pairs. In our case, the keys here indicate smith commands and the values are the test objects that get executed for the command.

The first two commands use default test objects appropriate to the type of command. The pdfs command executes a tex() object that does all the xetex processing of test files. Likewise the test command executes a default tests() object which implements the regression testing.

Our new command report also uses a tests() object. But in this, we give another dictionary of key, value pairs. The key is a subdirectory under the TESTRESULTSDIR and the value is a cmd() object that gives the command to execute. In this case, we are running the uforeport script. The reference to '${SRC[0]}` says to use the first element from the inputs. The inputs has 3 elements: the font, the text file to test and the corresponding references/ font file. We only need the first of these and list indices all start from 0 in python. In addition, we use the parameter coverage to say that we only want to run tests one per font, and not one per test file per shaper per font. The > ${TGT} says that the output that the program produces, which would normally be printed on the screen is to be sent to the target log file instead.

Another thing we have changed is that rather than hardwiring various of the specialist programs into our wscript, we now will get smith to go and search for them. We introduce another new Python concept: the function. Each smith command will search for a function in your wscript with the same name as the command and will execute it. For more information of what to do then you should read the manual for the underlying framework that smith is built on, which is waf. The variable passed to us is a waf context that can be used to do various things like add commands to the build process, etc. In our case we want to have smith search for various programs. find_program() is the key that tells smith to search for the programs. In the case of ufobackgroundclean, that is necessary for the build, so if it is missing we want the configuration to fail. But in the case of the test script, we only lose the ability of that one test type if the script is missing, so we don’t want to fail the configuration. This is a marginal call, but we do at least get to see the pattern for achieving this. In each case find_program() takes a list of paths to search, and it only searches those directories, not directories below those, unless explicitly listed.

This final example shows how to use the designspace() object.

There are multiple ways of adding OpenType information to a font. One is to already have it in the source font. In this case, we need to indicate that we are working with an OpenType font, even if everything is internal to the font. The font builder needs to know for font testing purposes or if the font is generated from a legacy font.

This will generate tests pertinent to OpenType testing. See the section on font tests.

One approach sometimes used for FontForge based projects is to keep all the lookups in one font and then to share these lookups across all the fonts in a project. For this we simply specify a sfd_master attribute and the font builder will use sfdmeld to integrate the lookups in the master into each font as it is built. (You will have to install FontForge yourself as it is no longer part of the smith toolchain default dependencies)

Obviously, if the sfd_master attribute is the same as the source file then no merge occurs. This is an alternative way of specifying that the font is OpenType.

Another approach to adding OpenType tables to a font is to use an external tool or text file to create the lookups and then to have smith compile them into the font. Two formats for source files are supported: Microsoft’s VOLT (Visual OpenType Layout Tool) and Adobe’s Feature File.

Feax is a set of extensions to provide easier and more powerful ways to write fea code. It is a fea preprocessor. For the specification of the feax language see feaextensions.md.

makefea is the script to generate fea from a feax source file.

Adding Graphite tables to a font is much like adding VOLT information. The relevant files are declared either to the font or a gdl() object. For example:

Notice that the ap and classes attributes have the same values and meaning as for OpenType tables. This is because the information is used in the creation of both sets of tables. The Example.gdl is created by the make_gdl process and it pulls in mymaster.gdl during compilation.

The complete list of attributes to a gdl() object are:

Fonts can also be built from another font, either legacy-encoded or generated from a source font or fonts. This can be achieved by giving a legacy() object as the source attribute for the font. For example, for a font generated from a legacy font using ttfbuilder we might do:

The legacy object creates the source font that is then copied to the output and perhaps smarts are added too.

The complete set of attributes to a legacy() object is:

WOFF and WOFF2 files are TTF files in special compressed formats used for webfont delivery. Smith can generate both WOFF and WOFF2 files. For example:

The first parameter to woff() is the name of the woff file(s) to be generated. Filename extension, if present, is ignored.

The woff object takes these optional attributes:

By default, the font version is extracted from the input ttf and used as the version for the woff font. To override with a specific version use the params attribute:

To use a command other than psfwoffit to create woff files, the cmd attribute can provide the desired command and its options. For example, to use ttf2woff to create woff file:

Fret is a font reporting tool that generates a PDF report from a font file, giving information about all the glyphs in the font.

The fret object takes these attributes:

An alternative to the font object is the designspace object. A designspace specification normally defines a family of related fonts, and therefore the designspace object typically results in a number of fonts being generated — in essence the designspace object creates multiple font objects. Most of the attributes of a font object also apply to a designspace object, the differences are described below.

Instead of a source attribute, the designspace object uses a designspace file. Each instance described in the designspace file is treated as a source, and the designspace object iterates over all these instances and builds output from each.

Thus the minimum needed for the designspace object is a designspace file and target attribute:

Except for source and sfd_master, all other attributes of the font object can be used with the designspace object. Additionally the following attribute can be used:

Note, however, that in contrast to the simplest font object, the target attribute cannot be as simple as Example-Regular.ttf but must be an expression that yields an appropriate filename for each instance. This will also be true for some other attributes as well, for example the attachment point information specified by the ap attribute will need to be different for each instance.

To facilitate this, the designspace object provides a number of variables whose value is based on the particular instance being processed. To prevent possible name conflicts, the designspace object uses a DS: prefix for each of the variables it provides.

For a given instance, each attribute and each location introduce one or more variables. Consider the following instance definition:

Based on the corresponding instance attributes, the following variables will be defined:

Additionally, for each variable above, three additional variables are defined. Adding _DASH to the variable name results in a value where all spaces are replaced with a hyphen. Adding _NOSPC produces a value where all spaces are removed. Finally, adding _BASE provides a value which is the basename (without the extension) of the original value. For example:

Based on the location specified for the instance, the following variables are defined:

One additional variable provides the path from the build directory to the instance UFO, which for our example would be:

There are various source file types that can be used as the basis of many tests. These are:

Test source files are stored in a standard place in the tree. The global variable TESTDIR can be used to specify where that place is, but the default is tests/. There are a number of test targets already defined in smith that can make use of these and other test information. The TESTDIR may also be a list of paths. The list of test directories can be extended with those specified in EXTRATESTDIR which may also be a string or list. These extra directories may be overridden using a ;-separated list specified in the SMITH_EXTRATESTDIR environment variable. This may, in its turn, be overridden by a ;-separated list specified in the command line option --extratestdir. If any of the directories in the list of test directories does not exist, it is quietly ignored.

Each test target, whether standard or user defined, creates a .html file in the results/tests directory (or as specified by the TESTRESULTSDIR variable). The file name is target\index.html from which links to the actual test results can be found.

Sometimes you want to create test files as part of the build. This can be done using testFile(). It takes the same parameters as for a create() and it does create the file, but it also adds it to the list of source test files as if it was stored in the tests/ directory (or wherever you have specified that test files are stored).

It is possible to add your own tests to the smith test system. One can create a variant of one of the standard tests listed above, and associate it with a new target. Or one can run a separate command to execute the test. The test is specified using a testCommand() function that takes a single fixed parameter of the target the command is to be associated with and a list of named parameters. It is possible to specify more than one testCommand be associated with the same target, in which case all the testCommands will be run when that target is specified.