After cloning or parsing of a code fragment, absolute and relative positions have been defined. For seamless embedding, these might not be wanted and can be unset: MiscKit.unindent will do this for a subtree.

Comments

Java offers three kinds of comments: The SingleLineComment //..., the ordinary multi line Comment /* ... */, and the special DocComment /** ... */ in front of named member declarations. RECODER assigns comments to one adjacent program element, following an assignment heuristics considering empty lines (as sketched in the table below). Each ProgramElement can contain a list of comments, which in turn might preceed or follow this element (Comment.isPrefixed()). The pretty printer knows how to handle both cases.

######
// comment

######

######

// comment
######

######

// comment

######

As all heuristics it may fail, thus associating a comment with the wrong element. This has no visible consequences unless the element is moved away - then, the comment moves with the (wrong) element.

As of RECODER 0.6, missing implicit tokens impose further, technical problems. As ending positions are still missing, comments that follow their logical parent element might not be assigned to this element when there are no tokens to be attached to. For instance, comments in empty statement blocks are assigned to the next visible element rather then the empty block, as the closing brackets are not registered. Ending positions will allow to fix these remaining issues.

Comment update is a major problem and there is little hope to deal with this. However, when restructuring a program, the documentation must be brought up to date anyway. Also, most programmers use sparse comments only...

Future RECODER versions might evaluate DocComments for the tags @see and @link in order to update entity names as part of cross reference information. To do so, comments must become proper ModelElements, including the package.html files.

Declarations, References, and Program Model Elements

For each program model element of the abstract model, there is at most one syntactic declaration, and arbitrarily many references to that element. The source code model provides a hierarchy of recoder.java.Declarations in the recoder.java.declaration package and recoder.java.References in the recoder.java.reference package that correspond to the respective abstract model entities.

Note that the declarations of variables and fields are not VariableDeclarations but VariableSpecifications which are children of the former. This is a consequence of the C-like syntax that allows to write code like "int i, j, k;".

Resolving Declarations

Accessing the program model element of a declaration could not be easier: The corresponding source elements already implement the abstract element interface, so the Method of a MethodDeclaration md is the object itself: Method m = (Method)md; There are additional casting methods in the SourceInfo service for sake of consistency.

Resolving References

Accessing the program model element of a reference is handled by the SourceInfo service. The service will automatically resolve inheritance and overloading issues properly.

Note that references such as a.b.c in a parsed tree cannot be resolved by considering the local context only. For instance, a or b could be packages, types, or variables. After parsing, RECODER will insert an UncollatedReferenceQualifier for unknown references. This reference must be resolved using the SourceInfo method resolveURQ. In the cross reference configuration, RECODER will resolve these references automatically when creating cross reference information.

Retrieving References

Finding the declaration fitting to a reference is easy. The opposite direction is also available: The CrossReferenceSourceInfo as an extension of SourceInfo provides all known references to a given program model element. Obviously, cross referencing will work for known sources only and requires a closed world. The cross referencer will not find references inside class files, nor will it find references in source files that have not been touched yet.

Note that references such as a.b.c in a parsed tree cannot be resolved by considering the local context only. For instance, a could denote a package, type, or variable. The parser will insert UncollatedReferenceQualifiers for these problematic references. The SourceInfo contains a resolveURQ method to assign this type of references. Depending on the configuration of RECODER, all references might be resolved automatically, so URQs only occur in new syntax trees.

The CrossReferenceSourceInfo service can also deliver known subtypes of given types using the cross reference information.

Projects

RECODER provides a set of services bundled in a service configuration. Configurations ensure a consistent set of service implementations suited for a particular task. For instance, the closed world assumption is necessary for consistent transformation, but is not suited for demand-driven analysis.

Service Configurations

The DefaultServiceConfiguration contains only a standard SourceInfo service without cross reference information and is suited for pure one-pass analysis purposes only. Uncollated references will not become resolved automatically, although this is still possible by manual calls to resolveURQ.

The CrossReferenceServiceConfiguration will automatically analyse all references in the program model and add the cross reference information. This is the configuration of choice for transformational tasks.

ProjectSettings

The recoder.io.ProjectSettings service is a global repository for important settings such as

• the class and source file search path used to find input data
• the output path used to write changed sources to
• the formatting options using during pretty printing

The project settings service can also locate the proper class file archives containing at least the java.lang system classes, and is able to load and write property files containing the project information.

Project Files

Project files contain all properties of the ProjectSettings in textual form. A typical project file lists all source files:

#RECODER Project File
input.path=converter/original
output.path=converter/modified
units=ConversionPanel.java,DecimalField.java,Unit.java,Converter.java,\
ConverterRangeModel.java,FormattedDocument.java,FollowerRangeModel.java

Project files allow to make project settings persistent in an easy way.

Example Analysis Applications

public class Demo { 
    public static void main(String[] args) { 
        System.out.println(recoder.convenience.Format.toString("%N",
	    new recoder.CrossReferenceServiceConfiguration().getNameInfo().
	        getClassType(args[0]).getAllSupertypes())); 
    } 
}

Transforming Programs

Program transformation applications are the primary intent of RECODER. The architecture attempts to facilitate the task of writing transformations; transformations operate on the abstract syntax of a program and do not have to maintain derived data (e.g. changing the name of a type if its name has changed) or the concrete syntax (e.g. adding a comma if a supertype has been added).

The following sections will show how to modify syntax trees, how to print the results, and finally several sections about the proper use and definition of program transformations.

Modifying Syntax Trees

Concrete syntax of the language is controled by the ProgramFactory service. This module provides a lot of factory methods for all source elements (one method for each constructor available), direct access to the parser, and a factory method for the pretty printer backend. The ProgramFactory is also used internally by the SourceFileRepository.

Building a syntax tree is possible by

• Using the deepClone method to clone a node or subtree.
• Calling the parse methods of the ProgramFactory for a given text reader or a given string containing the code.
• Creating syntax nodes with the ProgramFactory and linking them "manually".

These alternatives differ in the following ways:

• Cloning a subtree is very fast and will produce completely linked trees, while the root node shares the same parent as the original root - the parent does not know the clone, though.
• The parse methods are not type safe and may yield parser exceptions, but it is a very convenient way to quickly provide glue code via a template string. Parsing will produce fully linked trees (except for the root node, of course).
• Attaching new nodes to a tree must handle parent links correctly, as the nodes are linked in both directions: One must not forget to update the parent link of the child properly. This can be tedious, but it is necessary. The makeParentRoleValid convenience method does a complete traversal of the known children and sets the proper parent links, which is useful if efficiency is not critical. Also, the Transformation base class provides auxiliary methods that take on the parent linking.

New CompilationUnit nodes are not added to a common root node but registered to the system via the ChangeHistory service as special case of an ordinary transformation report. This protocol is described later.

TypeDeclarationMutableList list = u.getDeclarations();
if (list == null) {
   list = new TypeDeclarationArrayList();
   u.setDeclarations(list);
}
list.add(d);
d.setParent(u);

Removal of subtrees is easy to do: Simply remove a child node from it's parents list, or set the child attribute to null. There is no obligation to update the parent link, but transformations usually will have to give a report on the changes they have performed in order to obtain a conforming model. We will get back to that soon.

Printing Syntax Trees

Creating concrete syntax out of an abstract syntax tree is the job of a PrettyPrinter. Pretty printers are created byte the ProgramFactory service and are initialized with the current ProjectSettings.

Top-level printing of compilation units is triggered by the SourceFileRepository which can print out all units, or only all changed units, or single units that it knows about. The destination is a file corresponding to the full path name of a unit and the output path as defined in the ProjectSettings. It is not possible to write back subtrees or to other destinations using this interface.

Each SourceElement also defines a convenient toSource() method which creates a string with a dump of the subtree represented by the given root element. The output has no initial indentation (it is "trimmed").

ProgramFactory pf = serviceConfig.getProgramFactory();
PrettyPrinter pp = pf.getPrettyPrinter(new PrintWriter(System.out));
s.accept(pp);

The Change History

The ChangeHistory service is the central agenda mechanism of RECODER. It propagates changes of the syntactic model to services that use this information to update the model elements they maintain. Changes are reported by the automatic class definition loader contained in the SourceFileRepository and by transformations.

In terms of design patterns, the change history serves as a mediator and a subject of an observer pattern. Currently changes are interpreted by the SourceFileRepository and the SourceInfo. RECODER 0.6 does not yet use the change information to full extend; the update mechanism is not very fine grained. Model updates are currently very expensive until the report handlers are refined.

The change history only propagates changes on demand. This allows to bundle change reports which reduces the amount of update phases. Services that require up to date model information will request a model update from the change history (via an updateModel() call). Transformations will not have to do trigger the updates by themselves; instead, updating service queries will perform this call transparently.

The change history maintains a queue of change reports for propagations. In parallel, the service maintains a stack of changes that is used for rollbacks.

Program Element Visibility

Usually program transformations will generate small amount of new code. This code obviously is not yet supported by the services. It is therefore important to know if a syntax element is part of the known model or temporarily "invisible":

A program element is visible for the RECODER services if and only if it is either

• a compilation unit and known to the source file repository, or
• its parent element is visible.

Obviously, service queries only work for visible elements. To make a new syntax element visible, transformations must report them as "attached" to the change history.

Change Reports

Change reports describe syntactic changes of syntax trees in terms of two atomic transformations: Attachment of a new subtree, and Detachment of a subtree. It is possible to describe any syntactic change using a series of these atomic transformations. Such a series is the syntactic "footprint" of a metaprogram. Transformations must send change reports for every visible change they have performed, before a model update becomes necessary.

In case of a model update, all incoming change reports will be propagated to the respective services who will traverse the reported trees and update their caches.

Change reports are also used for rollbacks. If a rollback is requested, the change history will revert and execute all changes to that point (a detach becomes an attach, and vice versa) and update all data structures accordingly. To allow nested rollbacks, transformations must inform the change history when they begin to make changes. Undo operations are possible until a commit is performed.

Changes may stem from different transformations and may be redundant. For traversal purposes, not all changed trees must be visited - it is sufficient to descent the largest subtrees only. Change reports that deal with subtrees contained in other changed trees will be marked as "minor" and can be safely skipped. This does not apply to undo operations, however.

Transformation Objects

Transformations in RECODER follow the Command design pattern and are materialized as objects. This allows explicit management of transformation, to do rollbacks, and to access intermediate results from within other objects. The base class of any transformation is recoder.kit.Transformation.

Transformations know the CrossReferenceServiceConfiguration they are running in, and offer a lot of convenience functionality. They provide quick access methods to all services (such as getSourceInfo()), as well as routines to detach, replace, and attach nearly all possible combinations of program elements. These helper methods take over the following steps:

• If the child must be inserted into a null list in the parent, a corresponding list is created and attached.
• The proper parent link of the child node is set.
• A correct change report is sent to the change history.

• attachAsGuard(Expression,LoopStatement),
• attachAsInitializer(LoopInitializer,For),
• attachAsUpdate(ExpressionStatement,For,int),
• attachAsBody(Statement,LoopStatement)

A high-level transformation is executed in several phases and must follow a proper protocol which will be described in detail when the requirements are clear.

Transformation Types

While in general, a transformation can use any service queries, some transformations are syntactic, such as the atomic transformations.

Sometimes, it is useful to modify newly created, invisible syntax trees. Transformations that modify these are also invisible and may be marked as such. Invisible transformations must be syntactic and do not send change reports.

To support invisible, syntactic transformations, the Transformation class offers versions of detach, replace, and the attach variants, which perform the changes but do not send change reports. They are static methods and have a do prefix: doAttach, doDetach, doReplace. These variants are also valuable for small invisible transformation steps as part of a visible transformation, e.g. to construct a small tree "on the fly" before attaching the result and thereby making it visible.

Composing Transformations

Meaningful transformations are usually chains of existing transformations. Unfortunately, composing transformations is not trivial. For total correctness (that is, partial correctness plus termination), dependencies between transformations must be taken into account: each change of the model might invalidate or restrict known results. This influences results of old meta model queries as well as results of syntactic traversals. Syntax trees might have removed, added, or just moved with new identities.

For consistency and efficiency, queries and modifications should be kept separate as much as possible to avoid unwanted side-effects and to reduce the number of model updates. The Transformation framework supports this by separating the analysis and the transformation phase.

Proper Behavior

In order to receive all infrastructural benefit, transformations must obey certain protocol rules and should follow some conveniences. A transformation operates in three phases:

1. A new transformation object is created.
   • The current cross reference service configuration (or a subtype thereof) must be passed as an argument.
   • Further necessary initial arguments should be checked for consistency and stored in attributes until they are needed.
   • If initial arguments could be meaningful for external transformations, they should be made accessible.
2. The analyze method should derive all data necessary to perform the changes.
   • No visible element may be changed.
   • A ProblemReport must be set (to the report field) and returned.
     The field is used to double check the protocol: it must be set to a NoProblem instance when transform is called. Callers use the ProblemReport to display warnings or choose alternative strategies. There are three constants for positive reports: IDENTITY (the transformation phase will not change anything), EQUIVALENCE (the resulting program shows the same functional observable behavior, excluding introspective data), NO_PROBLEM (no particular guarantees).
   • Relevant results of the analysis phase should be made accessible in order to allow reuse by callers. If other transformations are used, they can be made accessible to save additional access methods.
3. The transform method performs the syntactic changes.
   • As the first action, the change history should be informed about the beginning of the transformation. The easiest way to do this is to call the corresponding method in the abstract super class: super.transform().
   • During the transform phase, no model update may happen, that is, no updating query may be performed. It is admissible to perform syntactic navigation.
   • The method may perform any syntactic change, but must report any visible changes in the correct order before leaving. This is done automatically when the convenience methods of Transformation are used.
   • Detached subtrees may never be manipulated or attached again! To move a subtree, a clone should be attached instead, using the deepClone() method.
     Detached subtrees are no longer part of the visible model, however references to the root are still stored and might be processed later on, either for model updates or rollback operations. Therefore, when the subtree may not be changed - visibly or invisibly.
   • Changes should be made explicitly accessible for callers giving the proper motivation - what has been changed, and why? It is important for calling transformations to know which elements have been removed, added, or are replaced by clones, in order to properly update cached results obtained before the transformation is executed. If other transformations are used, they can be made accessible to save additional access methods.

Documenting Transformations

Documentation of a transformation should inform about the purpose of the transformation (its pragmatics), the syntactic changes, and the semantic consequences: Is the result identical, or does it preserve the observable behavior, or is it just compileable, or is it only parseable, or none thereof?

Using Kits

The kit auxiliary classes support the Transformation classes. The contain auxiliary queries and factory methods for often needed syntax trees. Kit classes are roughly grouped by the most important model element dealt with: UnitKit, TypeKit, MethodKit, and so on. The MiscKit deals with general syntax trees.

To give an impression of the contents of the kits, the following table lists a few auxiliaries:

Appendix: Auxiliaries

The following sections describe auxiliaries that are necessary to work with RECODER.

Type-safe Lists

The RECODER API tries to cover type constraints in detail. As the current version of Java is lacking generic types, this is hard to do with containers of unclassified objects. Containers that occur in public signatures are therefore expanded for concrete subtypes. What would be otherwise declared as a List of Identifier, or List<Identifier> becomes a IdentifierList in that approach. In contrast to the standard Java collection API, the RECODER lists distinguish between mutable and read-only lists: You can browse through a IdentifierList, but you can change a IdentifierMutableList only.

Lists of elements that stand in a subtype relation also stand in a subtype relation: An IdentifierList is a ProgramElementList. To prevent dangerous manipulations, this is not true for the mutable versions: An IdentifierMutableList is not a ProgramElementMutableList.

List elements are addressed by indices rather than iterators or explicit nodes. This saves storage and provides a slim interface. The default implementations end with ~ArrayList and use the common array doubling technique. If elements must be added and the number is known a-priori (e.g. before concatenating a set of lists), it is wise to increase the capacity of the array.

The following examples illustrate the usage of the RECODER lists.

for (int i = 0, s = list.size(); i < s; i += 1) {
   dosomething(list.getMethod(i));
}

int pos = list.indexOf(m);
if (pos >= 0) {
   list.remove(pos);
}
list.add(n);

Generating Formatted Debug Output

The recoder.util.Debug auxiliary class contains methods for error logging and assertion checking.

Conveniently formatted output is produced by recoder.convenience.Format. The Format.toString method takes a customizable format strings, such as "%c \"%N\" @%p in %f", to produce suitable output, such as MethodDeclaration "example.Main.main" @6/78 in example/Main.java. The recoder.convenience.Formats interface provides some predefined format strings.

Appendix: RECODER Services

The following sections briefly describe the available RECODER services.

recoder.io.SourceFileRepository Retrieves compilation units from given locations. Retrieves compilation units by logical class name. Gets a list of known compilation units. Writes back single, or all, or all changed units.

This service delivers syntax trees from data locations, files, or by automatic lookup in the search path obeying the order of the path and the priority of different file types (e.g. class files versus java sources). Usually, the repository is called by an analysis service, but a direct use is also possible, for instance to preload units. The service keeps track of the origins and changes of a compilation unit and is able to write it back to file. Compilation units are cached and will not be parsed twice.

recoder.io.ClassFileRepository Retrieves class files from given locations. Retrieves class files by logical class name. Gets a list of known class files.

This service delivers class files from data locations. Usually, the repository is not called directly, but this is also possible. The service keeps track of the origins of a class file and will cache the results.

recoder.io.ProjectSettings Provides project configuration information. Imports and exports project files.

This service delivers project configuration information such as the search path for source and class files, and pretty printing styles. It can import and export these properties as standard project files.

recoder.service.NameInfo Find packages, types and variable or fields by name. Manage predefined types and packages.

This service knows all packages, types and variables/fields by name. In addition, the name info can deliver representations of primitive types and important predefined types and packages. This interfaces comes close to the definition table of a compiler.

recoder.service.SourceInfo Analyzes compilation units. Provides containment relations. Resolves names in a context. Computes types of entities. Resolves inheritance, overloading, visibilities.

This service can analyze program elements and keeps track of compilation units that have been analyzed by the service. All methods deal with program elements, sometimes in combination with names, and deliver semantic properties, or relations between them. While hidden from the public interface, the service knows about scopes. It can resolve inheritance and containment, find out the meaning of uncollated names in a path and associate program elements with their corresponding semantic entity, types being the most important thereof. The service also relates a program element with its properties, hiding the exact nature of a semantic entity. If this service cannot handle a query because it does not correspond to a syntactic entity, the service delegates to the responsible one.

recoder.service.CrossReferenceSourceInfo Gets all known references to ProgramModelElements. Resolve all references it encounters automatically.

The cross reference source info is an extension of the standard source info and is part of the CrossReferenceServiceConfiguration. This service collates references automatically and provides queries that deliver all references to these entities under a closed world assumption. This service cannot report references that are not yet part of the model, obviously.

recoder.service.ByteCodeInfo Analyzes class files. Provides containment relations. Computes types of entities. Resolves inheritance, overloading, visibilities.

This service corresponds to the SourceInfo, but deals with Java byte code instead. The biggest syntactic unit is a ClassFile in contrast to a CompilationUnit. At the moment, RECODER does not allow to synthesize byte code and does not scan the byte code at instruction level, so many queries of the source level have no relevant counterpart in the byte code info yet.

recoder.service.ImplicitElementInfo Manages implicitly defined elements. Provides containment relations.

This service corresponds to the SourceInfo, but deals with implicitly defined elements such as array types or packages.

recoder.service.ConstantEvaluator Evaluate Java compile-time constant expressions.

This service allows to evaluate expressions that are defined as "compile-time-constant". This is a very conservative version of constants and particularly important for some special cases concerning the type of a conditional operator result (?:).

recoder.service.ChangeHistory Queue atomic syntactic changes. Propagate change events on demand. Manages user-level transformation transactions.

This service keeps track of syntactic changes in a worklist (queue) and can notify services on demand. This function is also invoked when classes are loaded on demand; hence the service is the backbone of the whole system. The ChangeHistory also keeps logical blocks of changes on a stack. These describe user-level transformations and can be committed or rolled back like usual nested transactions.

Appendix: Glossary

Concrete Syntax

The common sequential representation of a program source.

Abstract Syntax (Tree) (AST)

Tree describing the hierarchical structure of program sources; ASTs often omit details such as whitespace or separators.

Program Model

A model of a program. An AST is a syntactic program model.

Metamodel

A model of a model; the metamodel describes entities of a given base model.

Program Analysis

An algorithm that derives information about a program, usually by inspection of its sources.

Semantic Analysis

Program analysis that checks if a program source conforms to the language specification.

Program Transformation

Syntactic change of a program, usually described in terms of the abstract syntax.

Metaprogram

A program changing another program; an implementation of a program transformation using an analysis to drive the transformation. Static metaprograms operate offline, while dynamic metaprograms change programs during runtime.