The Composer Tool

The composer is a command-line tool to compose files or directories by invoking type-specific composition functions on its operands. Some of the composition functions are implemented by other composition tools while others are handled internally by the composer itself. Together, the composition functions encompass several different file types, including .jak files, .properties files, and directories.

The composer's most important use is to compose directory hierarchies representing entire layers. In this mode, the composer recursively descends the hierarchies, identifying and composing matching files within the hierarchies. This document discusses the use of composer with special attention to the composition of layers.


Composer Basics

At a minimum, the composer requires one or more source operands listing files or directories to be composed and a target naming the file or directory into which to place the result. The source operands and the target may be specified in one of two ways. First, the target can be specified explicitly using the --target option, while the source operands can also be specified explicitly as non-option arguments. Here's an example command line of this type with two source arguments, base.jak and refinement.jak: 

composer --target=result.jak base.jak refinement.jak

Alternatively, both the sources and target can be provided in an equation file specified via the --equation option. If a file named result.jak.equation is defined to contain two whitespace-separated operands, base.jak and refinement.jak in that order, then the same effect as above can be achieved with the following command line: 

composer --equation=result.jak

In this command, the value of the --equation option specifies the base name of the equation file. The base name is also used as the default target name and the --target option can be used to override this default. More information can be found in the equation files section.

Now, what does the composer do when invoked with the above command? If base.jak and refinement.jak are readable files, the composer uses the file extension .jak to recognize that the source file types are those of Jak source files. In general, there may be multiple source types which will be resolved into a common type, if possible. Then, the common type is used to select a composition method. In this case, since base.jak and refinement.jak are both of type .jak, the common type is also .jak and, by default, the composer is configured to invoke mixin to compose .jak files. In this example, the resulting output would be written into the target, result.jak.

There are some important points to note. First, the source arguments are listed in a specific order. The left-most source argument is the base of a composition and successive source arguments are refinements applied in order from left to right. Second, the source arguments may be of different types as long as they can be resolved to a common type. Third, the type of the target argument is not used in determining the common type of the source operands. In particular, it's perfectly valid to place the result of a composition into a file of a completely different type than that of the sources.

Of course, the composer has other command-line options and it can be configured for several different compositions. Arguably, though, the most important composition handled by the composer is the composition of directory hierarchies that represent layers.

Composition Overview

In the example above, the composition of .jak files is handled by an external tool instead of an internal composition function. In the default composer configuration, .jak files are primitive operands as are all source files handled by external tools. More generally, though, the composer operates on units which may be directories, files or portions of files. Units are divided into two types: primitives, which are treated by the composer as opaque objects, and collectives, which are collections of named units.

All compositions, whether of primitives or collectives, are determined via the method suggested in the previous section. Given a list of source units, a common type is determined if possible. The common type is used to select a composition function via a configuration file. The source units are passed to the function for evaluation and the result is written to a target unit. If the source units are collectives, the composition function is a generalization of inheritance as implemented in object-oriented languages.

Here's how collective composition works. Suppose there are two collectives, the Base and the Refinement, each containing uniquely named units called members. For example, the Base could contain members named alpha and beta, while the Refinement could contain members named alpha, delta and gamma. Conventionally, the member names are called signatures and the collection of signatures for a collective is called the signature set. In our example, the signature set for Base is {alpha, beta} while the signature set for Refinement is {alpha, delta, gamma}.

Further, each member of a collective has an assigned value. For example, in object Base, suppose the member alpha has value 120.0 while member beta has value 2718. It's conventional to use dot notation to refer to a member within a collective. For example, Base.beta refers to member beta within collective Base.

Of course, the members of Refinement also have assigned values. In the table below, the collectives Base and Refinement are depicted with their members where the possible signatures are listed in the last four rows while the assigned values for the members of Base and Refinement are shown in the second and third columns, respectively. The intersection of a signature's row and a collective's column is blank if the collective has no member with that signature. For example, Refinement has no member with signature beta, so the cell at the third column of the row for beta is left blank.

Collectives Base and Refinement
Signatures Assigned Values
Base Refinement
alpha 120.0 123
beta 2718  
delta   0.005
gamma   "label"

The composition of two collectives (the sources) is another collective (the target) which also contains uniquely named members. The target's signature set is the union of the signature sets of the sources. So, if the sources are Base and Refinement, the target's signature set is {alpha, beta, delta, gamma}. The values assigned to the target's members are calculated by recursively applying composition for each signature in the target's signature set. The operands for each recursive composition are found by taking corresponding members from the sources in the same order as the sources. For example, if collective Target is the composition of Base and Refinement, in that order, then the value of Target.alpha is found by composing Base.alpha and Refinement.alpha in that order. The composition function to use in the recursive invocation is again determined as described above and the procedure repeats until no further compositions of collectives remain.

Here's a table showing the recursive compositions formed when composing Base and Refinement to yield collective Target. Only Target.alpha has two operands in the recursive composition since signature alpha is the only signature with members in both Base and Refinement.

Composing Base and Refinement into Target
Signatures Assigned Values
Base Refinement Target
alpha 120.0 123 compose (120.0, 123)
beta 2718   compose (2718)
delta   0.005 compose (0.005)
gamma   "label" compose ("label")

So far, the composition of numbers and strings has been left undefined. To complete the example, let's suppose that strings are composed by concatenation and that numbers are composed by subtraction. Then, strings and numbers will be primitives and the final result of composing Base and Refinement, in that order, is shown in the table below.

Composing Base and Refinement into Target
Signatures Assigned Values
Base Refinement Target
alpha 120.0 123 -3.0
beta 2718   2718
delta   0.005 0.005
gamma   "label" "label"

It's important to note that composition is not usually commutative. For example, if the order of the above composition were reversed, the value of Target.alpha would be +3.0 instead of -3.0.

Using the model path. Source operands are often part of a model, a collection of related components used to define a product line. The composer supports the model viewpoint by providing a search path, called the model path, for source operands and other related files. This is a list of directories separated by the system-dependent path separator character. On Unix systems, the path separator character is ":" and, on Windows systems, it is ";".

A model path can be specified with the --model command-line parameter. For example, on a Unix system, the composer could be run as follows: 

composer --model=myGUIComponents:/opt/components/GUI baseFrame editor

In this example, the composer would search for each of the two operands, baseFrame and editor, by first looking in the myGUIComponents directory, then in the /opt/components/GUI directory. Each search is done independently, so that baseFrame could be found in /opt/components/GUI while editor could be found in myGUIComponents. It's also possible to specify the model path by defining the property composer.model.path in a properties file.

Details of the model path search. The model path search is called resolution and it is similar to other path searches, such as the resolution of Java class files against the CLASSPATH environment variable. For example, only relative filenames are resolved against the model search path; an absolute filename is always used exactly as given. On the other hand, model path resolution differs from other path searches in two important ways. First, any relative filename, even those specifying a directory path, will be resolved against the model path. For example, the filename dsl/bali/Data.jak, which contains the directory path dsl/bali, will be resolved against the model path. Second, a source operand may be specified as a file URI. This means is that a "/" can always be used as the file separator character in source operands, even on Windows systems where the file separator character is normally "\".

Composing Layers (Directory Hierarchies)

In the composer, layers are implemented as directory hierarchies and directories are particular types of collectives. The members of a directory collective are the files within the directory and the signature of each file member is the name of the file. Composition of directories, then, is implemented as described in the previous section. To ensure that this is clear, this section contains a fully described example.

Suppose that the following composer command is issued where base, refinement and packages are all directories: 

composer --target=project base refinement packages

Let's suppose that the directory base contains files alf.jak and bet.jak while directory refinement contains files bet.jak and gam.jak. Further, let's suppose that both base and refinement contain subdirectories named network each with a single file named network.properties. Finally, let's suppose that the directory packages contains one subdirectory named network with Java files named Control.java and Link.java.

Viewing base, refinement and packages simply as collectives with member signatures determined by filenames results in the following tabular representation:

Collectives base, refinement and packages
Signatures Assigned Values
base refinement packages
alf.jak a .jak file    
bet.jak a .jak file a .jak file  
gam.jak   a .jak file  
network a directory a directory a directory

Since the sources are all directories, the target, project, will be instantiated as a directory and its members will be computed by recursively applying composition across members of sources with the same signatures. This results in the following sequence of events initiated by the composer, including several recursive invocations of the composer:

The first three recursive invocations of composer act on .jak files which are primitives. However, the fourth invocation is on sub-directories of the original source directories. Since directories are collectives, the composer again applies composition of collectives to create the target sub-directory project/network. Here's a tabular representation of that composition:

Collectives base/network, refinement/network and packages/network
Signatures Assigned Values
base/network refinement/network packages/network
network.properties a .properties file a .properties file  
Control.java     a .java file
Link.java     a .java file

This results in the following sequence of events with three recursive invocations of the composer:

At this point, all compositions are for primitives, so there are no further recursive invocations of the composer and no more compositions are performed. However, for top-level compositions of directories, the composer performs one final step. If (and only if) the target directory contains an Ant build file named build.xml and if there were no errors in composition, then the composer will start the build tool, Ant, with a command line such as the following: 

ant -buildfile <target-directory>/build.xml

For the example above, the top-level target directory is project, so the command issued is: 

ant -buildfile project/build.xml

For sophisticated builds, a build.xml file can be created as the result of a recursive composition performed while the target directory is being composed. Even though the build.xml file is generated by the composition itself, it will still be invoked in the final step as described above. A particularly useful method for generating a build.xml file is as the result of composing build.xml.vm files in the source directories. With the default file type configuration, .vm files are composed using Velocity as described below.

Basic Equation Files

When composing large numbers of files, it's inconvenient to enter the operand names on the command line each time the composition is run. It's much easier, in this case, to enter the operand names just once into an equation file. Then, the composer can be invoked so as to read the operand names from the equation file. This section describes basic equation files which, in the default file type configuration, are files with the .equation extension. A alternative format for more complex equations files is described in the next section.

Consider the example from the previous section. There, three directory operands (base, refinement and packages) were being composed into target directory project. Now, suppose we create an equation file, project.equation, that contains the following lines: 

base
refinement
packages

Then, the composer can be invoked with the following command line: 

composer --equation=project

This command line specifies several pieces of information. First, the base name of the equation file is project, as given by the value of the --equation argument. If desired, it's also valid to specify both the base name, project, and the extension, .equation, as shown below: 

composer --equation=project.equation

Either way, the base name of the equation file is also used to supply the second piece of information, the default target name. So the above command will place the result of the composition into a directory named project. The default can be overridden by supplying an additional --target argument. For example, 

composer --equation=project --target=george

specifies that the composition result will be named george instead of project.

Finally, the third set of information, the source operands, is read from the equation file. In this example, there are three source operands, one per line, in the equation file project.equation. Therefore, in this example, the following two command lines are equivalent: 

composer --equation=project
composer --target=project base refinement packages

Note: Comment lines can also be placed into equation files. If the first non-blank character on a line is a #, the line will be treated as a comment.

Composing equation files. So far, the examples have shown how equation files can be used to save typing. That's a bit boring, though useful. However, equation files themselves can be composed, which makes life much more interesting. This allows equation files to specify components that can be used in later compositions. Further, equation files implement a form of inheritance, allowing the same set of equation files to be composed in different ways to build different components.

Here's how inheritance works. There's a keyword, super, that can be used as the name of an operand within an equation file. The super keyword specifies that a base list of operands be inserted into the operand list. To make this clear, suppose there are two equation files as shown in the table below:

Files base.equation and packages.equation
File: base.equation packages.equation
Contents:
base
refinement
super
filesystem

These two equation files can be composed. Here's an example command line for such a composition: 

composer --target=result.equation base.equation packages.equation

The composition works from left-to-right. The contents of the first file, base.equation, provides the base to substitute for the super keyword in the second file, packages.equation. The result will be the following three lines 

base
refinement
filesystem

which are written into the target file, result.equation. It's also possible to compose more than two equation files. For example, suppose there's another equation file, network.equation, with the following contents: 

ip-v4
tcp
super
ftp

In this file, the keyword super occupies the third line (in general, super can be at any position in an equation file). Now, to continue our example, suppose we compose these three equation files with the following command: 

composer --target=result.equation base.equation packages.equation network.equation

The composition of the first two files, base.equation and packages.equation, is used as the base for substitution into the third file, network.equation. The result, written into file result.equation, will be the following lines: 

ip-v4
tcp
base
refinement
filesystem
ftp

The order of composition is important! Suppose we modified the above command line to reverse the last two operands, as shown below: 

composer --target=result.equation base.equation network.equation packages.equation

Then, the lines written into the target file will be in a different order: 

ip-v4
tcp
base
refinement
ftp
filesystem

The operand order must be arranged so that the result is in the desired order. Assuming that the order is correct, the result is another equation file that can be used in a following composition. For example, the target of the previous composition could be used to specify another composition as follows: 

composer --equation=result

The simple examples above give just a flavor of the possibilities. It's left to the reader to develop more sophisticated applications!

More Complex Equations Files

It's often the case that compositional designs are hierarchical, with applications being composed from sub-components which are themselves composed from smaller sub-components. Such designs can be represented with .equation files as described above. Sometimes, though, more compact representations are desired. These can be achieved with .equations (note the plural!) files as described in this section. The format for these files is the same as that specified load method for Java properties files. In particular, a comment line can be included by beginning the line with a "#".

For example, consider an application that is designed to run on a variety of different operating systems. Typically, such tools are composed of a portable component and a non-portable components that varies according to operating system. A design can insulate the portable component by providing an adapter to convert tool functions to O/S-dependent functions. It's natural to define an O/S adapter as a separate component, varying according to operating sytem, that is composed with the portable component to build a complete instance of the application. However, an O/S adapter is likely to be composed of several smaller components, as is the portable component. Here's a .equations file, testVersion.equations, that shows an example hierarchical design: 

# Base version with test stubs for the O/S adapter:
#
this = osAdapter portable
osAdapter = os/test/adapter
portable = support interface/command

Other than comments, an .equations file contains a set of unordered assignment statements where each assignment specifies a composition. In the example above, the left-hand side (the key) of each assignment defines a component to be the composition specified on the right-hand side (the value). The three components defined are this, osAdapter and portable. The keyword this designates the root assignment while the other assignments define sub-components to be combined into the root composition. Pictorially, the composition of this is hierarchical as shown below:

The image above shows a tree with a single root named this; three leaves named os/test/adapter, support and interface/command; and two internal nodes named osAdapter and portable. The evaluation of this is the composition of all leaves reachable from this, in order from left to right. The composer performs this composition with the following command: 

composer --equation=testVersion

The leaves are interpreted as URI names relative to the model path. As with basic equation files, the target name, testVersion is taken from the base name of the .equations file and the --target option can be used to override this default.

Composing .equations files. Additional power becomes available when .equations files are composed. Extending the above example, suppose there's another .equations file, win2kFeatures.equations, with the following content: 

osAdapter = os/windows/common os/win2k/adapter

Refining testVersion.equations with win2kFeatures.equations can be done with the following command: 

composer --target=win2kVersion.equations testVersion.equations win2kFeatures.equations

The result is a file, win2kVersion.equations, with content similar to that shown below: 

# Generated from [testVersion.equations, win2kFeatures.equations]
#
this = osAdapter portable
osAdapter = os/windows/common os/win2k/adapter
portable = support interface/command

Composition of .equations files is done by inheriting assignments where an assignment's signature is its key. In the above example, the last value assigned to key osAdapter was that defined in win2kFeatures.equations, so that is the value used in the result. The hierarchical structure of the resulting composition is shown in the following graph:

Another keyword, super, can be used to modify this type of composition. If an operand on the right-hand side of an assignment has the form super.key, then the previous value assigned to key to substituted at that point. Similarly, if super appears as an operand, then the previous value of this is substituted. As an example of the use of super, consider the files, one.equations and two.equations, shown below:

Files one.equations and two.equations
File: one.equations two.equations
Contents:
a = alpha
b = beta
g = gamma
this = a b g
a = aleph
b = super.a baffle
this = before super after

In file two.equations, there are two instances of super. The first is super.a in the value of b and the second is simply super in the value of t. Composing these two files yields the following set of assignments: 

a = aleph
b = alpha baffle
g = gamma
this = before a b g after

Again, it's left to the reader to develop more sophisticated applications!

Composer Configuration

There are two parts to configuring the composer. First, it's recommended that it be installed so that it has access to Ant and Java tools. Second, any non-standard properties, including file type preferences, should be placed into a composer properties file. This section describes these two layers of configuration.

Configuring composer for Ant and Java tools. Strictly speaking, it's not necessary to configure composer for Ant unless Ant is to be used in compositions. However, the standard distribution of the composer is defined to invoke Ant for top-level directory compositions as described previously. The invocation is done via the java.lang.Runtime.exec method which finds and executes Ant in a system-dependent fashion. In general, it is recommended that Ant be installed in a directory in a standard system directory. If this is not possible, the location of Ant's executable can be specified via the tool.ant property in a composer.properties file or on the composer command line.

Similarly, Ant may execute tasks that invoke Java tools such as javac and java. In order for this to be correctly done, the Java tools must be installed so that Ant can find and execute them. For recent versions of Java, Standard Edition, this is automatic except for Ant's external invocations of java, the Java virtual machine. Again, this is done via a java.lang.Runtime.exec method and it is recommended that java be installed in a system directory. If this is not possible, the location of a java executable can be specified in Ant build files with the jvm attribute of the Java task.

Configuring composer properties. The composer uses several properties to control the details of its operation. Most properties specify file types as described in the next section, but additional properties, as listed below, are used as well:

All of the above properties, as well as file type properties, are resolved by taking the first definition in the following list:

If a property is not found in the above list, it is undefined. The property files and resources referenced above, as well as embedded comments, must satisfy the format described in java.util.Properties.load. As an example, here are the default composer definitions for non-filetype properties: 

composer.directory.ignore : CVS
composer.file.ignore      : .*~|#.*#|%.*%|[.]#.*|core

tool.ant : ant

These are the only non-filetype defaults provided by the composer itself, though several other properties are defined by the Java platform. In particular, the default value for user.home is provided by the Java platform. File type properties are described below, along with their defaults as defined by the composer.

Supported File Types

Each unit processed by the composer is assigned a type as defined by properties in the properties search described in the previous section. In detail, the unit's type is found via the following multi-stage search through the properties:

The value found by the above search may be either a fully-qualified class name containing periods or an unqualified class name. An unqualified class name is converted to a fully-qualified name by prepending the package name composer.unit, which is a package distributed with the composer. In principle, a user could configure the composer to use file types other than those in composer.unit, but this is not currently recommended since the interfaces have not been finalized. However, the user is free to use any file type defined in composer.unit. Their descriptions follow:

Any one of the above file types may be assigned to a unit property as described in the beginning of this section. For example, if the user wants directories named with an extension .tmp to be ignored, it would be appropriate to assign file type FileUnit to property unit.directory.tmp. The standard distribution of the composer has the following set of default type definitions: 

unit.directory          : DirectoryCollective

unit.file               : FileUnit

unit.file.b             : BaliComposerFileUnit
unit.file.bak           : IgnoreFileUnit
unit.file.BAK           : IgnoreFileUnit
unit.file.drc           : DRCFileUnit
unit.file.equation      : EquationFileUnit
unit.file.equations     : EquationsFileUnit
unit.file.jak           : MixinFileUnit
unit.file.jar           : SingletonFileUnit
unit.file.java          : SingletonFileUnit
unit.file.prefix        : AdjoinFileUnit
unit.file.properties    : PropertiesFileUnit
unit.file.pyc           : IgnoreFileUnit
unit.file.pyo           : IgnoreFileUnit
unit.file.suffix        : ReverseAdjoinFileUnit
unit.file.swp           : IgnoreFileUnit
unit.file.timestamp     : IgnoreFileUnit
unit.file.txt           : LastFileUnit
unit.file.vm            : VelocityFileUnit

Any or all of the above definitions can be overridden as described in the configuration section.

Command-Line Invocation

The standard distribution of the composer includes an executable Jar file and a convenience script for bash shells under Linux or Cygwin. This section describes the command-line use of the convenience script.

In the standard distribution of the Jakarta Tool Suite (JTS), the composer script is created in the bin sub-directory of the JTS base directory. For ease of reference, the user may include this sub-directory in the executables search path for his or her system and, in the remainder of this section, this is assumed to be the case. Then, the basic form for the composer command is: 

composer [<option> ...] <source-file-or-directory> ...

The --target option (see below) and at least one source argument must be given on the command line, but there can be as many additional source arguments as desired up to system limits. Each source argument will be resolved as described in Supported File Types. In general , a multiple source types will be resolved to a common type by finding the least common ancestor of their individual types in an inheritance hierarchy of file types. The common type will be used to select a composition function as described earlier. Currently, however, the inheritance hierarchy has not been completely specified, so the wise user will ensure that all sources have the same type! It has been specified, however, that type FileUnit is the ancestor of all file and directory types while DirectoryCollective is an ancestor of all directory types.

In addition to the source arguments, there may be options on the command line as well. These options are of two kinds. There are Java options for the Java virtual machine and there are composer options for the composer itself. The composer script separates the two kinds of options, passing the Java options to the virtual machine and the composer options to the composer. Here's a summary of the available options:

Java Options:

-classpath <system-dependent-classpath>
This may be used to specify alternative locations for composer classes and the tools used by the composer. The default, which is to use the class path defined in the composer Jar file, is usually the recommended choice. The shortcut form -cp may also be used
-D<property>=<value>
This is one way to specify property definitions on the command line. As described in the Composer Configuration section, properties defined on the command line override all other property definitions.
-X<JVM-option>
Specifies a non-standard configuration option to the Java virtual machine. These are system-dependent options and the user should consult the Java documentation for his or her platform.

Some other Java options are available, but these are primarily for debugging and the casual user should avoid their use. The ultimate documentation for these is the composer script itself and the platform-specific documentation for the user's Java virtual machine.

composer Options:

--ant=<ant-executable>
This is an alternate method for defining the tool.ant property that specifies the location of the Ant executable.
--equation=<equation-file>
Specifies an equation file from which the target and source operands can be derived. The base name of the file is all that needs to be provided; the extension ".equation" will added. The file contents may include comments and character escape sequences as per properties files. After comment and escape processing, the source operands are listed in the file separated by whitespace and/or newlines. If no --target option is provided, the base name of the equation file is used as the target.
--help
This causes the composer to print a helpful message about the composer command-line parameters. Immediately after processing the --help option, the composer will exit without processing other command-line arguments.
--ignore=<pattern>
This specifies an overriding value for the composer.file.ignore property, thus allowing a short-hand way to specify which files to be ignored during recursive invocations of composer. For now, this option is not recommended since its definition is not finalized.
--layer=<base-layer-name>
This is an alternate method for defining the composer.layer.base property that specifies the base package name for .jak compositions. See the section on Composer Configuration for more details.
--logging=<logging-level>
This selects how much detail to report during execution. The logging level may be one of all (prints every logging message), fine (prints all progress messages plus some debugging messages), info (prints messages about files created and destroyed), warning (prints messages about risky but not fatal situations), severe (prints messages only about fatal errors) or off (prints no logging messages). There are other levels possible (see java.util.Logging.Level), but they aren't useful in the composer.
--model=<model-search-path>
This is an alternate method for defining the composer.model.path property that specifies the search path used to find model files. See the section on Composer Configuration for details.
--target=<target-file-or-directory>
A target is required by the composer and this option is one way to specify the destination of the composition. If this option is not present, but an equation file is specified via the --equation, then the composition target will be taken from the base name of the equation file. Regardless of how the target is determined, if the source operands are files the target will be a file and, if the operands are directories, the target will be a directory. If it already exists, its contents will be overwritten and, if it doesn't yet exist, it will be created.

Note that all Java options described above begin with one dash while the composer options described above begin with two dashes and follow POSIX conventions to specify values using an = symbol. Alternate forms are valid, but not recommended.

Programmatic Invocation

The current version of composer is intended as a stand-alone program started via the main method in class composer.Main. This is because composer is evolving conceptually and internal packages, classes and methods are subject to frequent change. Therefore, the recommended method for invoking the composer programmatically is as an external program via the java.lang.Runtime class.

ATS Home Page

Copyright © Software Systems Generator Research Group. All rights reserved.
Revised: April 17, 2003.

Jacob Sarvela
Last modified: Thu Jan 30 09:22:07 CST 2003