Major operating systems for wireless sensor networks (WSN) enforce an event-based programming paradigm for efficiency reasons. However, practice has shown that the resulting code complexity is hard to manage for software developers and leads to difficult problems during development, deployment, and operations. Thus, thread libraries for WSN applications have been introduced, but the efficiency constraints of the domain often lead to glaring restrictions of the supported thread semantics.

In contrast, compiler-assisted thread abstractions support threads without actually using threads at runtime. Instead, the thread-based code is automatically translated into equivalent event-based code. Currently, two different systems implement this idea, but both have severe limitations regarding both the thread semantics and the tool support.

Our goal is to demonstrate the potential of compiler-assisted thread abstractions by taking the next step and introducing a comprehensive system of compiler and debugger which supports cooperative threads with minor restrictions and which is also platform independent.

A Haskell-based compiler prototype shows the feasibility of our approach and preliminary results demonstrate that the generated code is efficient enough to be executed on wireless sensor network devices. Furthermore, existing optimization potential suggests that compiler-assisted thread abstractions can indeed outperform runtime-based solutions and compete with hand-written event-based code. We are currently working on completing the implementation of the tools and verifying these points by means of an extensive evaluation.

1. Introduction

Wireless sensor networks (WSN) aim to provide monitoring of physical phenomena over a spatial region without requiring existing infrastructure. To this end, battery-powered computing systems with sensors and wireless communication capabilities are deployed in the field and continuously deliver measured values via multi-hop routing to a base station.

Examples of existing WSN applications include monitoring of the permafrost in the Swiss Alps [Hasler2008], adaptive lighting in operational road tunnels [Ceriotti2011a], and wildlife monitoring [Dyo2010]. Such applications typically require a long system lifetime without manual maintenance such as renewal of batteries. As a consequence, energy efficiency is one of the most critical design criteria for WSN applications, which in practice implies resource-constrained devices. A typical example of such a device is the T-Mote Sky, which is equipped with a 16 bit, 8 MHz micro-controller, 48 kB of Flash memory and 10 kB of random access memory (RAM).

Major WSN operating systems such as TinyOS [Hill2000] and Contiki [Dunkels2004] account for the scarce resources by enforcing an event-based programming paradigm. The common reasoning behind this is that with event-based programming, a single dispatcher thread and a single stack suffice to execute multiple tasks concurrently.

Although efficient, practice has shown that the implications of this para\-digm often pose significant problems to developers, as it is difficult and cumbersome to manually manage the states and the control flow of event-based applications [Adya2002]. Furthermore, deployment environments typically differ strongly from lab environments and debugging is usually very time- and energy-consuming. Therefore, mistakes resulting from the inherent complexity of event-based applications tend to be very expensive to cope with.

To better support developers in programming WSN applications, investigations have been started to provide thread-based programming for WSN applications. On a first branch, common approaches to thread libraries have been adapted, sometimes by introducing restrictions in the supported thread semantics [McCartney2006, Nitta2006, Welsh2004]. On a second branch, compiler-assisted thread abstractions employ a compiler which takes a thread-based program as input and generates an equivalent event-based program as output. By these means software developers have the comfort of threads while the runtime still performs efficient event dispatching.

The goal of this thesis is to investigate the potential of compiler-assisted thread abstractions for resource-constrained systems. In particular, we aim at demonstrating that compiler-based solutions can not only support almost complete thread semantics to the programmer but also outperform runtime-based solutions in terms of code efficiency. To this end, we are building an accordant compiler called Ocram and will evaluate its performance characteristics on the basis of typical WSN applications. The final outcome of the thesis will thus consists in both a quantitative and a qualitative analysis of the potential and limits of compiler-assisted thread abstractions.

In the next section we will discuss two existing systems that already provide compiler-assisted thread abstractions for WSN applications. Being the first of their kind, though, both systems have sever limitations. Ocram is supposed to overcome these limitations as described in section 3. Section 4 subsequently presents published results, the current state of the work and future plans of it.

2. State of the Art

The first system that provided a compiler-assisted thread-abstraction for WSN applications is Protothreads [Dunkels2006]. It has been specifically designed for Contiki and its usage is deeply embedded in Contiki’s libraries and runtime system. Technically, Protothreads are a set of C preprocessor macros that enable the syntactical illusion of threads and blocking operations. Instead of waiting for the completion of a blocking operation, though, the runtime memorizes the current code location and returns from the handler function. When resuming the thread, the same function is called again and a switch statement, which is expanded from the Protothreads macros, brings the execution back to the location where the previous blocking operation occurred.

As the C preprocessor can only locally replace language tokens, this approach has a number of limitations. First of all, the state of automatic variables is not preserved across calls of blocking operations. Second, blocking operations can only be called in the topmost thread function, which severely restricts the software architecture of Protothreads-based applications. Last but not least, certain user mistakes are not detected at compile time but have to be indirectly examined by observing unexpected execution behavior. Such mistakes can be as subtle as using additional switch statements that interfere with the expanded ones or relying on the state of an automatic variable after a blocking operation.

In contrast, the first system that employs a dedicated compiler to provide thread abstractions was TinyVT [Sallai2008]. It has been specifically designed for the TinyOS operating system which is implemented in the nesC programming language [Hill2000] and enforces a component-based software architecture with event-based interfaces. TinyVT enables software developers to implement single components sequentially as it is possible to inline event handlers in code blocks following a special await statement. Also, the runtime preserves the state of local variables across such operations.

Although TinyVT overcomes many of Protothreads’ drawbacks, the supported thread semantics is still rather restricted. First of all, inlined event handlers may not contain await statements. Additionally, it is not possible to split the implementation of a component into multiple functions, which implies that code can not be shared between multiple threads, which in turn implies that TinyVT has no support for reentrant code.

Besides the above mentioned limitations, both systems provide no support for debugging. The consequence in practice is that software developers use source-level debuggers to step through the generated event-based code. We consider abstractions without debugging support as incomplete because they sometimes fail to hide lower-level details although their primary purpose is to relieve the user from having to deal with them. Even worse, software developers additionally need to understand the implementation details of the abstraction itself in order to perform the necessary back-mapping to identify the cause of the observed error in the abstract program.

In contrast to Protothreads and TinyVT, Ocram translates from and to standard-compliant C code, supports thread semantics comparable to what common thread libraries provide, and is platform independent with respect to the underlying event-driven runtime environment. Furthermore, we want to provide a corresponding source-level debugger that works on the level of the thread abstraction and offers a feature set that is comparable to those of common source-level debuggers. Finally, we expect that our findings are transferable to other domains where the event-based programming paradigm is prevalent. The next section presents the details of this system.

3. System Design

The Ocram compiler translates thread-based code (T-code) to event-based code (E-code) while both T-code and E-code comply with the ISO/IEC 9899 international standard (C99). Section 3.1 describes the semantics of the T-code and the limitations concerning the set of supported language features. Next, section 3.2 illustrates how Ocram translates T-code into E-code. The interoperability with the runtime environment that executes the E-code is discussed in section 3.3, which completes the description of the system basics. Subsequently, section 3.4 enumerates a set of optimizations opportunities that can be exploited to generate more efficient E-code. Finally, section 3.5 describes the design of the T-code source-level debugger and the requirements it poses on the compiler.

3.1 Thread Semantics and Limitations

On the T-code side, Ocram supports cooperative threads with shared memory. We assume that the underlying runtime environment defines in its application programming interface (API) the set of blocking functions together with non-blocking functions and required types. Every call to a blocking function is an implicit yield point where control could be passed to a different thread. In addition to that, the runtime environment can support explicit yield points by means of a blocking function that performs no operation.

Every blocking function is called a critical function and every function that calls a critical function, i.e. contains a critical call, is also a critical function. Every thread is started by the call of its thread start function, which usually is a critical function itself. Critical functions are reentrant and can thus be used by different threads. Furthermore, arbitrary non-critical functions can be used to perform additional computations. As threads are not preempted they can easily share memory and thus communicate with each other by means of global variables. In general, T-code is basically just C99 code, which has the additional benefit that existing development environments can be used to write it.

Since the thread semantics are implemented at compiler level, there are some inherent limitations, though [Bernauer2010b]. First, the number of threads must be known at compile time. Second, recursive functions must not invoke critical functions. And third, function pointers must not be used to invoke critical functions. We argue, though, that these limitations do not severely affect programming of WSN applications. In particular, recursive algorithms tend to have an undecidable stack consumption and thus should generally be avoided in embedded systems. Again due to the constrained resources of the employed devices, it is often not sensible to support an arbitrary number of threads. Finally, the use of function pointers is not uncommon, but can be avoided by a case differentiation with moderate overhead. In any case, the compiler can reliably detect any violation of these constraints.

3.2 Compilation Scheme

As a first step, the compiler must rewrite the control flow of the program to replace critical calls with non-blocking split-phase operations [Hill2000]. Additionally, it must rewrite the data flow to preserve the value of automatic variables. Amongst many possibilities, we are currently investigating the following compilation scheme as our preliminary results suggest that it produces efficient E-code in most cases.

For every critical function, the Ocram compiler generates a so-called T-stack structure that holds all automatic variables, all function parameters, the function’s return value (if existent), the continuation of the current caller, and an instance of the T-stack structure of each critical function that is directly called by the respective function. Furthermore, one instance of the T-stack structure of every thread start function is generated and used by the E-code to access the corresponding variables. We call such an instance a T-stack.

As the scope of a T-stack structure is the execution of the corresponding function and as every instance of a T-stack structure belongs to exactly one thread, all nested T-stacks can share memory and are thus grouped in a C union. In fact, a T-stack represents the overlay of all snapshots of the runtime stack of the corresponding thread if it would actually be executed.

Concerning the control flow, a critical function must only trigger the respective operation and return immediately. The platform abstraction layer, which is covered in section 3.3, implements the blocking functions accordingly. For the other critical functions, the compiler replaces every critical call with the following sequence of statements: First, the callees parameters are written to the T-stack. Second, the continuation information for the callee is written to the T-stack. Third, the actual call is performed. Last, the function returns.

The bodies of all critical functions that are used by a thread are all inlined into one common thread execution function for each thread. Thus, the continuation can simply be the address of a label and a call of a critical, but not blocking, function is just a goto – Computed gotos are a GNU extension to the C programming language. If it is important to stay standard compliant they can be emulated via less efficient jump tables. – The first statement of every inlined body, as well as every statement after a critical call, is equipped with a unique label. Furthermore, every return statement in the T-code is replaced by a goto statement that uses the continuation information stored on the T-stack.

As there is a one-to-one mapping between thread execution functions and T-stacks, read and write access to the T-stack involves no indirections. The C compiler that processes the E-code can thus generate efficient memory access code despite the nested structure of the T-stacks.

3.3 Platform Abstraction Layer

The above described compilation scheme depends on proper event-based implementations of the blocking functions that are used by an application. Their role is to implement the systematic interface assumed by the code generation by means of the interface of the respective underlying runtime environment. This manual effort is necessary because existing runtime environments have non-systematic interfaces so that the code generation needs some sort of user assistance to be able to use them. Amongst many possible ways of formalizing those interfaces we think that simply implementing them once is justifiable and feasible.

Clearly, this platform abstraction layer (PAL) introduces an overhead in terms of code size, RAM usage and CPU cycles which we have to account for when evaluating the overall system performance. The size of this overhead inherently depends on the semantics of the underlying interfaces. Given a runtime system that actually support Ocram natively this overhead could even disappear.

As the Ocram compiler poses little requirements on the PAL, it is in general not restricted to the WSN domain. In any other domain, where the event-based paradigm is prevalent for efficiency reasons and C99 is used, software developers could benefit from our compiler-assisted thread abstraction.

3.4 Optimizations

The compilation scheme of section 3.2 supports the most generic case that can be encountered. In practice, though, many typical recurring patterns exist and allow for shortcuts that can be exploited by a compiler.

First of all, automatic variables that are never written to before a critical call and read afterwards do not need to be preserved, so they do not have to be saved on the T-stack. Instead, they can stay automatic variables in the E-code as well and, thus, be saved on the single runtime stack which saves memory and access time.

As a second example, if a function is only called from one location in the whole program there is no need to save the continuation in memory. Instead, it can be hard-coded into the E-code thus saving memory and lookup time. The callee can additionally access the variables of the caller in case of read-only parameters.

We believe that further investigations will reveal additional potential for optimizations that all sum up to a significant performance benefit of compiler-assisted thread abstractions compared to runtime-only solutions.

3.5 Debugging Support

The role of the debugger is to undo the compilation so that the E-code runtime can be presented by means of the T-code abstraction. In general this involves managing the mapping of both the data and the control flow. To this end, the T-code debugger utilizes and controls an E-code debugger which is just a standard C debugger, possibly attached to the execution via JTAG (IEEE 1149.1) or other platform dependent means.

When the user wants to know the value of a T-code variable the T-code debugger only needs to know which E-code variable holds this value and request it from the E-code debugger. Similarly, when the user installs breakpoints or issues step-through commands, the T-code debugger needs to know to which location in the E-code this corresponds to and issue the proper commands to the E-code debugger.

To enable the debugger to perform these tasks, the Ocram compiler must generate corresponding debugging information for all variables and all statements of critical functions. This relatively simple technique is enough to add significant value to the thread abstraction provided by our Ocram compiler as discussed in section 2.

4. Past, Current, and Future Work

In Bernauer2010 and Bernauer2010b we have presented a first compilation scheme that differs from the one described in section \ref{sec:scheme}. Since then we have been working both on finding better compilation schemes and implementing the Ocram compiler.

For the latter, we eventually have decided to use the Haskell programming language, for which the Language.C module provides a comprehensive parser and pretty-printer for the C programming language. The current version of the Ocram compiler is a proof-of-concept implementation. Although it supports the translation of basic applications, many C language features are not supported yet. Furthermore, we have implemented a basic PAL for both Contiki and TinyOS and successfully executed generated E-code on a T-Mote Sky device.

As a next step, we plan to implement the T-code debugger and to extend the feature set of Ocram and exploit possible optimizations to achieve higher efficiency. Our goal is to outperform existing threading libraries in an evaluation involving realistic WSN applications.
Finally, past work has shown that the efficiency of the compilation scheme highly depends on application properties such as the number of reentrant functions. On the long term we thus envision a system that measures relevant application properties and chooses the best compilation scheme accordingly, possibly involving user interaction to achieve a proper trade off.

Meta

This is a research proposal for my Ph.D. studies. (pdf)

Example

So, here is an example DSL for tournaments and the list of the techniques that I encountered and consider essential for building internal DSLs in Scala. I don’t claim – and I don’t think – that this list is complete, though.

import scala.collection.mutable.ListBuffer

class Setup extends TournamentBuilder {
    "3:2" ==> {
      --> ("butcher" vs "hulk")
      --> ("chaos" vs "crusher")
      --> ("diabola" vs "electra")
    }
}

object Main {
  def main(args: Array[String]) {
    new Setup()
  }
}

class Tournament(val bestof: Int, val winners: Int) {
  def ==> (block: => TournamentBuilder) : Unit =
  {
    val config = block.list
    run(config)
  }

  def run(config: ListBuffer[(String, String)])
  {
    for (game <- config) {
      Console.print("running best of " + bestof +
	" in " + game._1 + " vs. " + game._2 + "\n")
    }
    Console.print("The best " + winners +
	" players now proceed to the next round\n")
  }
}

class MatchBuilder(p1: String) {
  def vs(p2: String) = (p1, p2)
}

class TournamentBuilder {
  implicit def strToTourn(mode: String): Tournament =
  {
    val parts = mode.split(":")
    new Tournament(parts(0).toInt, parts(1).toInt)
  } 

  implicit def strToMatchB(p1: String): MatchBuilder =
    new MatchBuilder(p1)

  var list = new ListBuffer[(String, String)]()

  def -->(config: (String, String)): TournamentBuilder =
  {
    list += config
    this
  }
}

technique 1: method names

Scala allows to use almost any sequence of Unicode characters as method names. So, especially the ASCII artists can use their creativity and the others can check out the Unicode tables for arrows, mathematical operators, or any other proper symbols.

Be aware though, that the usage of the DSL becomes cumbersome, though, because Unicode symbols are usually not bound directly to some keys on the keyboard. (@Vim users: check out this howto). So, for my example I sticked to good old ASCII. And I wasn’t very creative either. But I can assure you that ==> and --> have special semantics for tournament planners

technique 2: infix notation

In Scala, every method with a single parameter, i.e. any binary function, can be written in infix notation without parantheses. This increases the readability of the DSL because you can write "butcher" vs "hulk" instead of "butcher".vs("hulk") – which doesn’t work yet because vs is not a method of String; see the next technique.

technique 3: implicit convertion

Scala’s implicit feature is really powerfull and widely used behind the courtains. In the above example, the strToMatchB method is implicitly called whenever there is a String object but a MatchBuilder object is expected. So, "butcher" vs "hulk" becomes strToMatchB("butcher").vs("hulk"). The trigger for this implicit method call is the method vs which is only defined for MatchBuilder objects.

technique 4: code blocks

Code blocks, a.k.a. closures, are first-order citizens in the Scala world. The curly braces allow to denote literals and the type system provides types for them which are essentially simple function types. In the above example, the code block expects no parameters but results in a TournamentBuilder object. With this technique, you can provide grouping and scoping for your DSL.

technique 5: string processing

In the above example, "3:2" is supposed to be the tournament mode which reads as “every match is best of 3 and the best two players enter the next round”. This is just an example to show that besides all DSL magic you can still use plain old strings and parse them. Either with simple string utility methods, with regular expressions or with parser combinators. Depending on the requirements of the DSL using small sub-grammars can greatly increase the readability of the language.

Execution

This program’s output is:

running best of 3 in butcher vs. hulk
running best of 3 in chaos vs. crusher
running best of 3 in diabola vs. electra
The best 2 players now proceed to the next round

First, the string "3:2" is converted into a TournamentBuilder by strToTourn. Next, the method ==> is invoked which executes the passed code block. There, a chain of --> method calls is executed while each incarnation updates the member list and returns the same TournamentBuilder object. To this end, the --> method expects a tuple which is constructed by means of the MatchBuilder class as explained above. Finally, the Tournament simply takes the MatchBuilder‘s list as its configuration for the holding of the tournament.

Meta

I want to thank fire for being my sparring partner on programming languages in general and Scala in particular. Go and check out his blog for more good stuff about Scala.

1. Introduction

Business processes drive the manufacturing of products and the operation of services in today’s economy. They describe the interactions between humans, IT-systems, and the environment that are necessary to reach the business goals. With increasingly evolving sensor, wireless and distributed systems technologies, more and more business processes use wireless sensor networks (WSN) to gather information about or interact with the entities they control. Examples include safety processes for storing hazardous materials [1], securing shipping and handling of containers to comply with government regulations [2], and automating hospital processes [3].

Business processes are designed and modeled by business analysts who are domain experts in their field but do not necessarily have a background in IT. Thus, the actual implementation of a business process is typically performed by software developers. This prevalent approach poses two major problems. Firstly, the implementation usually deviates from the intentions and expectations of the business analysts because cross-domain communication is prone to misunderstandings. This deviation typically renders the respective business process less effective the larger it becomes. In order to avoid this, technology implementations should follow the business process and not vice versa [4]. Secondly, changing the model of a business process necessitates the adaption or even recreation of parts of the implementation. As this is expensive and time-consuming [5], business processes tend to adapt slowly to changes of requirements or business facts.

In this paper, we propose to use the standardized Business Process Model and Notation (BPMN2) as the interface language between business analysts and software developers and to apply code generation to turn BPMN models into executable code. BPMN models are refined by additional models while commonly programmed components form the base case of this recursion. These components and the code which is generated from the models constitute the final application. Software developers can decide which levels of abstraction should be modeled and which should be programmed. At the topmost levels, business analysts, which represent our target user group, model the business process in the familiar BPMN while software developers relieve them from mastering distributed computing problems by implementing all models and components below.

In our approach, instead of being just a documentation blueprint, a BPMN model actually defines the execution of the respective business process and can be simulated and debugged directly in the model editor. Additionally, BPMN supports the integration of multiple involved distributed systems into a single top-level model, thus covering the demands of business processes which use WSN. To enable better integration into the environment, reduce power consumption, and decrease production and deployments costs [6], we focus on the lower-end spectrum of embedded devices such as 8-bit MCUs with 128 kB of non-volatile memory, and 8 kB of volatile memory. To this end, our compiler generates efficient code which runs on such resource-constrained sensor nodes.

The quantification and evaluation of modeling and programming abstractions is notoriously difficult, but we make a series of arguments to support our approach. Firstly, BPMN is an established standard and business analysts already use it to model business processes. Hence, it is an adequate tool for such tasks and the addressed user group is capable of using it. Next, we show that attaching execution semantics to BPMN models is possible. We do this both in theory and in practice. Firstly, by discussing how the code generation works and secondly, by presenting results of a series of case studies for which we used a proof of concept implementation of the required tools. Additionally, we demonstrate that the generated code is efficient enough to be executed on resource-constrained sensor nodes. Last, we argue that simulating and debugging the models of business processes is an intuitive task for business analysts because this functionality is integrated into the model editor. In summary, we enable business analysts to control, simulate, and debug the execution of their business processes on an abstract level.

This paper is organized as follows: Section 2 presents related work and argues how our approach advances the state of the art. Next, Section 3 gives a brief introduction into the BPMN notation and semantics using an example business process. We then outline how BPMN models can capture WSN requirements in Section 4, and show how WSN applications can be generated from BPMN models in Section 5. Our modeling approach requires an integrated tool-set which we describe in Section 6 while in Section 7 we stress the debugging aspect. Subsequently, we evaluate our proof of concept implementation of the tools in Section 8 using a representative set of WSN applications before concluding in Section 9.

2. Related Work

Probably the best-known modeling language is the Universal Modeling Language (UML) which is a generic set of standard graphical notations that capture different system aspects. Rhapsody is an industry solution for UML-driven development and automatic code generation for embedded systems. Recent work in this area even addresses WSN [7] using Sun SPOTs with a 32-bit ARM core. In comparison, our solution supports sensor nodes with significantly less memory and computation resources. Additionally, for the description of a single business process, UML requires multiple models which cover different aspects and different involved entities of the business process. This reflects the fact that UML is a rather technical notation which does not fit the needs of business analysts.

Other proposed solutions for graphical programming and model-based development of sensor network applications [8][9] are based on the LabVIEW and Simulink proprietary standards. Additionally, Viptos, Flow, and Srijan are modeling environments for wireless sensor networks with custom graphical notations based on data flows. In either case, the intended audience again are scientists, system engineers, and software developers rather than business analysts.

To address the needs of the business domain, the Business Process Model and Notation (BPMN) became an Object Management Group (OMG) standard for business processes. Today, this language is widely used by business analysts, consultants, and executives as a lingua franca. BPMN is additionally suitable for transformation to execution languages such as the Business Process Execution Language (BPEL) where a BPEL engine executes models while communicating with the environment via Web Services. Existing approaches which integrate sensor networks into business processes focus on this approach [10][11][12]. For the resource-constrained sensor nodes which we want to support, a model interpreter like a BPEL engine is not a viable option. Instead, we use a compiler to generate efficient code which executes directly on the targeted sensor node platform. Furthermore, with our approach, different communication channels can use different protocols such as IEEE 802.15.4 or TCP/IP depending on the respective requirements.

In summary, none of the existing solutions bridges the gap between the model description provided by business analysts and the corresponding implementation of a business processes for the resource-constrained sensor networks we are targeting. On the one hand, although some of the modeling solutions may accommodate the resource requirements, they expect the business audience to learn and understand a technical language and the associated tools. On the other hand, solutions targeted particularly at the business domain require too many device resources. We combine the benefits of existing work and use an open industry standard for modeling business processes. Our approach provides a unified view over all involved systems, supports simulation and debugging of business processes directly in the model view, and generates efficient code.

3. Business Process Example and Background

This section introduces the reader to a subset of BPMN terms and concepts while we refer to the standard for an exhaustive description. Generally, a BPMN model describes a process with the interactions and behaviors of involved entities which cooperate in order to fulfill required business goals. In BPMN terminology, a pool represents such an involved entity and acts as the topmost container for its model. Its graphical representation is a rectangle which can be collapsed to visually compress a model.

Figure 1 shows an example BPMN process for parcel delivery with three pools: Headquarters, Truck, and Parcel. Only the Headquarters pool is expanded and shows the details about enclosed tasks, control flows, message flows, and data flows. This is an example of a classical BPMN model as there is no code generation attached but the tasks are performed manually instead.

The actual work performed by a pool is divided into tasks which are shown using boxes with round corners. A task may be recursively refined to contain other models in which case a plus-sign annotation is placed in the lower part of the box. Otherwise, the task is a basic task which is implemented directly in the language of the target platform.

Changes in the control flow are described using a gateway rhombus with an enclosed symbol which defines the semantics. A parallel gateway has a plus sign and forks the control flow if there are multiple outgoing sequences and joins the control flow if there are multiple incoming sequences. An if gateway has an X-symbol and represents a conditional decision point where the process follows the respective sequence flow that matches the condition.

During execution, a process may wait for a certain event to occur (catch) or it may trigger (throw) events to notify different tasks or other pools. Examples of possible events include starting and ending a task, receiving or sending a message, and signaling exceptional conditions.

The BPMN representation for events is a circle. To visually distinguish between the different event types and their properties, BPMN uses icons and different styles for the circle line. Catch events have a hollow icon, while throw events have a filled icon. Furthermore, start events have a thin border and end events have a thick border. A catch event that does not terminate the respective process has a dashed line, e.g. quality breach in Figure 1. In contrast, other catch events such as damaged in Figure 2 do terminate the respective process and are depicted with continuous lines.

The data objects required by a process to fulfill its work can be explicitly represented in BPMN using data items (papers folded on the upper-right corner) and data stores (database cylinders). Both are connected to readers and writer by means of association flow arrows which are depicted as a dotted arrow. The difference between data items and data stores is that the former are volatile while the latter are persistent.

Communication between pools is explicitly modeled using communication flows which are represented as dotted lines with a hollow arrow head on the receiver side and an empty circle at the sender side. An envelope on the communication flow arrow marks the actual definition of the communication and specifies its properties.

Figure 2 shows the model of the Parcel pool. This process is automatically executed on the WSN nodes and demonstrates one of the major contributions of our work: The business analysts define reactions to business events on an abstract level and hereby control the actual execution of the business process because their model will be used to generate the corresponding application that is executed on the sensor node.

4. BPMN Models for WSN Applications

Models for WSN applications should support the distributed, heterogeneous and reactive characteristics of this domain. For energy efficiency considerations, models should also allow control over the protocols used for communication. Additionally, since a WSN is usually connected with other systems, the modeling language should support such integration.

The syntax and semantics of BPMN need neither be extended nor restricted to fulfill these requirements. The reactive nature of WSN applications is captured by BPMN events and the processing of events by BPMN sequence flows. Furthermore, communication only takes place via messages between pools and pools can have multiple instances. Thus, different entities of a distributed system such as various kinds of WSN nodes can be modelled with different pools.

The visual integration of the different entities is provided by the fact that pools can be collapsed and integrated into a system-wide model which shows only the general communication flow between the pools. Moreover, a definition of a communication flow has an envelope component attached to it and the properties of this component defines the details of the employed communication protocol. Thus different transmission modes such as broadcast and unicast, and different communication protocols such as IEEE 802.15.4 and TCP/IP can be chosen for each communication flow separately.

By this means, manually performed and automatically executed business processes can also be integrated into a system-wide model. To this end, base stations could send emails to an operator who sends back commands via HTTP requests, for instance. The communication between the operator and the WSN is simply represented with properly annotated communication flows in a BPMN model and the code generation takes care that the communication routines which are provided by the software developers are invoked.

In BPMN, each pool contains a task which uses sub-tasks. Such a sub-task can be refined by an additional model or can be a basic task which maps to an API function of the target platform. In the end, every sequence flow results in a sequence of basic tasks being called. The set of all basic tasks is part of the platform API which the software developers implement once and reuse for all business processes which involve the given target platform.

While graphical modeling provides a better overview over a process and is more intuitive than programming, it is not a suitable tool for all jobs. In particular, software developers generally prefer writing code when dealing with low-level implementations. Due to the recursive nature of BPMN models, our approach allows to choose the border between modeling and programming at arbitrary levels of abstraction. Hence, the building blocks for the models which are created by the business analysts can contain both basic tasks and tasks which are modeled. As a rule of thumb, general sequence flows of processes should be modeled while algorithms and device drivers should be programmed.

5. Code Generation

The core of our approach is a compiler which translates models into executables. In this section, we describe the code generation part of this translation which takes a sound BPMN model as input and yields an intermediate representation which is further processed by later steps. A sound model does not have a lack of synchronization and is deadlock-free as defined in [13]. This basically means that it is not possible that two sequence flows execute the same part of the model at the same time, and that there is always at least one sequence flow executing.

The translation of association flows is straightforward. Data items become variables and access to data stores is mapped to calls of persistence functions provided by the platform API. This translation is correct as long as no task is ever invoked more than once at the same time. We call this the static mode because it provides a one-to-one mapping between tasks and its enclosed data objects.

In contrast, in the dynamic mode there are parallel tasks which are invoked more than once at the same time. Then each invocation of such a task needs its own set of data objects. Thus, in the generated code, the corresponding variables are dynamically instantiated with each invocation and access to them is performed indirectly.

It is undecidable in general if a task is parallel or not. However, it is possible to identify most non-parallel tasks by a simple sequence flow analysis. For the undecidable cases the dynamic mode is always safe but it might be needlessly inefficient. In these cases the user might be able to assist by annotating tasks as non-parallel which is a property that can be checked at runtime by the generated code.

In contrast to the associations flow, the translation of the sequence flow is more complex. To support standard sensor nodes we have to address the fact that major WSN platforms are event-driven due to the efficiency benefits of this paradigm (Contiki, TinyOS, Mote Runner). Thus, the compiler has to translate the sequential and parallel computation semantics of a BPMN model into equivalent event-based code.

Event-based platforms typically use a single dispatcher which constantly takes the oldest event out of an event queue and calls the corresponding event handler. Event handlers may not block when waiting for something. Instead they only trigger a corresponding split phase operation and delegate the remaining processing to a callback which is a second handler that is registered at the runtime environment.

By following this pattern the compiler maps each sequence flow to a chain of event handlers which are causally related to each other by the respective events. As long as a split-phase operation has not yet completed, the dispatcher calls handlers which correspond to other sequence flows, thus processing multiple sequence flows virtually in parallel.

We call a basic task with a split-phase interface a critical task. Additionally, all tasks which enclose critical tasks are also called critical. Outgoing sequence flows of critical tasks, catch events, start events, receive events, and outgoing sequence flows of parallel gateways represent all possible start or resume points of sequence flows at runtime. Thus, for each instance of these entities a function is generated which executes the corresponding sequence flow.

If a sequence flow reaches a task, the generated code simply calls the function corresponding to the start event of the invoked task. Furthermore, for every possible continuation – i.e. catch events at the border of the task or outgoing sequence flows in the case of a critical task – the pointers of the corresponding functions are saved in extra continuation variables. These variables are managed the same way as data item variables are.

If a sequence flow reaches a throw event, the generated code simply consults the corresponding continuation variable and calls the continuation. If a sequence flow reaches an end event, the generated code simply returns from the function in the case of a non-critical task. Otherwise it consults the continuation variable for the outgoing sequence flow and calls the continuation. Basic critical tasks are manually implemented along these lines, so that the computation can continue after completion of a split-phase operation.

Terminate events require more effort as the BPMN semantics require that all active sequence flows of the enclosing task are canceled. As sequence flows always continue execution until they reach a critical basic task, canceling them simply requires to suppress the invocation of the callbacks. Thus, for every critical task a terminate function is generated which recursively invokes the terminate functions of the enclosed sub-tasks. In the case of critical basic tasks, the terminate function is manually implemented and suppresses the invocation of the callback in a platform-specific way.

The transformation of if gateways simply generates a corresponding if structure. Concerning parallel gateways, the translation exploits the fact that in a sound model there is always a one-to-one mapping between fork and join gateways. Note that for this to be true one has to consider an implicit join gateway in front of all end events. Furthermore, every throw event is implicitly followed by an end event. These model transformations are performed automatically and do not alter the model semantics.

Each pair of join and fork gateways share a common counter variable for the number of active sequence flows between the fork and the join. So, the translation of the fork gateway is code which first increases the counter by the number of outgoing sequence flows and then calls the respective functions of the outgoing flows in turn. Accordingly, a join gateway results in code which decrements the counter by one. If and only if the counter equals zero the function that covers the single outgoing sequence flow is called.

Listing 1 shows the transformation of the model in Figure 3 in static mode. The tasks off and toggle are non-critical sub-tasks because they only invoke the non-critical basic tasks led_setStatus and led_getStatus (line 20, 24 and 26; not shown in the model). In contrast, sleep is a critical basic task. Thus the continuation blink_after_sleep is stored in the continuation variable continue_sleep before the task is invoked (line 7 and 14). Furthermore, the timeout data item is associated with the sleep task, so that its value is passed to the sub-task (line 8 and 15).

6. Tools

The realization of our approach requires a set of integrated tools: an editor, a compiler and a simulator.

The compiler transforms a model into an executable in a three-phase translation. In the first phase, the model is checked for consistency and soundness. Next, the compiler performs a pattern-based platform neutral translation as described in Section 5. The third phase then is platform specific and translates the intermediate representation from phase two into suitable code for the target platform. For each entity, the compiler generates one binary using a back-end for the respective target platform. These binaries can be installed on real hardware or executed in the simulator.

There are several constraints on the model which should be checked by the compiler in phase one. For example, unconnected flows, unused data or undefined tasks are not allowed, and all thrown events must also be caught. Furthermore, a type checker should ensure that data which is associated with a basic task has a type which is suitable to what the task expects. Finally, software developers should be able to implement further checks in order to ensure the correct usage of the building blocks they provide. For example, certain basic tasks might only be allowed to be called once or it might be compulsive that other basic tasks are invoked in a certain order.

As mentioned previously, phase three is platform specific. As the concepts of the intermediate representation are very basic, we expect that it is possible to implement them on different target platforms. The standard compiler tool chain of this platform finally processes the output and links it against the implementation of the platform API and other required platform libraries in order to produce the final binaries.

Compilation and simulation are triggered by the business analysts who use the editor as their primary tool. It allows them to create, browse, and manipulate business process models. Furthermore, it offers means to manage, create, and import model libraries. What distinguishes this editor from all other BPMN editors, though, is the seamless integration of the compiler and the simulator. The editor controls these tools by sending commands and receiving resulting information which are visualized in the model. For example, in case of a constraint violation the compiler propagates the information about the constraint and the violating BPMN component back to the editor. And in case of compilation or runtime errors, the compiler or the simulator can infer the causing BPMN component (c.f. Section 7) so that the editor can then highlight this component in the model.

In the overall development cycle, there are different kinds of possible failures. Constraint violations indicate errors in the model which the business analysts have to correct. In contrast, compilation and runtime errors are the responsibility of the software developers. Either by changing the implementation of the platform API or by adding additional constraints, they have to ensure that a model which passes all constraint tests does not cause such failures. This is important because business analysts typically cannot and should not handle compiler error messages and stack traces. Last, runtime exceptions can occur as part of a valid process and are represented as events in BPMN models where they can be caught and handled. In general, we recommend to avoid the usage of exceptional events in higher level models because they introduce a non-sequential control flow which is sometimes difficult to follow for people with little background in IT.

Section 8 presents a proof of concept implementation of the above described tools. Before that, the following section elaborates on the debugging support, which is one of the major contributions of our work.

7. Debugging

A major benefit of a modeling or programming abstraction is the concealment of lower-level details. Without being forced to understand and master all those details, a user of the abstraction can create and maintain applications faster and more intuitively. This benefit is undermined if the application contains an error and there is no debugging support on the application’s level of abstraction. In such a case, the user has to infer the error in the application code from the details of the generated code. Many WSN programming abstractions suffer from the lack of debugging support [14], which we believe hinders their adoption in practice.

Consequently, we want to support debugging on the abstraction level of the BPMN models. To this end the compiler generates a symbol table during phase two. This symbol table maps all components of the input model to locations in the generated code. This information can be used by the tools to retrace the translation between the model and the executable in both directions.

Concerning the tools, debugging support means that everything should be integrated into the editor which is the primary tool for the business analysts. They should be able to set breakpoints at arbitrary points of sequence flows either for a single instance or for all instances of an entity. If such a breakpoint is hit during the runtime of the binaries, for each instance the current progress of all sequence flows and the current value of all data objects should be visualized in the editor. Furthermore, it should be possible to alter the value of data objects and to inject events so that different sequence flow paths can be tested.

A simulator which supports these features must be able to simultaneously execute all instances of all entities. If one instance hits a breakpoint, the execution of all instances has to be stopped so that the editor can query the progress of sequence flows and the value of data objects. To this end, the symbol table helps to map between graphical components and points in the generated executables.

Simulating and testing business processes under various possible conditions gives confidence to the business analysts that the created business process will work as intended once it is deployed. For that, it is crucial that the simulation environment matches the real environment of the business process as close as possible. Furthermore, the models and the basic tasks should not contain non-deterministic behavior because this cannot be reliably tested with the described simulation setup. We suggest that the software developers ensure this by creating models and basic tasks accordingly and by adding constraints to the compiler.

8. Evaluation

To evaluate our model-driven methodology, we implemented the required tools as described in Section 6. Our proof of concept implementation supports all described requirements except for changing data object values in a stopped simulation. Furthermore, injecting external events is not integrated into the model editor but has to be done by means of command line tools.

Figure 4 shows our model editor which is a web-based application based on Oryx, a BPMN 2.0 editor distributed under the MIT license. We extended the functionality of the Oryx editor to include plugins that integrate the compiler and the simulator into the editor. The communication interface between the web browser, our compiler toolchain, and the sensor network simulation framework uses a RESTful API with data encapsulated in JSON messages.

We use the Mote Runner platform as WSN development environment. The platform hides the intricate details of the underlying hardware by using an efficient virtual machine and micro-manages power on behalf of applications. Applications are programmed using high-level languages which are further transformed to optimized byte-codes. These features relieve our compiler from hardware-specific details so that it can generate portable high-level code. Mote Runner additionally provides a simulation environment with a source-level debugger interface which our integrated editor exploits to simulate and debug the same application byte-codes that are executed on real sensor nodes.

The compiler plugin calls the compiler tool which generates C# source code for the Mote Runner platform. The simulator plugin creates the corresponding sensor network configuration for the simulator according to the pools and their respective cardinality. With the aid of the symbol table from the compiler, errors and information from all tools are processed and reported back to the modeler in the browser with warning or error indicators overlaid on the graphical symbols.

To show the generality and expressiveness of the approach, we used our editor to model several applications corresponding to several WSN archetypes [15]. Firstly, we refined the parcel monitoring scenario described in Section 3 using a single-hop star topology with a Truck and several Parcels. Secondly, we modeled a multi-hop tree-based TDMA protocol where the Master root sends beacons which are re-broadcasted in assigned slots by Slave nodes. This shows that we can even express low-level communication and synchronization details. In addition, we modeled Gather which is a high-level data collection application using a decentralized multi-hop library for the communication. Last, we modeled the base case Blink which toggles an LED every 500 ms to clearly expose the systematic overhead of our approach.

To quantify the efficiency of our approach we measure the usage of resources of the generated code against the hand-written equivalents. To this end, we compiled each application once in static and once in dynamic mode with all tasks treated as parallel tasks (c.f. Section 5). This is only possible because in none of our examples we encountered the need of this feature. Nevertheless, we think it is interesting to know the costs of supporting all BPMN features.

The total amount of consumed energy is an important efficiency measurement because the life time of battery-powered sensor nodes depends directly on it. The maximum stack size and the maximum heap size show how efficiently RAM resources are used while the size of the binary does the same for the flash resources. Finally, the code size is a common indicator for code complexity. All measurements were obtained using the simulation environment running for 30 seconds which is enough time to measure and explore both the normal sequence flows as well as the exceptional code paths.

Once a process behaved in the simulation as expected, we used Iris sensor nodes which have an 8-bit micro-controller, 128 kB of flash and 8 kB of RAM for a test deployment. We did not measure the resource consumption on the real hardware because the Mote Runner simulator provides accurate measurements by design. Instead, we thereby only verified that the generated code is efficient enough to be executed on such a resource-constrained device.

Figure 5 shows the results of the evaluation. First, we see that the energy consumption overhead for each compilation mode is only 1%. This is because WSN applications are reactive by nature, where sleeping periods dominate computations.

Second, the overhead for the RAM usage averages at 10% in the static case and 50% in the dynamic mode. In contrast, the size of the binary and the code show that the systematic transformation of the model results in twice more code than a software developer needs for the same task. On average, the overhead for the flash consumption is 44% for the static mode and 3 times more space for the dynamic mode.

In practice, the energy consumption is the most important efficiency characteristic of an application as long as the employed mote provides enough resources to host it. As we could show that the generated code fits on a typical mote and the additional energy consumption is very low, we argue that the benefits of the code generation outweigh the introduced overhead. This is especially true for the static mode for which the overhead is moderate given that flash space is usually not the major limiting resource on motes.

Concerning the dynamic mode, it is currently not clear if parallel tasks are used rather often or rather seldom when modelling WSN applications. All we can say is that the total overhead of the compiled code will fall between the two measured extremes depending on how often parallel tasks are used. Additionally, our proof of concept implementation does not exploit possible optimizations which can reduce the overhead of the translation.

9. Summary and Conclusion

In this paper we focus on business processes using wireless sensor networks. Our main contribution is enabling business analysts to control the execution of such a process on the abstract level of its BPMN model. Furthermore, we enable them to test and debug their models on the same level of abstraction. By this means, we close the gap between the specification and the actual implementation of business processes.

We achieve this by introducing a compiler which translates the sequential execution semantics of BPMN models into event-driven code suitable for common sensor network platforms. Software developers provide a set of building blocks which can be reused by different models. These building blocks can be either directly programmed or also modelled in BPMN depending on the desired level of abstraction.

We additionally elaborated on the set of required tools and presented a proof of concept implementation thereof. Our evaluation results on several case studies show that the generated code is only 1% less energy efficient than hand-written equivalents. Concerning RAM and flash usage, though, the overhead is higher. Nevertheless, we could show that the generated applications fit on standard motes.

In summary, we argue that the introduced overhead is worth the benefits of keeping business processes synchronized with their implementations. We argue that the parts of a business process which involve WSN can and should be automatically generated directly from the model specification. Thus, organizations can dynamically adapt their technology implementation according to their business processes and not vice versa.

Meta

• This is a paper published at the 7th IEEE International Conference on Distributed Computing in Sensor Systems (DCOSS 2011).
• This is joint work with Alexandru Caracas from IBM Research, Zürich.
• This work has been partially supported by the National Competence Center in Research on Mobile Information and Communication Systems (NCCR-MICS), a center supported by the Swiss National Science Foundation under grant number 5005-67322, and by CONET, the Cooperating Objects Network of Excellence, funded by the European Commission under FP7 with contract number FP7-2007-2-224053.
• This work has been supported by Thorsten Kramp from IBM Research, Zürich and the Mote Runner team.
• .pdf, BibTex

Language Oriented Programming

First of all, in the context of language workbenches, differentiating between programs and models becomes cumbersome and adds no benefit to solving a given problem. That’s why some people refer to a program or a model as a “mogram”, but I prefer the general term document because in fact it doesn’t really matter if the generated output has execution semantics or not. So, a document is an instance of a language which is a set of meta-classes, so-called concepts, which can contain other concepts or refer to them. An instance of a concept is called a node, so a document is a tree of nodes which refer to each other.

Given this terminology, the basic idea behind MPS is to create, view and manipulate document trees instead of text files [Dmitriev]. The major advantage of this approach is that no parsing is necessary, thus the design and the composition of languages is not constrained by technical necessities such as the unambiguity of the grammar or problems with left recursion and so on. In fact, I see a lot of analogies to what makes XML extensible and why Lisp is such a flexible language. In either case, there is a simple syntax for describing document trees while the creation and composition of such trees is unrestricted in large parts. In other words, there is a single parser which is capable of parsing all supported documents.

The document trees in MPS are stored as XML. But the user interface is not a text editor but a composition of tree editors instead. Every concept has an editor attached that provides projections of corresponding nodes. Editors form a hierarchy just like concepts do because an editor of a concept can use the editors of containing concepts. Furthermore, editors can be conditional so that different projections of a given tree are possible under different circumstances. The major benefit of the tree editors is that in contrast to XML and Lisp the contents of a tree can be displayed in a nice and domain-specific way. In most cases, a projection of a tree looks like text but in general graphical projections are also possible.

Generators translate documents into different documents. To this end, there are various kinds of generators that support different transformation techniques such as local search-and-replace of nodes or pattern-based transformations along the lines of XSLT. Document transformations can be chained and the language of the final document is called a base language and has a so-called TextGen aspect attached to each of its concepts. This aspect specifies how a respective node can be translated into text. So, MPS effectively generates text which can be further processed by common compiler tool chains.

In contrast to other code generation techniques, MPS completely knows the target language of a generator. Thus the IDE supports you with context assisting, constraint checking, syntax highlighting and all the other features, which other language workbenches only provide for source languages. However, this also means that if you want to generate code for a certain language there must be an MPS implementation of that language available. Up to now, MPS originally only supports the Java programming language.

Generators and all other aspects of a language are themselves documents of languages provided by MPS and are created by means of the respective tree editors. In other words, MPS consists of a set of domain-specific languages, each for a different aspects of the domain of creating domain-specific languages. This recursion is not only cool but also important because it is possible to extend the language for building generators and thus introduce specialized generators for a given type of a language. In my opinion, this rocks.

The problems with tree projection

As outlined above, MPS documents are created and manipulated by means of projection-based tree editing only. Although most projections look like text at first sight, the underlying tree is omnipresent. In fact, what you have to do to create a document is to create its defining tree. Basically, this means that you have to know how the trees of your language are structured, create one node after the other with the help of the context assistant and set their properties by means of the editors. As this is cumbersome and time consuming, MPS supports a bunch of mitigating techniques.

For example, it is possible to define so-called aliases that are strings which when entered into the editor are replaced by a node of a particular concept. Additionally, there are so-called side transformation rules which execute a mini-generator if a given string is entered left or right of a node satisfying a given condition. So, instead of creating a Plus node with two Number children you could type + next to a Number node to get the same effect. As a last example, intentions offer context-specific transformations which can be triggered manually. Such local transformations in various different flavors all help to make creating documents more comfortable and manageable.

Up to my experience, though, the tree nature of the documents never vanishes. If you want to change your document in a way which has not been anticipated by the language creators and thus is not directly supported, you end up in manually manipulating the tree again. For example, when enriching the behavior of concepts with Java-code, you sometimes create an if statement with a rather complex condition. If after creating the statement you realize that in fact you need the negation of that condition, there better is an intention prepared for that. Otherwise, you copy the condition into the clipboard, and paste it back as the child of a newly created node representing a negation expression. If the structure of the language is complicated, sticking nodes together to form the intended tree becomes really difficult.

In case of the MPS languages, things are even worse because there is no way to create the intended document node by node. Instead, you have to use the ways that are predefined by the IDE. A good example in this context is the fact that there is no way to turn a concrete concept into a interface concept or vice versa. As in MPS a concept can inherent from at most one other concept, you need interface concepts to simulate multiple inheritance. In a text-based environment replacing the keyword class with the keyword interface would be all you had to do to change the type of the concept. In the current MPS version, though, you have to remove the concept, create a concept interface, define its properties, children, references, and so on and update all references to the concrete concept to point to the concept interface instead. The upcoming MPS version is supposed to have refactoring support for this case, but still this example demonstrates that in MPS you need tool support for every single possible creation or manipulation task. Things become messy for the user otherwise.

Aside from the above-mentioned cases, the tree nature of MPS documents additionally manifests itself when navigating a document. You have to get used to the fact that moving the cursor or deleting stuff works on nodes instead of characters. If you know the structure of your tree this allows you to navigate your document incredibly fast. But if you don’t, it can be confusing. And if the focus is on a property of a node, the navigation does work on the granularity of characters. Maybe I didn’t try hard or long enough, but navigating document trees in MPS never felt like an intuitive task to me.

Finally, the tree editors always show the results of the creation and manipulation of nodes. But they never show, how this result has been achieved. So if you want to figure out how something can be done you can not just look at other examples because they don’t tell you how to interact with the IDE to get what you see. In contrast, in case of text-based languages, you simply copy the text and you are done.

If you want to create a new language in MPS, you have to complete the following tasks:

• You have to define the structure of your language, i.e. the concepts and their hierarchy.
• You have to define a type system in order to constraint valid documents in ways which can not be expressed by a reasonable set of concepts.
• You have to specify constraints for properties, child concepts and references to nodes. For example, this is needed to support scopes in your language.
• If you create a base language you have to define the TextGen aspect for all concepts.
• If you don’t create a base language you have to define generators which process documents of your language towards a given target base language.
• You have to define and tune the editors for the concepts to get a nice visualization of the documents.
• You have to identify common usage patterns and define local transformations and intentions for them.

All but the last two tasks have to be done in other language workbenches as well. But the last two tasks are extra work which is necessary to enable of the projection-based tree editing and to compensate for its disadvantages.

The MPS implementation

The MPS project is a huge scientific and engineering effort and I really respect the work that has been done by JetBrains. From a user’s perspective, though, unfortunately it is not enough yet. MPS requires and contains a lot of features which means that a lot of code paths have to be tested and a lot of documentation has to be written. Both things require time and money which might be the reason why they have not been sufficiently done yet. So, if you use MPS, be prepared to encounter bugs. I did so on a daily basis. And if you want to figure out how to do something you better have somebody to ask because the documentation is incomplete and partially inconsistent. Don’t get me wrong, I don’t blame anybody. I just want to point out why in my opinion MPS is not yet ready to be used by the public.

The features that do work, though, are really cool. For example, the context assistant is really powerful and it is very easy to use it for your own languages. In fact, all you have to do is derive your concepts from INamedConcept whenever they have a name and use smart references which are concepts with a single reference with granularity one. The context assistant resolves such references and displays the name of the nodes in question in its menu. Furthermore, by means of reference constraint you can limit the set of nodes which are possible as a valid reference and thus offered by the context assistant menu.

Something I find a little bit weird is the language used for the editors. An editor is a set of cells arranged in rows and columns. Each cell has a number of styles and properties which configure the respective projection. Some of these properties are defined directly in the editor of the editor. Other cell properties are only accessible by means of the so-called inspector which is a general property editor for MPS objects.

In my experience, getting the editors right is not trivial. For example, when projecting a do-while-loop the editor could look like [> do %body% while( %condition% ); <] which is a horizontal alignment of the constant strings do, while(, and ); and the child nodes named body and condition. If the editor of the body is a block statement and thus uses multiple rows, in the projection of a do-node the while-part follows right of the body but on the same line as the do-part. I know that there should exist a combination of cells with the right properties and styles to get a C-like projection with the while-part following the curly braces of the block statement but I never managed to figure it out.

Conclusion

Me experience with MPS comes mainly from my efforts to create a C99 implementation for MPS. The C99 standard is complex and partially weird. So the disadvantages of MPS that I encountered are amplified by the peculiarities of the C99 language. I am sure, that in many cases, DSLs are much simpler and thus the problems of tree-based editing are not so predominant.

In conclusion, I have no final opinion yet, to be honest. It’s clear that you have to get used to the projection-based tree editing. But as I think people should be willing to learn new paradigms if it pays out, the real question is whether it pays out or not.

Being relieved from parsers really eases the creation and especially the composition of languages. But as discussed above, projection-based tree editing is cumbersome because you basically have to create every single node manually. Maybe, in the future, somebody will come up with a way to create document trees by means of projection-based tree editing which is as fast as writing text which a parser turns into a document tree. Until then, special tool support is required to accelerate the creation of document trees and make it feasible. And providing this tool support is time-consuming because you have to identify all common usage patterns to begin with.

Furthermore, I have the impression that no matter how much tool support you provide your users have to learn the structure of the language. Compared to learning a syntax this seems to be harder to manage. But maybe it’s just a matter of getting used to it again. I don’t know yet.

I created the model of this MPS language according to the C99 standard grammar. Thus, I had to dive into its documentation and I have found a number of surprising features and properties of the C99 programming language. Not all of them are important, but if you want to be nitpicker in the next C discussion, here is your ammo

So, did you know, that

the size of a byte is implementation-defined? (3.6)

In contrast to an octet, a byte does not always consist of 8 bits. Instead, it is an “addressable unit of data storage large enough to hold any member of the basic character set of the execution environment”.

argv[argc] shall be a null pointer? (5.1.2.2.1.2)

Very convenient indeed. But I haven’t seen code yet which relies on this.

0 is not a decimal constant? (6.4.4.1)

There are three different types of integer constants, i.e. octal, decimal and hexadecimal integer constants. Octal integers constants are prefixed by 0, hexadecimal constants are prefixed by 0x and for decimal constants there is no prefix. Thus a single 0 is an octal integer constant.

there are hexadecimal floating point constants? (6.4.4.2)

There are two different types of floating constants, i.e. decimal and hexadecimal floating constants. Octal floating constants are not supported.

As e is a valid hexadecimal digit, the delimiter for the exponent part of a hexadecimal floating constant is p. The base for the exponent is two and the exponent itself is a decimal number. So 0xap1 equals 20, and 0xap10 equals 10240.

there are two different types of for-loops? (6.8.5)

We know that the header of a for-loop consists of three optional parts separated by semicolons. But in fact that’s not true. Instead, only the second and the third header part are always optional expressions. For the first header part, though, things depend on the type of the for-loop.

There are two such types. In case of the first type, the first header part is also an optional expression. As this expression is evaluated once when before starting the first iteration of the loop, it could likewise be moved in front of the loop. What the first part of a for-loop is most used, though, is the declaration of a variable which is scoped by the loop body. As declarations are not expressions, this is not possible with this type of for-loop.

That’s why there is a second type of for-loop, where the first part is a declaration. The funny thing is, that in this case the first semicolon is not part of the for loop. Instead, it is the delimiter of the declaration. In order to disallow for-loops with only one semicolon, the first header part is not optional in this case. So for(; i<2; i++){} is a for-loop of the first type with an omitted expression as its first header part.

a typedef declaration does not introduce a new type? (6.7.7)

Instead, only a synonym for the type is specified. I have seen people ranting about C being not type safe, because the compiler allows to use a foo when a bar is expected, if both identifiers are typedefs of the same type. d’oh

the typedef specifier is a storage-class specifier? (6.7.1.3)

The other storage-class specifiers are extern, static, auto, and register. They all deal with how the declared object is stored. typedef is a storage-class specifier, “for syntactic convenience only”.

Storage-class specifiers precede declarations which specify “the interpretation and attributes for a set of identifiers”. If the storage-class specifier is typedef, these identifiers become synonyms for the respective type. So, typedef int a, (*b)() introduces at one swoop a as a synonym for int and b as a synonym for pointer to function with no parameters and returning int.

From my experience, I have never seen anybody using this feature. So I wonder, how convenient this grammatical hack really is.

the coupling of switch statements and case labels is rather loose? (6.8.4.2)

The body of a switch statement is a statement. Most often this is a block statement with case labels and/or a default label. But this is not required. Instead, switch(42) i++; is valid C99 code, where i++ is not executed, because there is no matching case constant expression and no default label in the body of the switch statement.

In general, possible jump targets of switch statement are all case labels which are “in or on the the switch body”, excluding those which are part of an enclosed switch statement. Apparently, being “in the switch body” recursively includes any block statements, enabling things like duff’s device. As far as I can tell, though, this is not explicitly stated in the standard.

the grammar defines a rather large superset of the language?

A grammar describes the building blocks of a language and the rules of combining them. The C99 grammar is very permissive concerning the combination rules. Many things are later disallowed by textually written constraints. But some things are not, although they make no sense.

An example for additional constraints is that the grammar allows an arbitrary list of type qualifiers and type, function and storage-class specifiers preceding a declaration (6.7). Later, 6.7.1.2 states that “at most one storage-class specifier may be given” and 6.7.2 requires that “at least on type specifier shall be given”. Furthermore, the set of allowed unordered sublists of all type specifiers is enlisted. For example, int static long volatile unsigned i; is a valid declaration, because static is a storage-class specifier, volatile is a type qualifier and unsigned long int is a valid sublist of type specifiers.

An example for – in my opinion – forgotten constraints is that a struct definition is a struct specifier which is a type specifier which is a declaration specifier which is part of a parameter declaration (6.7.5). This means, that void f(struct foo { int i; } j); is valid C99 code, although the syntax highlighting of my Vim goes crazy about this. GCC issues a warning, that “‘struct foo’ declared inside parameter list” and “its scope is only this definition or declaration, which is probably not what you want”. Indeed, 6.2.1.4 says, that “if [...] the type specifier [...] appears within the list of parameter declarations [...] the identifier has function prototype scope, which terminates at the end of the function declarator.” I can hardly imagine a case in which a type definition with function prototype scope is needed. So I wonder, why this isn’t forbidden.

there is a clear definition of side effect? (5.1.2.3.2)

“Accessing a volatile object, modifying an object, modifying a file, or calling a function that does any of those operations are all side effects”. It can be so easy.

_Bool is a standard type? (6.2.5.2)

Before C99 there was no Boolean type. Instead, everybody had its own type synonyms and pre-processor macros. In order to no collide with existing code, C99 introduces the keyword Bool with a underscore prefix. But actually you are not supposed to use it, as there is a macro bool which expands to _Bool in stdbool.h (7.16). Furthermore, there are macros for true and false which expand to the integer constants 1 and 0, unless you redefine them. By the way, “an object declared as type _Bool is large enough to store the values 0 and 1″.

_Complex is a keyword? (6.2.5.11)

Common floating represent real numbers. If you prefix a floating type with _Complex it becomes a complex number and is stored as an array of two elements of the corresponding floating type (6.2.5.13).

_Imaginary is a keyword? (6.4.1)

But it is only reserved for specifying imaginary types. It is not normative because “there is little existing practice to validate the design decisions” (G.1).

there are different character sets? (5.2.1)

The basic character set contains the common ASCII upper and lowercase letters, special characters and white spaces. The extended character set contains the basic character set and zero or more implementation defined characters. This is meant to support characteristics of the the various locales. Furthermore, C99 distinguishes between the source character set and the execution character set. The former is the set of characters which form source files. The latter is the set of characters available at execution time. Elements from the execution character set are represented by corresponding elements of the source character set or by escape sequences.

some special characters can be represented by trigraphs? (5.2.1.1)

The output of the puts("wtf??!"); is “wtf|”, because “??!” is a trigraph sequence which is replaced by “|” “before any other processing takes place”. In total, there are nine different trigraph sequences, all starting with “??”. So you should escape question marks in string literals. The output of puts("wtf\?\?!"); is “wtf??!”.

Fortunately, GCC disables trigraphs by default and issues a warning if the source files contains trigraphs.

some punctuators can be represented by digraphs? (6.4.6.3)

In contrast to trigraphs, digraphs are replaced after tokenization. In particular, this means that digraphs within string literals are not replaced. In total, there are six different digraphs, all starting with an opening angle bracket or with a percent sign. The represented punctuators are opening and closing square brackets, opening and closing curly brackets and single or double hash.

Introduction

Pervasive computing aims at enriching modern life with IT services which form a smart infrastructure embedded in everyday environments. With increasingly cheaper, smaller, more efficient and in general more capable hardware, more and more pervasive applications are becoming possible. Examples include healthcare platforms for diabetes management and measurement platforms for human contact networks and epidemiology research.

The unobtrusive integration of pervasive applications into the real world often requires tiny, battery-powered devices to cooperate in a distributed fashion over wireless links. Due to these requirements, pervasive applications tend to be complex and thus difficult to develop, deploy and maintain. In order to make the complexity manageable, suitable programming abstractions which help writing and maintaining code for the devices are required. Unfortunately, programming abstractions offered by high-level programming languages tend to require more CPU and memory resources than a pervasive computing device typically offers. As a consequence, pervasive applications are often implemented in the C programming language which offers only few and rather low-level programming abstractions. Thus, most pervasive applications are notoriously hard to write, deploy and maintain.

The need for programming abstractions for resource-constrained devices has since long been recognized by the research community, in particular in the field of wireless sensor network (WSN). WSNs are a pervasive computing technology which supports monitoring of physical phenomena over a potentially large spatial region. Because the peculiarities of pervasive computing which drive the demand for programming abstractions are especially predominant in WSNs, a number of solutions have been proposed in the past few years [Mottola].

Little has been done, though, to also support debugging on the level of the abstraction [Mottola]. As a consequence, software developers often have to understand lower-level details such as the syntax and semantics of generated code in order to trace program errors. As hiding lower-level details is one of the main goals of abstractions, we consider programming abstractions without debugging support as incomplete.

We thus want to investigate programming abstractions with debugging support for resource-constrained devices. In particular, we focus on WSN because many programming abstractions without debugging support already exist in this field. We furthermore expect our results to be also applicable to other fields of pervasive computing.

State of the Art

WSN programming abstractions can be classified by two major orthogonal dimensions: node-centric vs. distributed and imperative vs. declarative [Mottola]. Node-centric abstractions focus on programming single nodes which might or might not form a distributed system. In contrast, abstractions for distributed applications account for the distributed nature of the application and support the programming of groups of nodes such as spatial or logical neighborhoods.

Irrespective of the first dimension, with imperative abstractions the intended application processing is expressed through statements that explicitly indicate how to change the program state [Mottola]. In contrast, for declarative abstractions the application goal is described without specifying how it is accomplished [Mottola]. Out of the four possible classes of programming abstractions only three are relevant in practice and we discuss them in the following.

Protothreads Protothreads and TinyVT are node-centric, imperative programming abstractions. They offer compiler-based thread abstractions by taking a thread-based program as input and generating a semantically equivalent event-based program for the targeted event-based operating system. While this combines the comfort of thread-based programming with the efficiency of events, both of them have no debugging support. Thus, a developer is forced to step through the generated event-based code in order to identify defects in the thread-based program.

A second class of programming abstractions is imperative and distributed programming abstractions which are commonly known as macro-programming in the WSN community. While the existing abstractions like Kairos and Regiment are powerful, their practical adoption seems to suffer from the lack of debugging support. In this case, besides the knowledge of lower-level details, a software developer additionally needs special tools such as distributed debuggers (Clairvoyant) because a single macro-program controls the behavior of many distributed devices. Compared to node-centric abstractions, the lack of debugging support on the level of the abstraction is therefore even more severe. To the best of our knowledge, the only macro-programming language with debugging support is MacroLab which is a dialect of MATLAB and can be debugged with the Macrodebugger, a post-mortem debugger specifically crafted for MacroLab.

Two well-known declarative and distributed programming abstractions are Cougar Cougar and TinyDB. They model a WSN as a relational database and support SQL-like queries to retrieve information about the monitored physical phenomena. While this is what users presumably want, if an error occurs somewhere in the distributed system, the user has to resort to other means like passive inspection of the network traffic (SNIF) in order to find the cause of the problem.

Since WSN applications are rather diverse, suitable programming abstractions will have to be specifically crafted for a given application domain. Such programming abstractions are usually implemented by so-called domain-specific languages (DSL). In software engineering there is an ongoing trend towards DSLs in general and DSLs for building DSLs in particular (actifsource, MPS, Xtext, Spoofax, MetaEdit+). With the support of these so-called language workbenches it is rather easy to quickly introduce DSLs and to implement the corresponding compiler. Again, though, the issue of debugging on the level of the abstraction has not been solved yet.

Goals of my Thesis

The goal of my thesis is to introduce debugging support of programming abstractions for resource-constrained devices. To this end, we will focus our investigations on two major topics.

First, as a concrete example we want to introduce a debuggable, compiler-based thread abstraction for event-based operating systems. The expected outcome is that from a developer’s perspective there is not much difference from using our compiler to using a thread library. But still, the achieved efficiency is as close as possible to the efficiency of hand-written event-based code. Furthermore, source-level debugging of the thread-based program is possible.

Compared to thread libraries there are theoretical limits of what features a compiler-based approach can offer. These limitations stem from the fact that a compiler can only process decidable problems. In order to stay within these theoretical bounds, we can neither commonly support calling blocking operations by function pointers nor calling blocking functions from recursive functions. Furthermore, dynamically starting new threads is not possible. However, as it is advisable to statically ensure that the scarce resources of a given device suffice the demands of a given program, such dynamic features are rarely used for embedded systems. Thus, we don’t consider these limitations to be severe, especially considering that we expect to be able to get close to the efficiency of the event-based paradigm.

Additionally, if the compiler records which thread-based code was translated into which part of the event-based program, it becomes possible to perform source level debugging in the thread-based code. In principle, this technique is very similar to what C tool chains use in order to support source-level debugging of C programs. Thus, we expect to be able to reuse and build upon a large number of existing work such as the DWARF Debugging Standard.

For the second part of the thesis, we want to investigate in more general terms how programming abstractions can be designed and built in a way such that debugging support is included. Therefore we want to identify standard debugging techniques for the various types of programming abstractions encountered in the WSN field. By extending existing language workbenches we enable the design and implementation of various types of DSLs with the respective debugging support integrated. The expected outcome is that software developers will be able to easily create programming abstractions like TinyDB or Kairos with integrated debugging support.

For node-centric and imperative DLSs, simple debugging information emitted by the compiler is known to suffice. For distributed DLSs, though, existing debugging tools such as distributed debuggers (Clairvoyant) or passive distributed assertions (PDA) must be incorporated into the tool chain such that information about an error which is reported from the debugging tool points to the error’s cause in the program. In the case of declarative programming abstractions, debugging is even harder because there is no one-to-one mapping between the program and the generated code. Thus, more detailed information is necessary in order to be able to map from the generated code back to the program.

Meta

• This is a paper published at the Doctoral Colloquium of the 8th ACM Conference on Embedded Networked Sensor Systems (SenSys 2010).
• This work has been partially supported by the National Competence Center in Research on Mobile Information and Communication Systems (NCCR-MICS), a center supported by the Swiss National Science Foundation under grant number 5005-67322, and by CONET, the Cooperating Objects Network of Excellence, funded by the European Commission under FP7 with contract number FP7-2007-2-224053.
• I want to thank Prof. Dr. Kay Römer for supporting me with this work.

Introduction

Event-driven programming has traditionally been the paradigm of choice to program resource-constrained sensor nodes [Contiki, TinyOS]. The use of events indeed allows to efficiently handle the concurrency-intensive operations such devices must typically deal with [TinyOS]. In contrast, using threads in such a high-concurrency context induces a significant overhead in terms of memory usage and number of context switches. Therefore, although some thread-based solutions exist [Mantis OS, Darjeeling], major operating systems for sensor nodes such as TinyOS or Contiki rely on an event-driven paradigm.

However, event-based programs are notably harder to write, maintain, and debug than their thread-based counterparts [Adya2002]. Some authors thus introduced ad-hoc thread libraries to event-based operating systems for sensor nodes in order to facilitate application development [Duffy2007, TOSThreads, TinyThread,YThreads]. As an alternative, thread-based programs can be automatically translated into equivalent event-based code using an appropriate compiler. Such compiler-based approaches can apply static code optimizations thus producing potentially more efficient code than library-based solutions. Existing approaches such as Protothreads and TinyVT have a number of limitations, though. To the best of our knowledge, our system, which we discuss below, is the only to compile almost arbitrary cooperative-thread-based code written in the C language to event-based code for any event-based runtime environment.

Since the thread semantics are implemented at compiler level, there are the following limitations. First, the number of threads must be known at compile time. Second, recursive functions must not invoke blocking functions. And third, function pointers must not be used to invoke functions that directly or indirectly invoke blocking functions. We argue, though, that these limitations do not severely affect programming of sensor nodes. In particular, recursive algorithms tend to have an undecidable stack consumption and thus should generally be avoided in embedded systems. Again due to the constrained resources of sensor nodes, it is often not sensible to support an arbitrary number of threads. Finally, the use of function pointers is not uncommon, but can be avoided by a case differentiation (i.e., if case one, call function A; if case two, call function B; …) with moderate overhead. What’s most important about these constraints is the fact that the compiler can reliably detect violations of them.

The remainder of this paper is organized as follows. After briefly reviewing related work in the section “Related Work”, we detail how we can automate the threads-to-events code transformation step in
the section “From Threads to Events”. The preliminary evaluation of our system, presented in the section “Evaluation”, shows among other things that the generated code can compete with hand-written code in terms of RAM consumption. Finally, we provide our conclusions.

Related Work

In the context of wireless sensor networks (WSNs), thread programming on top of event-based systems had first been enabled by the development of ad-hoc thread libraries. For instance, TinyThread provides multi-threading to TinyOS programmers. This library includes a tool to estimate the per-thread stack depth and thus allows to minimize RAM overhead. Similar solutions [Duffy2007, TOSThreads, YThreads] also provide cooperative or preemptive thread abstractions on top of TinyOS or other mote operating systems. However, since libraries implement the thread semantics at runtime, their usage does not allow to apply static code optimizations at compile-time. In contrast, compiler-based approaches can exploit this potential for optimization.

The probably best-known compiler-based solution in this context is the Protothreads approach by Dunkels et al. [Protothreads]. Programs written using Protothreads are transformed into event-based code using the C preprocessor. However, while using the C preprocessor guarantees portability across C compilers, it also introduces some limitations. For instance, certain C language constructs such as switch statements may not be used and values of local variables are not retained across context switches [Protothreads]. Furthermore, thread functions are not reentrant, blocking calls may only occur in the top-level thread functions, and debugging is performed in the generated code.

In contrast, we argue for a comprehensive threads-to-events compiler, as introduced by TinyVT. This approach extends the event-based nesC programming language by introducing a wait operator that suspends program execution until a certain event occurs. The wait operator also specifies the event handler code that should be executed when the event occurs. In addition, TinyVT provides automatic variables that can be accessed from the event handler code. In contrast to our work, however, TinyVT does not completely hide the event semantics of the underlying system. Furthermore, it was specifically designed for nesC/TinyOS, while our approach supports a wide range of event-based runtime environments. Finally, TinyVT does neither support reentrant calls of functions containing wait operators nor event handlers containing wait operators.

From Threads to Events

The main challenges related to the transformation of thread-based code (T-code) into event-based code (E-code) stem from the fact that a call to a blocking function such as read or sleep must be rewritten into triggering the operation and registering the continuation, which is executed by the runtime environment upon completion of a blocking operation. Every function which potentially calls a blocking function, either directly or indirectly, is affected by the transformation. We denominate such functions critical functions and a critical call is a call of a critical function.

The transformation of a critical call into a corresponding split-phase operation has two major implications. First, the code following a critical call must be callable, thus the control flow must be modified appropriately. Second, automatic variables in critical functions that are set before the critical call and read afterwards must be preserved.

Control Flow

To be amenable for event-based execution, critical functions have to be split into a part that triggers the critical call, and a part that is invoked upon the completion of the critical call. However, if the critical call is nested in control structures such as loops, the transformation may be more complex and is performed as follows. First, the T-code is transformed into an equivalent GOTO program which is a sequence of commands with labels. It is well known that this transformation is always applicable and actually this is what C compilers do in general. Second, the resulting code is split into basic blocks at all labels and critical calls. Third, each resulting basic block is put into its own E-code sub-function. Then, all GOTO commands and transitions between basic blocks are replaced with calls to the corresponding E-code sub-functions. Furthermore, the call to the critical function is replaced by the call to the corresponding first E-code sub-function. Finally, C code is generated for all those functions.

Consider for example the following function f in which statements_x represents an arbitrary sequence of C language statements and cond is a Boolean expression. The E-code obtained by transforming the GOTO program back into C functions is also shown below.

// T-code
statements_1;
while(cond) {
    statements_2;
    crit();
    statements_3;
}
statements_4;

// GOTO program
      statements_1
loop: IF !cond GOTO end
      statements_2
      CALL crit
      statements_3
      GOTO loop
end:  statements_4

// E-code
void f_1() {
    statements_1;
    f_2();
}

void f_2() {
    if (!cond) {
        f_4();
    } else {
        statements_2;
        crit_1();
    }
}

void f_3() {
    statements_3;
    f_2();
}

void f_4() {
    statements_4;
}

Likewise, crit() is compiled into a set of C functions, such that upon completion f_3 is invoked as described in the section “Returning”. Furthermore, the value of an automatic variable of f must be retained across calls of the generated functions f_x. This is described in the following section.

Local Variables

For every critical function a statically allocated state structure is generated. This structure holds all automatic variables and all parameters that need to be retained across critical calls. Due to the possibility of reentrant function calls, there is one copy of this state structure for each thread. The resulting E-code then accesses those state structures instead of automatic variables. Note that non-critical functions are not affected by this transformation.

To minimize the RAM consumption, we allocate the state structures for certain functions to the same memory location. In particular, if a critical function sequentially calls two other critical functions, so that the local variables of the latter two functions cannot be active simultaneously, we can allocate the state structures of those two functions to the same memory location using a union. In contrast, the local variables of a function and its calling function are active simultaneously and thus cannot share the same memory location. Using this principle, we can translate the call graph of a critical function directly into a structurally equivalent nested data structure of C structs (for nested calls) and unions (for sequential calls), holding function state structures as in the following example. Here, state_x denotes the state structure of a function x. run_send is the start method of a thread and sleep, read, and send are blocking functions.

// T-code
void run_send() {
    while(1) {
    sleep();
    aggr();
    send();
}    

void aggr() {
    read();
    //...
}

// corresponding states
struct {
    state_run run;
    union {
        state_sleep sleep;
        struct {
            state_aggr aggr;
            state_read read;
        } struct1;
        state_send send;
    } union1;
} state_send;

There is one such nested data structure for each thread, representing the call graph starting at the main function of that thread. As multiple threads can make reentrant calls to the same function, there must be a mean to retrieve the structure corresponding to the current thread. To this end, we generate a mapping function that returns a pointer to the proper state structure instance. This mapping is based upon the value of a global variable tid, which holds the identifier of the current thread. For instance, for function aggr in the example above the mapping function would look like:

state_aggr* map_aggr() {
    switch(tid) {
        case SEND_THREAD:
            return &state_send.union1.struct1.aggr;
        //...
    }
}

We emphasize again that the above transformation only applies to critical functions. Non-critical functions are just copied from T-code to E-code. In addition, only variables whose values need to be retained across calls of sub-functions are affected by the transformation. These are all variables that are written in one sub-function and read in a subsequent one. All other variables such as iteration variables of for loops not containing critical calls remain automatic stack-based variables.

Returning

If a T-code function is only invoked from one place in the code, then the control flow can be hard-coded into the generated E-code. In contrast, if a function may be called from multiple code locations, an explicit return address needs to be saved. In our compilation framework, we include the return address in the state structure of the invoked function much like a local variable while the same optimizations as described in the section “Local Variables” apply. The caller has to set the return address before calling the function as in the following code referring to the example from the section “Control Flow”.

void f_2() {
    if (! cond) {
        f_4();
    } else {
        statements_2;
        map_crit()->cont = f_3;
        crit_1();
    }
}

void f_4() {
    statements_4;
    (*map_f()->cont)();
}

Likewise, return values of critical functions are also stored in its state structure. After return, the caller can retrieve it from there. So a complete state structure for a critical function int f(int x) would look like this:

struct state_f {
    void (*cont)(); // continuation
    int res; // result
    int x; // parameter
    ... // other local variables
}

Platform Abstraction Layer

The so-called platform abstraction layer (PAL) contains event-based implementations of blocking calls in the T-code. The compiler generates E-code skeletons for these functions, but the actual calls to the underlying event-based runtime environment have to be inserted manually. Besides adapting the API of the runtime environment to the generated E-code, the PAL also enables portability.

As an example consider the blocking function sleep in the T-code. There is a hand-written E-code implementation of sleep_1 and sleep_2 in the PAL. Furthermore, the compiler generates the state structure state_sleep according to the T-code signature of sleep, and the map function map_sleep according to the call graphs of the program. sleep_1 uses the event-based API of the runtime environment to set up a timer and register sleep_2 to be called when the timer fires. sleep_2 will then call the continuation which was saved in the proper instance of state_sleep by the caller of sleep_1.

While a thread is waiting for the completion of a blocking function, events related to other threads can of course be dispatched. This requires that a (lightweight) context switch is performed by saving the value of the global variable tid holding the current thread ID in sleep_1 and restoring it in sleep_2.

Interfacing the Compiler

In order to perform the transformation, our compiler needs some additional information besides the T-code. In particular, it must know the set of blocking functions, the total number of threads, and the start functions for each of them.

In the context of this work, we assume the declarations for the blocking calls to be available in a T-code header file. The number of threads and their start functions are specified in the main function of the T-code program. More specifically, each invocation of the built-in function RUN_THREAD causes the global variable tid to be incremented. The pointer to the start function of the thread is passed as an argument to the RUN_THREAD function. This start function will eventually call a blocking function and return. Once all threads have been started in this way, the generated E-code variant of main passes control to the event dispatcher loop of the event-based runtime environment.

Evaluation

To verify the feasibility and the practicability of the proposed code transformation, we performed a preliminary experimental study using an application that performs data storage and collection. We started with a hand-written event-based implementation of that application. We then rewrote the same application using cooperative threads with blocking I/O. The purpose of this thread-based implementation was to serve as input to a manually performed code transformation as described in section \ref{sec:transformation}, resulting in a generated event-based version of the application. In the following, we compare the efficiency of the generated event-based code with respect to the hand-written E-code.

The considered test application performs distributed data aggregation with local storage using three main tasks. First, it continuously reads a sensor and stores the values in flash memory (collect task). Second, it periodically receives radio messages from other nodes containing their sensor readings and also stores them on flash memory (receive task). Finally, it regularly reads the flash memory to compute minimum and maximum readings and sends the results over the radio (send task). In the thread-based application, each of these three tasks is performed by a separate thread. Furthermore, the function that writes data to the flash is reentrant since it is invoked by both the receive and the collect threads.

In order to evaluate the correctness of the transformation we first executed both implementations of our test application in our own simulation environment. By analyzing the corresponding API call traces we were able to verify that they behave equivalently. To this end, we choose TinyOS 2.x as a model for the underlaying runtime environment. However, since the transformation applies solely to application code, only the API of the operating system but not its implementation is of significance. Therefore, we simply manually compiled the necessary TinyOS interfaces (e.g., Read, Send) to C code along the lines of the nesC compiler.

Compared to directly working with TinyOS, this approach gave us greater flexibility as we could provide different implementations of the operating system, depending on the needs of the various tests and measurements we performed. The complete source code of the study along with more detailed documentation is publicly available.

Besides verifying the feasibility of our approach we also compared hand-written and generated code in terms of consumption of ROM, RAM, and CPU resources on a sensor node equipped with an AVR ATmega128.

Results

We cross-compiled the hand-written and the generated event-based application code using avr-gcc version 4.1.2 for the atmega128 microcontroller with optimization option -O2. We then linked the cross-compiled code against an empty implementation of the operating system. The size of the DATA and the BSS sections of the resulting binaries then allows to directly compare the RAM consumption of the two applications. The following table shows the resulting sums of both sections in bytes and the ratios for different sizes of I/O buffers (see the section “Discussion” for a discussion of I/O buffers). Note that the numbers for the generated application include the PAL.

The size of the TEXT sections of the cross-compiled object files indicate how much flash is needed to store the program in the microcontroller. The results are 3166 bytes for the generated application vs. 1080 bytes for the hand-written one, which is a ratio of 2.93. As the PAL is extra code which does not exist in the hand-written case it influences the estimation of the code transformation. Excluding the PAL, the code size of the generated code reduces to 1684 bytes or a ratio of 1.56.

We finally measured the consumption of CPU resources by running the applications in the Avrora simulator. For that purpose we wrote a simple event dispatcher implementation of the operating system. Additionally, we introduced a new main function that after setting up the environment makes each thread perform one iteration of its periodic work.

The results are 4875 CPU cycles for the generated code vs. 1679 cycles for the hand-written code, making for a ratio of 2.78. In order to measure the properties of the code transformation itself we confined the sums on cycles which are executed in the application code, excluding the setup phase and the PAL. Then we measured 1368 vs. 633 cycles, making for a ratio of 2.16.

Discussion

The selected application represents a “stress-test” for our compilation framework as the code mostly consists of invocations of blocking calls without much “computations”. As in our framework overhead results from blocking calls but not from computations, this application can be considered a worst-case scenario.

Another important issue regarding the interpretation of the performance results is the implementation of the PAL. As discussed in the section “Platform Abstraction Layer”, the PAL needs to perform a context switch by saving and restoring the current thread ID. Unfortunately, the TinyOS API does not allow to pass parameters to callback functions when triggering the split-phase operation. Such a mechanism would allow a very efficient implementation of context switches. Instead, our PAL implements a rather inefficient mapping from callback handles to thread identifiers.

For RAM consumption, though, the results for the generated code including the PAL are comparable to those for the hand-written code. This is due to memory sharing of state structures holding local variables in the E-code. The measurements for different sizes of the I/O buffers make this apparent. In the hand-written application, each buffer is a separate static variable in a translation unit. In the thread-based application, these buffers are automatic variables and thus become part of the state structures where they share memory. Thus with larger I/O buffers the ratio becomes smaller to a point where the generated code even becomes more efficient than the hand-written one.

The code size overhead can be explained by the very nature of the code transformation, as every critical function results in the generation of several sub-functions and a map function. The bottom line is that the number of functions is roughly doubled. As the application does not do much computation, the administrative overhead of each single function is significant. Thus doubling the number of functions results in roughly twice the code size.

As the ratio of the cycle counts is higher than the ratio of the code size, the generated code must consist of more expensive instructions on average. According to the AVR ATmega128 datasheet this is due to the call and ret instructions which have a very high code-size-to-cycles ratio. This affects the generated code because it contains, as we outlined above, many function calls.

We expect realistic applications with more computations to result in better ratios for code size and cycles. However, we also expect the ratios to increase if the number of critical calls per function increases. A more thorough study of those factors is subject of ongoing work.

In general, we need to investigate further optimizations to reduce the number of generated E-code functions. For example, the transformations of the application used in the evaluation resulted in sub-functions which do nothing but call another sub-function. Clearly, this could be avoided. Along these lines we believe that in several cases the transformation could result in more efficient code than in the general case as described in the section “From Threads to Events”.

As pointed out in Darjeeling, in the context of WSNs it is often acceptable to trade off lower RAM footprint against a higher execution time. In our case, efficient storage of the state structures has to be paid with the costs of the map functions.

Conclusions

We provided the design of a comprehensive compilation framework that automatically generates event-based code from thread-based programs. We tested the feasibility of our approach on a simple but representative case study that can be considered a worst-case scenario. Our preliminary experimental results show that the generated code can compete with hand-written code in terms of RAM usage, while both code size and execution time are larger. We believe, though, that additional optimizations are possible and will reduce both ratios. Furthermore, we expect the ratios to decrease for applications performing longer and more complex computations.

Meta

• This is a paper published at the ACM HotEMNETS 2010 Workshop on Hot Topics in Embedded Networked Sensors.
• This is joint work with Kay Römer, Silvia Santini and Junyan Ma.
• This work has been partially supported by the National Competence Center in Research on Mobile Information and Communication Systems (NCCR-MICS), a center supported by the Swiss National Science Foundation under grant number 5005-67322, and by CONET, the Cooperating Objects Network of Excellence, funded by the European Commission under FP7 with contract number FP7-2007-2-224053.
• .pdf, BibTex.

Introduction

Computer systems tend to get more complex over time. The pervasive computer, for example, consists of many resource-constrained embedded devices that form wireless ad hoc networks to cooperatively solve a task. Programming these systems is difficult.

One common way of addressing this difficulty is to raise the level of abstraction in order to relieve the programmer from lower-level details. Hereby the basic idea is to provide proper programming abstractions by means of a formal meta-language to a software developer in a given application domain.

The benefits of programming abstractions are frequently cited. More abstraction means less code which in turn means fewer possibilities for bugs, be it typos, copy-and-paste errors, inconsistencies, or logical errors. Furthermore, by explicitly representing the concepts of the problem domain in the programming language, a program is easier to create and understand for domain experts. On the long way from an idea in somebodies mind to an executable representation of the same, some steps have to be performed manually and some can be automated. The overall goal of programming abstractions is to reduce the manual part as much as possible.

For the scope of this article, we will adopt the term programming abstraction that is embodied in a meta-language. A program is an instance of a language and forms a solution for a problem in the domain of that language \cite{dmitriev}. A programming abstraction is implemented by mapping it to lower-level abstractions which are themselves embodied as a program in a target language and so on until a program can be directly interpreted or executed by a virtual or physical machine. We focus on programming abstractions in the field of pervasive computing, in particular for applications of wireless sensor networks (WSN).

In this field, several programming abstractions do already exist. For most existing programming abstractions though, debugging cannot be performed at the conceptual level of the programming abstraction, but has to be performed at the level of the target language to which the programming abstraction is mapped [Mottola2010]. This is a major difficulty, as the developer suddenly has to become aware of the target language, and has to understand its semantics and the semantics of the mapping to the target language.

In this position paper we discuss the problem of debugging abstract programs and we postulate that researches should address the question how such programs can be debugged at the conceptual level of the programming abstraction, thus hiding the very existence of the target language, its semantics, and the mapping — much like most C programmers are unaware of the syntax and semantics of the assembly target language.

Besides the C programming language and its tool chains, there are various different technologies which raise the level of abstraction in order to mitigate the growing complexity of computer systems. For example, the software engineering community has devised Model Driven Development. Others have proposed Language Oriented Programming [Dmitriev2004] or Intentional Programming [Christerson2008]. Additionally, internal and external Domain Specific Languages (DSL) are hot topics in the design of programming languages. While some debugging concepts from those technologies could potentially be applied to the pervasive computing context, pervasive computers differ from other systems in their distributiveness, scale, heterogeneity, resource and energy constraints, as well as their deep embedding into the real world. Hence, we believe that research is required to devise not only tools, but the fundamental concepts of how to debug pervasive computers.

In the remainder of this paper, we present requirements for debugging and a coarse taxonomy of programming abstractions. For each class in the taxonomy, we discuss the challenges involved in debugging, what kind of debugging support is required, and how severe the lack of the latter is.

Requirements and Taxonomy

While it would be desirable to build systems that are correct by design or at least to perform a complete static verification of the program code, this is only partially possible for pervasive computers because it would require a complete formal model of the real world. Unlike other computer systems, pervasive computers closely interact with the real world, not only by means of sensing and controlling physical processes, but also through undesirable interactions due to environmental influences. For example, environmental temperature has a strong influence on the frequency and start-up time of crystal oscillators, on received radio signal strength, and on the (dis)charging characteristics of batteries and is thus sometimes causing unpredictable partial node and communication failures. Hence, debugging pervasive computers is a necessity. By debugging we mean finding and removing the causes of incorrect application behavior by examining the its runtime.

In particular, we consider two common types of bugs. Firstly, logical errors in the application code and, secondly, incorrect assumptions in the application code about the semantics of the target language and system to which the application code is mapped. In existing systems, incorrect assumptions concerning the reliability and the timing of message delivery for example are a common cause for failures.

In principle it is possible – and this is what we are aiming at – to find the above two types of bugs at the conceptual level of the programming abstraction, i.e., without exposing to the user the target language, its implementation, and the mapping of application code to the target language.

Debugging techniques for pervasive computers have to meet a number of constraints. Firstly, it must be possible to debug large heterogeneous networks of pervasive computers. Secondly, resource and energy constraints of pervasive computers require that resource and energy consumption of debugging techniques is minimized – otherwise we run the risk of severe probe effects, where the debugger significantly alters the program execution, resulting in (dis)appearance of bug symptoms.

To better understand the problem of debugging abstract programs, we will discuss debugging characteristics of different classes of programming languages for pervasive computers. For this taxonomy, we see two major dimensions: imperative vs. declarative languages and languages for node-centric vs. distributed applications. Actually, those two orthogonal dimensions form a subset of current and common taxonomies for WSN programming models [Mottola2010] and we consider them crucial for debugging.

With imperative languages the intended application processing is expressed through statements that explicitly indicate how to change the program state [Mottola2010] while with declarative languages the application goal is described without specifying how it is accomplished [Mottola2010]. This difference is important because with imperative languages the mapping between source code and target code tends to be one to one. But by raising the level of abstraction towards declarative languages the mapping becomes more complex and even ambiguous, thus making it hard to perform the back-mapping which is required for debugging.

The second dimension of our taxonomy addresses the problem of shared state which is changed concurrently. As opposed to node-centric programs, with distributed programs well-known concepts such as breakpoints need to be reconsidered to make them applicable for distributed applications.

In the following sections we discuss the four classes of programming abstractions which are formed by those two dimensions. While we focus on programming abstractions for pervasive computers, we also look at related application domains that raise similar problems as an inspiration of how one might tackle the problem for pervasive computing.

Imperative Meta-Languages

In the WSN community the most famous example of an imperative meta-language is probably TinyOS [Hill2000]. The programming language of TinyOS is nesC, which is a DSL for the domain of event-based and component-based applications. The nesC compiler translates nesC code into C code which is then compiled by a platform compiler for an embedded platform such as avr-gcc.

Imperative programming languages are a list of commands and command abstractions which tell the executing machine what it is supposed to do. So the most natural way of debugging is to observe the sequential execution of these commands by means of a source-level debugger. Programmers are familiar with this concept because most tool chains provide such as debugger. In fact, a C programmer would very likely refuse to use a tool chain which does not offer a debugger that allows to set breakpoints in the source code and to monitor variables during execution. This is because by the lack of a source-level debugger he is forced to trace execution on the assembler level, find the flaw there and infer the location of the bug in the C code. Not only must he know the complete target language but also the details of the transformation performed by the compiler in order to be able to undo it.

In the case of TinyOS this means that a developer must use the debugger of the C tool chain and step through the generated C code. As a consequence she must be familiar with the implementation-specific naming scheme for the generated C symbols [nesC reference] and must know how the wiring of nesC components is mapped to C code in order to map back the location of a bug in the generated C code to the corresponding location in the nesC code.

To support source level debugging, C compilers add debugging information to the assembly output, indicating which line in the C source code has been mapped to which lines in the assembly code. A source-code debugger can use this information to map the execution state of a binary program to lines in the source code and to names of variables. As C itself is a programming abstraction, this technique can be adopted to support debugging of imperative meta-languages.

In fact this has been done for the YETI Eclipse plugin [Burri2009] which is the first source-level debugger for the nesC programming language. It uses the Eclipse C/C++ Development Tooling (CDT) to communicate with an underlying instance of the GNU Debugger (GDB) and automatically performs the inverse mapping of the nesC compiler. This is possible because the nesC compiler records which part of the generated C file originates from which part of the various nesC input files by exploiting C preprocessor instructions. To the best of our knowledge, the YETI project is gladly embraced by the TinyOS community because it enables the developer to completely ignore the fact that nesC programs are in fact mapped to C.

Along the lines of YETI, we suggest that programming abstractions should always be designed such that they are amenable to source-level debugging. Probably the most notorious imperative programming abstraction without debugging support are C++ templates. Compiler error messages caused by bugs in C++ template code are so verbose and hard to read that projects like stlfilt try to provide a reverse mapping by means of compiler-specific heuristics. An additional problem with C++ templates is that they are also used to implement declarative languages as opposed to imperative ones, as discussed in the subsequent section.

Declarative Meta-Languages

Declarative languages specify a goal, but not how this goal can be achieved. Therefore, a program written in a declarative languages is often called a model. Parser generators are a well known example of tools which process declarative input in order to generate executable implementations of the parser. In the case of Boost.Spirit, the intended parser is declared by means of instantiations of C++ templates. Other projects like ANTLR use a dedicated meta-language instead. Another common case of declarative meta-languages are persistence and database abstractions as in the case of the Hibernate Framework or RubyOnRails.

In general, debugging becomes harder in the case of declarative languages. First, because they tend to be more abstract and second, because the transformation to an imperative target language is not a simple one-to-one mapping any more. As declarative languages are not based on the sequential execution of commands, debugging concepts such as single-stepping and breakpoints are not applicable.

Instead debugging can be supported by providing different views on the model so that the programmer can perform a visual inspection. Visual inspection is only applicable to small models, but fortunately raising the abstraction often leads to reducing the size of the code. Intentional Software applies this approach and claims to have received positive feedback from their costumers [se-radio151], which in their case are usually domain experts without IT background.

An alternative general concept for debugging models is to simulate or execute example cases. KODOS, for example, offers such an approach for debugging regular expressions. Given a regular expression and an example string, the application highlights the parts of the string which match and displays the contents of the matching groups. So KODOS basically supports and automates trail-and-error cycles which otherwise would have to be performed manually.

In the worst case without debugging support, the programmer is forced to read or debug the generated code. As the gap between the programming abstraction and the target language is typically rather high, the disadvantages of this approach as described in Sect. \ref{sec:imp} are even more severe.

Imperative and Distributed

The pervasive computer is a massively distributed system with unreliable components. As programming such distributed systems is difficult, imperative programming languages have been devised that abstract from certain distribution aspects. One example for such a language is Kairos [Millstein2005], which allows computations involving different nodes without explicit network communication. This is very convenient but yet it is completely unclear how to debug Kairos programs. Traditional single-node source-level debuggers cannot be applied here, as single stepping or breakpointing one node may violate timing assumptions in the remaining network which can itself lead to failures. Clairvoyant [Yang2007] addresses this issues by adding WSN specific features to the GDB, i.e. global stop and continue commands. The downside still remains, though, which is that interactive debugging sessions on deployed networks deplete the batteries due to heavy usage of the energy consuming radios thus reducing the overall network lifetime.

A promising imperative meta-language for WSNs is MacroLab [Hnat2008] which is a vector-based meta language and framework inspired by Matlab. MacroLab programs are composed of computations on vectors containing data on nodes such as sensor values, internal state, and parameters for actuators. Hereby the communication of the vectors among the nodes is hidden from the programmer. Though, the most important fact is that there is a debugger (MDB) [Soocor2010] for MacroLab. MDB is a post-mortem debugger which synchronizes and analyzes log files written by the individual nodes. It supports common debugging concepts such as breakpoints and stepping for both temporally and logically synchronous views. Because MDB does not operate at runtime, it can and does support time travel and rendering effects of hypothetical changes. Furthermore there is a deployment mode in which no logging is done but the timing of the debugging mode is preserved thus avoiding probe effects.

Apart from MacroLab and MDB, few approaches exist to assist in debugging networks of pervasive computers at the conceptual level of the programming language. One such approach is to annotate the program with distributed assertions [Römer2008] that formulate hypothesis about distributed variables which are checked by a a runtime system. When placed carefully in the code, failed assertions can help debug. However, this is largely decoupled from the programming abstractions offered by the meta language. As in this example, today one typically has to resort to distributed debugging techniques that are independent of the actual programming abstractions. Dustminer [Khan2008], for example, applies data mining techniques to event traces obtained from nodes to find bug symptoms. Other tools [Ramanathan2005, Chen2008,Ringwald2006,Rost2006] can detect certain predefined problems such as node reboots, routing loops, or network partitions. But there is no easy way to relate these symptoms to possible causes in the program.

Declarative and Distributed

Languages in this class offer declarative programming of a distributed system as a whole. One well-known example in the context of WSNs is TinyDB [Maden2005] which provides a database abstraction. The outputs of distributed sensors form a virtual database table, over which the user can issue SQL queries. Hereby the user is largely unaware of the fact that he is actually programming a distributed system.

With respect to debugging, the difficulties of declarative and distributed languages multiply in this class of our taxonomy. It is already unclear which debugging concepts should be applied to declarative queries, let alone the adoption of such concepts to a distributed setting. To the best of our knowledge no solution exists to support debugging in this class.

Conclusion

We postulate that programming abstractions should be designed in a way that allows debugging at the same conceptual level as programming. The goal should be that developers can work on the abstract level without having to think about lower-level implementation issues. In the case of debugging pervasive applications, though, this goal is hardly achieved as there are only a few good examples such as YETI or MDB.

We believe that the reason for this is that augmenting programming abstractions with debugging capabilities is not merely an engineering issue. Research is required to devise fundamental debugging concepts for declarative and distributed languages. Known source-level debugging concepts such as single-stepping or breakpoints are neither applicable to declarative languages nor to distributed settings. Once we have devised such concepts, we can investigate the implications on the design of debuggable programming abstractions.

Meta

• This is a position paper, published at the The International Workshop on Programming Methods for Mobile and Pervasive Systems 2010
• This is joint work with Kay Römer.
• This work has been partially supported by the National Competence Center in Research on Mobile Information and Communication Systems (NCCR-MICS), a center supported by the Swiss National Science Foundation under grant number 5005-67322, and by CONET, the Cooperating Objects Network of Excellence, funded by the European Commission under FP7 with contract number FP7-2007-2-224053.
• .pdf, BibTex

In order to have representative results I should compile for a platform which is typical for the WSN community. I have chosen the MSP430 because there is a package of a GCC cross compiler for that target in the TinyOS Ubuntu repository.

At first, the numbers looked promising. But at a closer look something was wrong. Investigating the problem emerges something weird.

The function ec_sub6_collect_run from the generated application is defined as:

It is merely an artefact of the code generation and will most probably be target of optimization soon. Nevertheless the generated assembler code should not harm too much. But here is the surprise. In the assembler code this function consists of 196 bytes of code! How can this be?

Disassembling the binary reveals this:

WTF?!? The second command is a ret. This ends the function. There is no jump before that. So everything below is unreachable code. And anyways, it does not seem to do something meaningful. As long as I haven’t completely misunderstood something, I think this is a bug.

If I would not have had a closer look I would have trusted the output of the tools and used it as an argument for or against my hypothesis. Because, as we all know, you can trust your tools. Ouch!

Well, could be a known issue and maybe its even fixed. So I checked the compiler version. Turns out, it’s GCC version 3.2.3! According to the GCC release documentation this release is from October 2003. It is over six years old. Double-checking other distribution channels makes clear, that this is really what the TinyOS community uses.

I mean, maybe we should use recent tools before we complain that deploying motes is hard. Maybe it’s due to compiler bugs that motes tend to inexplicably reboot now and then.

How can I measure the properties of the generated code now anyway? Do I really have to build a cross compiler from source? So much for unexpected problems…

In my opinion, parsing expression grammars are the tool of choice for growing a grammars. Among the existing PEG tools, the Rats! parser generator of the eXTensible C framework offers the most. Besides many handy features there is a complete C grammar and a pretty printer. Ready to take off!

Because my compiler does more than simple search-and-replace transformations, I won’t get along with xform, which is a domain specific language for AST transformations of the xtc framework. Instead, I think I will need full turing-completeness. So I just want to use the AST from the xtc parser and plug my transformation engine there. For the implemenation of the latter I have chosen Scala, because it integrates well with Java and offers many features like functional programming and pattern matching, which should serve well for compiler tasks.

Parsers generated by Rats! produce ASTs which are composed of GNodes. A GNode basically is a tuple of a name and a list of children, which often are GNodes again and sometimes are Strings or other leafe objects. A compiler which works on such an AST must repeatedly compare strings and address child nodes by their index in the children’s list. This makes the compiler’s code hard to write, read and maintain. Furthermore the toolchain is unable to assist me as it has no information about what I am doing.

In principle, though, Scala could help me as it is a strongly typed programming language. In order to take full advantage of this I need to transform the GNode-based AST into Scala objects. Every name of a GNode corresponds to a case class with the types and the names of the members set according to the children’s list of that node. In other words, I specify the meta-model of the C AST in terms of Scala types. Then I can use Scala’s pattern matching to find nodes of the AST, use functional programming to do the transformation and use the Scala AST classes to construct the result. And for all those tasks the Scala compiler will assit me with type checking. Sound’s good.

But here is the the problem: There are approx 140 different names of GNodes in the C AST of xtc. Writing all corresponding Scala classes plus the transformation code for both directions is a cumbersome and time-consuming way. Writing tests for all possible cases is even worse. And maintaining all this code for later modifications of the source language’s grammar is hell.

Instead, I would like to generate that code. What I need therefore is a machine-readable form of the meta-model, which basically is specified by the Rats! grammar of the language. But there are a number of Rats! rules such as lifting and desugaring, which make the meta-model of the resulting AST slightly different from the grammar.

But fortunatelly xtc includes Typical, “a language and compiler for specification and generation of type checkers”. Among other things Rats! support to dump the meta-model of any grammar in Typical notation, and there is a Rats! grammar for the Typical format. So, I could use this chain of tools and plug in a code generator at the end, which produces the meta-model in Scala type system notation.

At a closer look, though, some information is missing in the Typical format, which in particular is the name of child nodes if they are not GNodes. Without that information the generated Scala classes would end up with names such as string1, string2, etc, instead of structureTag or alike. This is not very nice.

That’s why I though about introducing a backend next to Typical which is capable of dumping the meta-model as a UML class diagram in the XMI format. Unfortunatelly xtc seems to be not prepared for such extensions. The code which does the printing in Typical format is part of the AST class, which is the father of the JavaAST class. An instance of the latter is created by the Rats class and passed to a TreeTyper instance, which invokes the printing. So, it would need some hacking to integrate an alternative dump format into this code.

But the real show stopper is something else. Rats! parser have no problem with producing two GNodes with the same name but with different types of children.

$ cat example.rats
module example;

public generic Foo = Bar \ Baz;
generic Bar = "bar";
generic Baz = "baz";

$ java xtc.parser.Rats -ast -variant example.rats
[...]
mltype foo = Foo of [
    | `SomeBar of bar
    | `SomeBaz of baz
  ] ;
mltype bar = Bar of string ;
mltype baz = Baz of string ;

I don’t fully understand the Typical format, but as far as I can see this means, that the child of a Foo is either a Bar or a Baz, but there is no common base class of them. Looking into the generated parser code confirms this:

$ java xtc.parser.Rats example.rats
$ cat example.java
[...]
  public Result pFoo(final int yyStart) throws IOException {
  [...]
    // Alternative 1.
    yyResult = pBar(yyStart);
  [...]
      Node v$g$1 = yyResult.semanticValue();
      yyValue = GNode.create("Foo", v$g$1);
  [...]
    // Alternative 2.
    yyResult = pBaz(yyStart);
  [...]
      Node v$g$2 = yyResult.semanticValue();
      yyValue = GNode.create("Foo", v$g$2);
[...]

So it seems to me, that Rats! and static type systems do not fit together in general. But there might be a way out.

$ cat example2.rats
module example2;

public generic Foo = Bar @Bar / Baz @Baz;
generic Bar = "bar";
generic Baz = "baz";

$ java xtc.parser.Rats -ast -variant example2.rats
[...]
mltype foo =
  | Bar of foo
  | Baz of foo ;

By using node markers we can force the names of the generated GNodes to a specific value. For the meta-model this means that Foo is an abstract class and Bar and Baz inherit from it.

So I will have to find out, how I can rewrite the C grammar in a way that the language is not changed but the resulting meta-model can be statically typed. And then I will have to hack an XMI backend and finally write the code generator for the Scala classes. So much for unexpected problems...