System architects design distributed applications leveraging computer resources connected by a network infrastructure: CPU-intensive tasks may be located on powerful core servers; application-data storage may be managed by specialized database servers; and data-collection, actuation, and user interaction may occur on nodes located at the edge of the network. The architecture must accommodate coordination of the distributed resources as well as ensuring that the necessary data is available to all the computation, decision- making, and visualization nodes.

Photo 1: The U.S. Navy’s LPD-17 vessel uses the Ship-Wide Area Network (SWAN), which is responsible for shipboard control, machinery control, damage control, integrated condition assessment, steering control, advanced degaussing, mission control systems, navigation systems, and communication systems.

The problem is that distribution of all the data and stateful coordination between distributed nodes is complex and technically difficult. Much of the distributed infrastructure complexity stems from designs built around the processing endpoints.

In a distributed, embedded application, an architect would typically focus on two separate areas: First, data acquisition endpoints — the devices that interface with automated processes and continually capture data from those processes; and second, the computation within the endpoints or on separate nodes within the network — the applications that process and react to data in real time.

For example, a shipboard system has a variety of embedded devices that perform many discrete tasks including positioning, navigation, engine control, radar, targeting, command and control, and fire control, among others. Many of these devices and their supporting systems produce data that would be essential to the successful operation of others, or at the very least would significantly enhance their ability to produce quality results. These systems are indubitably connected through a network — often standards based — so data communication between devices is common and continuous.

Designing Around Processing Endpoints

The “traditional” way to ensure that these disparate devices and systems are able to leverage data from one another across a network is to examine the data requirements of each device, and draw one-to-one interfaces between those devices. Once the relationship has been established, the data can be passed between those two devices, likely using direct messaging or a transactional client-server call.

The processing-centric approach requires that both the data-producer and data-consumer know both the existence and location of one another, and use identical data structures and/or user-defined network protocol to exchange the data item.

This approach works, and can be efficient under many circumstances. However, this design also assumes that the network and application-architecture is static and unchanging. If an acquisition radar unit is swapped out for an upgraded model, the targeting computer code has to be adapted to recognize and work with the new hardware. If the radar’s data is in a different format, further code changes and additional testing is required.

That is the case if only two devices are involved. In practice, the radar data might be used by half a dozen or more shipboard systems. Further, it is possible that devices are co-dependent on one another and have embedded stateful semantics in the data they are exchanging. For example, the radar and targeting computers may use data from one another to successively refine initial estimates, or to locate and successively target multiple targets.

Given this complex yet deceivingly straightforward design approach, maintaining these distributed systems throughout their lifecycle can be a significant technical, logistical, and financial challenge. Every upgrade and system modification would, at a minimum, require extensive testing to ensure that changes did not introduce incompatibilities and that system interactions remain consistent. Code changes will be likely.

Information-Centric Approach

Instead of a design based on the processing endpoints and the resultant producers or consumers of data, it is often easier to begin the application design process from an information-centric or data-centric perspective.

The information-centric approach first defines an information model that captures the essential state and events in the system, and then develops components that can produce and process that information. In other words, rather than looking at the processing devices first and deriving specific data requirements for each processing device, it approaches the design from the standpoint of what data is generated through acquisition or processing. If the data can be useful to any other process, it can be made available without knowing where, when, or how it might be used.

Photo 2. The Aegis combat system is a sea-based system intended to intercept short- to medium-range missiles. Subsystems such as shipboard navigation are integrated into the combat system with the radar, weapons, and tactical displays.

This emerging alternative for asynchronous, decoupled data transfers is publish-subscribe, often shortened to “pub-sub.” In this model, data sources or producers simply publish data on the network. This could be a memory-to-memory transfer if high performance is required, it could be a database or other persistent storage, or it could be a direct message on a network.

Those processes with a need for that data can subscribe to it through the publish-subscribe service. When data is updated and published, data arrives at the subscribers over the appropriate physical medium. Subscribers simply obtain the data, and use it in their own processing. The resulting system is decoupled in location (a device can be anywhere in the network), time (stateful data interaction handled in the middleware not special application logic), function (multiple sources or sinks of data can be easily added and removed), and protocol (each consumers can managed their own conditions for data delivery).

A publish-subscribe model for data-distribution enables the implementation of a data-centric architecture across a large-scale network. Using a shipboard example, the GPS receiver can publish position data to any system that needs it. Navigation computers and targeting systems can all subscribe to the GPS location data.

There is no specific need for commonality of data formats in a publish-subscribe model. Because the source and destination of a given data item is unknown, any required data conversion could occur at the handoff. In fact, data consumers may well have different data format requirements, necessitating individual conversions based on those needs. Any required data conversions can be managed by the middleware that provides the publish-subscribe services.

Further, because ship position continually changes in real time, there would be multiple data points that describe all the position updates. In some applications, there is the need to persist such data for later use (an application joins the network and needs to know where the ship was one hour ago). For most applications in the real-time network, data points can be consumed as quickly as produced and only the current value has any value. Using the data “how you want it” necessitates decoupling; how you use the data from how it is sent, which is an intrinsic property of the information-centric approach, eliminating the need for special protocol intelligence at each device endpoint.

But the real advantage of the publish-subscribe approach is that devices can be upgraded or added without the need to change code and re-test the entire resulting configuration exhaustively. If new data is available on the network, other devices may require additional code in order to make use of that data, but in practice this is significantly simpler than modifying and testing a large number of specific one-to-one connections.

This characteristic also makes distributed applications highly scalable. Because there are no fragile one-to-one data connections and because devices can be added with little or no change to underlying code, expanding the application to include a larger network with more endpoints is straightforward.

Designing with Data-Centric Principles

Data-centricity provides a general guide for how to design distributed applications. Many distributed-application system architects today use procedural or object-oriented principles in creating the fundamental design, often using UML sequence, class, and state diagrams.

These design methodologies tend to treat transporting and consuming data as second-class citizens, focusing instead on the step-by-step processes by which devices make computations and produce actionable results. A data-oriented methodology, on the other hand, focuses on the flow of data through the application.

Taking a closer look at the alternative design/development methodologies only highlights the issues. Procedure-oriented approaches focus on the computational aspects of the application, and concentrate on the device endpoints of the network where almost all processing occurs. Data is structured in order to assist with the computation, rather than making the data available to begin with. Once the computational processes were designed, the problem of getting data to those processes remained. However, this makes data movement almost an afterthought, and problems of timeliness and data formats become more difficult to address. Ultimately, the focus on computation means that the endpoint devices must have functions and data that are known and accounted for by one another.

Object-oriented methodologies base designs on the definition of objects and their interactions. Because objects are defined by their computational processes (methods) as well as data, there appears to be a good balance between code and data in this approach. However, object-oriented methodologies presume that data is an inherent part of the object. The data formats are fixed within the object, and the methods act upon the data only within the context of the object. There is no inherent provision for exchanging data between objects, and for adapting data to the formats required for various processes.

The overriding characteristic of these common design methodologies is that they envision a distributed application as a set of processes, designed as procedures or objects. Data is assumed to be contained within each of the processes. Getting data to and from a particular process was a problem to be addressed after all of the processes were defined.

Alternatively, a data-centric methodology provides a much more natural and streamlined way of viewing and modeling many distributed applications. Such a methodology focuses on how the data that is moving and transforming in the system, rather than the processes that are performing those actions.

In other words, the processes encapsulated in the endpoints become secondary. The flow of data defines the essential aspects of the application, and can be modeled with UML class and interaction diagrams. That is not to say that the computational processes are not important, but rather that they are not essential at a high level of design in understanding and mapping out the application. In real-time systems, getting the data to the process that requires it, when it requires it, is essential.

Examples of Data-Centric Applications

There are many real-world applications that are driving the data-centric design evolution. These systems, through necessity of maintainability, flexibility, upgradeability, and affordability, are moving towards a data-centric architecture. The architecture is being realized using COTS middleware and open publish-subscribe data models.

Photo 1 shows a ship in the U.S. Navy designated LPD-17, the first of the San Antonio class of Navy vessels. Real-time messaging middleware underlies a system on this ship class called the Ship- Wide Area Network (SWAN). The SWAN is responsible for shipboard control, machinery control, damage control, integrated condition assessment, steering control, advanced degaussing, mission control systems, navigation systems, and communication systems.

The SWAN is absolutely critical to the ship’s operation and achieving its mission. It comprises hundreds of computers. Because this ship may find itself in the middle of a conflict, it must also be able to take a hit anywhere and continue operations. This required the shipbuilders to look for middleware that supports features like automatic discovery so there would be no configuration. It supports redundant data sources, sinks, data paths, and transports. In fact, you can physically cut the ship network in half and the two halves will work independently. If you reconnect them, they will automatically rediscover and heal the system.

Combat systems are the most demanding real-time systems. Advanced radar systems must handle thousands of simultaneous “tracks” — continuously updated positions of aircraft, ships, vehicles, and missiles in an area. The thousands of radar tracks generate a lot of data, and it must be delivered on time. Many different applications, from display to track managers, need the real-time data. Photo 2 shows an Aegis cruiser launching a missile. The Aegis combat system integrates subsystems like the shipboard navigation system and the radar, weapon system, and tactical displays. A change to any results in costly changes to all. Data-centric approaches are being used to decouple these subsystems, not necessarily improving system effectiveness, but rather making the system easier to build, maintain, and evolve.

Implementing Data-Centric Applications

Data-centric applications can be implemented using the precepts of data-oriented programming. Some of the concepts behind data-oriented programming are familiar to software engineers using more common approaches, but some deserve further explanation.

In general, the tenets of data-oriented programming include the following principles:

  • Expose the data. Ensure that the data is visible throughout the entire system. Hiding the data makes it difficult for new processing endpoints to identify data needs and gain access to that data.
  • Hide the code. Conversely, there is no reason for any of the computational endpoints to be cognizant of one another’s code. By abstracting away from the code, data is free to be used by any process, no matter where it was generated. This provides for data to be shared across the distributed application, and for the application to be modified and enhanced during its lifecycle.
  • Separate data and code into data-handling and data-processing components. Data handling is required because of differing data formats, persistence, and timeliness, and is likely to change during the application lifecycle. Conversely, data processing requirements are likely to remain much more stable. By separating the two, the application becomes easier to maintain and modify over time.
  • Generate code from process interfaces. Interfaces define the data inputs and outputs of a given process. Having well-defined inputs and outputs makes it possible to understand and automate the implementation of the data processing code.
  • All code is loosely coupled. With well defined interfaces and computational processes abstracted away from one another, devices and their computation can be interchanged with little or no impact on the distributed application as a whole.

This approach to application design is effective in systems where multiple data sources are required for successful completion of the computing activity, but those data sources reside in separate nodes on a network in a net-centric application infrastructure. For network-centric distributed applications, applying a data-oriented programming model lets you focus on the movement of data through the network, an easier and more natural way of abstracting and implementing the solution.

Data-centric design, and data-oriented implementation, can bring about a more robust and scalable distributed system, and one that is easier to maintain and enhance over time. For real-time distributed applications that are highly dependent upon the movement of data through the system, the advantages of using data as the design and implementation focal points can make the difference for a successful project.

This article was written by Dr. Gerardo Pardo-Castellote, CTO, and Gordon A. Hunt, Chief Engineer, at Real-Time Innovations, Inc., Santa Clara, CA. For more information, Click Here 


Defense Tech Briefs Magazine

This article first appeared in the December, 2007 issue of Defense Tech Briefs Magazine.

Read more articles from the archives here.