MUFASA: Multidimensional Framework for Advanced SESAR Automation.
Lochkeed Martin UK
Over the years, several broad concerns have been raised about over-reliance on automation in such complex human-machine systems as Air Traffic Management (ATM). Automation, it is feared, can lead to a host of human performance problems, including “out-of-the-loop” situation awareness and vigilance problems, transient workload peaks, loss of skills, difficulties in reassuming manual control, and decreased job satisfaction. Such broad objections, and the equally general mantra that we must develop “human-centred automation,” however, are not very prescriptive and in the end do little to further what should be a serious discussion. Given the inevitable trend toward greater, and more sophisticated, automation in ATM, it is essential to move the argument beyond such generalities.
The Multidimensional Framework for Advanced SESAR Automation (MUFASA) project aims to develop a framework for designing future levels of ATM automation, based primarily on human-in-the-loop simulation. Building on the team’s research into innovative space- and time-based CD&R displays, it aims to experimentally manipulate automation level, traffic complexity and, most notably, “heuristic conformity” or the fit between human and automation solutions to medium term planning and separation conflicts. This will help refine a framework addressing potential interactions and tradeoffs between these dimensions in terms of both human performance (workload, situation awareness) and system performance. Notably, it also permits a way to quantify and empirically define “automation bias.” Results aim to extend current automation design principles, but also inform the design of advanced ATM automation for the mid-and far terms. Some of the notable characteristics of the MUFASA project are the following:
• It systematically evaluates algorithmic and heuristic approaches to CD&R automation.
• It is perhaps one of the first times that research has tried to empirically define and quantify user trust and acceptance in automation: as willingness to accept automated advice that fits with one’s own preferred way of working (because that advice is in fact an unrecognisable replay of one’s own solution).
• It brings together a consortium that offers renowned research capabilities and facilities, ANSP operational expertise and partnership with industry.
• It combines ongoing and complementary work into both optimised "machine centred" and heuristic "human centred" approaches to ATM display and automation technology.
• The project schedule calls for academic laboratory simulations to scale up for replication with the consortium’s ANSP partner.
• The project would extend current state-of-the-art with respect to automation design principles.
The stated goal of advanced ATM automation is to modulate controller workload and help accommodate higher traffic loads. However workload benefits might not be enough. As automation becomes more advanced and assumes more of the “thinking” of the controller, issues of controller acceptance can become more critical. Further, advanced automation would often take the form of machine generated advisories. If controllers failed to accept such advisories, they would be more likely to disregard or override automated assistance. Previous research has demonstrated that operators are sometimes simply reluctant, on general principal, to embrace new automation. Some past attempts to introduce medium term advisory automation have failed specifically because controllers would not accept fully autonomous CD&R. If we are to follow a human-centred automation design approach, this suggests that design will have to address not only whether automation works “properly,” but also whether it works in a way that the controller accepts.
To get at these different aspects of design, MUFASA aims to refine a multidimensional automation framework consisting of
• Levels of automation,
• Complexity of the problem controllers face, i.e. air traffic complexity, and
• Conformity of the solution presented by automation, i.e. the fit between human and machine
This 3D view of automation, as shown below, can describe a static automation design but also a dynamic adaptation of automation in step with changing conditions, for instance as complexity increases the automation approach should likely be adapted.
A multi-dimensional view of automation
Level of Automation
A number of broadly accepted taxonomies of human-machine system control have categorised levels of automation, from fully manual to fully automatic, according to level of autonomy. For instance, does automation propose but defer? Does it implement but accept override? Or does it simply implement by fiat? The foundation for the framework is the concept of automation levels. Such levels refer to the degree of authority and autonomy delegated to automation.
The move toward trajectory-based air traffic operations should dampen much of the need for tactical intervention, and shift the focus of problem solving (and advanced automation) upstream to the planner (and eventually the MSP). The concern here is that the job of the planner/MSP is made more difficult by the redistribution of traffic complexity.
Air traffic complexity is generally defined as the static and dynamic characteristics of the airspace and traffic that contribute to controller workload. Traditionally, traffic density has been the single factor most associated with complexity. It is increasingly clear however that traffic density by itself does not capture the richness of what controllers find complex and what ultimately drives their workload. Past attempts to assess complexity have generally relied on geometric relationships between aircraft, or on observable physical activity. A recent EUROCONTROL literature review identified over 100 different complexity metrics that have been proposed and/or evaluated. Some of the main factors include: Traffic load; Mix of climbing and descending traffic; Traffic flows converging at the same point; and others.
As automation evolves in step with SESAR’s Service Levels, it is likely to act more as an advisor, providing solutions to the controller (regarding, for instance, airspace reconfiguration, planning conflicts and medium-term traffic deconfliction). That is, automation will be taking over a greater role in the medium-term timeframe between the tactical controller and CFM time horizons. This would be the case for complexity management tools, and those used by the MSP. One potential human performance problem is that advisory automation that provides an optimised (e.g. single vector) solution can conceal its “reasoning” and introduce and paradoxically present a solution that the controller cannot easily evaluate. A technology-centred approach says that this is not a problem, that if the solution is optimal it should be accepted and the controller should manage only by exception.
This could be the case. After all, most of us live every day with autonomous and optimised automation. One example is the automation involved in passenger lifts, which through fuzzy logic learn to optimise their behaviour (i.e., where to pre-position themselves). As long as input-output relationships are reasonably clear, the user does not have to wonder “what is the automation doing now?!” However (and disregarding display solutions) as the ATM planning horizon expands it becomes harder for the controller to evaluate a single vector solution. This invites the question of whether controllers will [a] accept such solutions and [b] be able to evaluate them in any event. With careful design and simulation set up, the MUFASA team intends to test this notion using human-in-the-loop simulation that experimentally manipulates the fit between human and machine solution. This will allow the team to empirically test the potential for automation bias to impact acceptance and performance with automated CD&R.
We call this dimension “heuristic conformity,” or how well automated solutions fit with the controller’s own way of working. A mathematically “optimised” solution can (but need not) differ from a controller’s own, which may rely on rough-approximation, “good enough” solutions. If the automation and its interface complicate the evaluation of optimised solutions, human performance can suffer.
Notice that this multi-dimensional framework starts from the common levels-of-automation taxonomy but also incorporates dimensions relating to the problem space (complexity of the problem) and the solution space (conformity of the solution). As suggested in a hypothetical 3D contour graph (below), there is likely a non-linear trade-off between the three dimensions. Further, there might be a dissociation between such tradeoffs in terms of human and system performance. In fact, each of the various human and system performance measures might actually show a different contour graph.
Hypothetical tradeoffs between LOA, Complexity and Conformity.
MUFASA aims to conduct two human-in-the-loop simulations built around the Technical University of Delft’s prototype Solution Space Display (SSD), which provides air traffic controllers a visualisation of conflict-free velocity vectors that yield GO and NO-GO regions. In essence, this SSD approach subscribes to a middle level of automation in which the choice of resolution manoeuvres is still left to the controller. That is, the automation is used to collect and process the available data in order to visualize the “solution space” that the air traffic situation affords.
MUFASA aims to extend the capabilities and application of TUD's ATM simulation and display work, in two major ways. First the project will modify the capabilities to incorporate a higher level of command authority that not only suggests a resolution space but suggests / dictates a resolution vector. According to accepted automation taxonomies, this would shift automation from assisted manual control to either management by consent (i.e. the controller must concur) or management by exception (i.e. the controller must veto). Notice that this extension of the TUD capability with automated solutions provides the opportunity to systematically evaluate and compare algorithmic and heuristic approaches. That is we can by modifying the simulation capabilities specifically address the issue of heuristic conformity, and whether automated solutions fit with those of the human.
Realtime simulations will experimentally manipulate both the source of strategic resolutions (human vs machine) but also the degree to which those resolutions fit with the human’s own solution (conformal vs non-conformal solutions).
MUFASA will systematically evaluate human-machine system performance under (non) conformal automation, that is when automation-generated solutions (do not) match those of the human. The overriding research question is how human acceptance of strategic advisories differs by the presumed source of that advice. Does acceptance of automated solutions differ when a controller thought that the machine had provided a solution? This represents perhaps one of the first times that research has tried to empirically define and quantify user trust in automation: as a willingness to accept automated advice that fits perfectly with one’s own preferred way of working (because it is in fact an unrecognisable replay of one’s own solution).
Solution Space Display (SSD), a prototype approach to CD&R
Examples of the research questions to be addressed over the course of the two real-time simulations include the following:
• Can higher levels of automation benefit system / human performance over baseline SSD?
• Is there a trade-off by complexity level?
• Does willingness to consent / veto differ by complexity level?
• How do automation (algorithmic) solutions differ from baseline (human)?
• Do controller resolutions differ from those of automation?
• Do controllers consent to automated solutions?
• Does consent vary by conformity (i.e. whether automated solutions differ from their own)?
• Do controllers reject their own resolutions when they mistakenly believe these come from automation?
Based on a combination of empirical and analytic approaches, MUFASA aims to develop a multi-dimensional framework of advanced ATM automation that can guide far term automation development but also help drive mid term design. The project marries the complementary theoretical / basic research expertise of academia, with the pragmatic operational demands of air traffic management. Contact with industry is to be maintained throughout the project, which aims to focus on tangible real-world results that can be shared at a knowledge dissemination workshop. Among the transition issues to be addressed will be acceptance of automation, potential impediments (and solutions) to deployment, personnel implications (e.g. selection and training) and role redefinition.