SPAD: System Performances under Automation Degradation.
Deep Blue (IT)
Association pour la Recherche et le Developpement des Methodes et Processus Industriels (FR)
Université Paul Sabatier (FR)
Increase of automation is one of the main consequences of the changes foreseen by SESAR in ATM. bringing a range of new challenges including those related to possible degradations. We will need to be in condition to understand, monitor and manage the propagation of automation degradation of a single system to the overall ATM system, and to confine and absorb degradation problems, both with and without human contribution. We will also need to understand and estimate the implications of degradations for the overall ATM system performances. These aspects will be investigated by SPAD that aims at:
- understanding, modelling and estimating the propagation of automation degradation in ATM;
- estimating the consequences of automation degradation on ATM performances;
- supporting an effective intervention for the containment of automation degradation.
In the future of Air Traffic Management there is an increase of traffic demand and new business challenges that will bring the current ATM system to its capacity limits within the 2013-2015. As a consequence an overall productivity improvement is urgently needed and the paths to this have been settled by SESAR in the ATM Target concept. Common denominator of the key enablers, forecasted by SESAR, is an increase of automation that will support and in some long term case even completely replace human tasks, in order to meet the new capacity and efficiency necessities. Human operators will be able in this way to manage a higher number of tasks and will shift their roles toward more strategic ones. Some of the keys to the future Concept for the ATM system are a drastic reduction of controllers’ task load trough increased automation in conflict detection and resolution, as well as higher levels of automation for data gathering and management.
High levels of automation imply low system flexibility, key feature necessary to deal with non-standard or unexpected events. This is also related with the so called “Planning vs. Flexibility paradox”. Indeed a system which has been carefully planned, and thus standardised and automated, is hardly able to face up with non-standard and unplanned events such as technical failures. This may bring to a higher sensitivity of the ATM system to degradation problems.
There is a need to understand how automation degradation will propagate in the SESAR scenario, where the level of interconnections will significantly increase. The increasing coupling of the ATM systems makes harder to identify and isolate failures when they occur, and to detect minor malfunctions before they propagate to the whole system. There have been many studies focused on automation and automation classification. A lot has been done in order to establish automation taxonomy, but little is known about automation degradation, how it propagates in a complex system and about the link between degradation and system performances. This knowledge gap has to be filled to cope with the challenges in the SESAR future of ATM
Project Objectives and expected results
The aims of the project are:
- To understand, model and estimate the propagation of automation degradation in ATM, and to evaluate the associated consequences on ATM performances.
- To validate the above results on a large ATM system with high degree of automation.
- To develop a demonstrative prototype for monitoring degradation and estimating its propagation and the related reduction of performances in a large ATM system with high degree of automation.
The concrete outcome of the project will be:
- A federation of models, able to describe automation degradation propagation and its effects on ATM performances. In terms of concrete application we will deliver instantiations of the federation of models, for each of the systems studied in the project.
- A validation report with the evaluation of the performances of the federation of models and recommendations for further improvements useful for researchers investigating the same subject.
A prototype for monitoring automation degradation. Using the predictive ability of the model federation. The prototype will be able to estimate degradation propagation and its effect on ATM performances. In addition, by using performance indicators, that relate to system functions, the model federation embedded in the prototype will enable the identification of possible interventions, either to sustain performance at an appropriate level or to ensure a graceful degradation.
We consider ATM as a Large Scale Socio Technical System, a system composed of systems, that combine their resources and capabilities in order to achieve a common goal. Modelling of systems of systems is particularly complex due to a set of reasons that include: the need to consider multiple levels and domains; the overall complexity and the variety of component systems; the level of uncertainty that remains in their behaviour and interactions.
An emerging approach to study such systems has been to combine models offering different perspectives of the system under study and analysing it at different levels of granularity. Combination of existing models for the analysis of systems of systems is especially promising when models have sufficient capability for the questions of interest. This approach is particularly relevant for us since we don't need a full homomorphic abstraction of the ATM system. We intend to study the propagation of automation degradation and the related influences on performances, thus our interest will be limited to only a few aspects of the ATM system. At the level of the single component system, where the degradation starts, we will be interested to the core and critical functions of the system.
At the integration level, when we consider this system in the context of being integrated with other systems, our interest shift to the interactions between the systems and the link with the overall performances. For this reason we do not intend to develop a large scale stand-alone model but rather to focus on a limited number of essential specific aspects of the systems, using upgraded versions of existing models combined in federation.
Each model will focus on a specific characteristic (e.g. functional aspects, interactions and propagation, human behaviour and his interaction with the system) and represent a part of the whole ATM system with variable levels of granularity (from coarse to fine grain) depending on the interest of the analysis. An example of the areas investigated by different models is shown in Figure 1.
Figure 1 - Possible federation
Federating Models - We plan to adopt a federation of models to guarantee that interaction between models have shared meaning at the conceptual and technical level. In particular, our federation will address:
- information exchange mechanisms to ensure the capability to exchange information between federated models during the analysis;
- compatibility of the entity representations to ensure federated models will have meaningful and compatible exchange of information about the entities;
- environmental representation to ensure federated models will reference a shared and correlated environment.
To achieve the first objective of our project, the federation of models shall be able to work at different levels of abstractions from the single system till the top system of systems level. At the system level we need an articulated model of what is required for the system to carry out its operations, to monitor and measure automation degradation and its containment including the possible contribution of humans to resilience. When considering this system in integration with other systems we need a model of interaction and coupling between the different systems, to understand and measure degradation propagation and the link with the overall performances. Objectives of the development of the Federation of Models are to provide a framework allowing the modelling of Large Scale Socio Technical Systems performance variability under different conditions, with different levels of granularity. This Federation of Models consists in integrating FRAM method with HAMSTERS and ICO notations and tools (Figure 2). The integration of FRAM, HAMSTERS and ICO leverages the high-level view on complex socio-technical systems provided by FRAM with the fine-grain view on human-system interaction provided by HAMSTERS and ICO. The main contribution is to associate performance variability analysis phase of the FRAM method with quantitative user and system performances evaluation support from HAMSTERS and ICO.
Figure 2 - Federation of Models within a Models-Based process to assess LSSTS performance variability
Adapting the Models with the Case Studies - Models are based on abstractions, idealization, and assumptions. In order to get trustworthy results from the models these shall be adjusted to the reference system. We plan to use scenarios from two reference systems having different levels of automation and different implementation perspectives. These systems, i.e. our case studies, are:
- An Arrival Manager (AMAN), that is a ground based planning tool, suggesting to the air traffic controller an optimal arrival sequence of aircraft and providing support in establishing the optimal aircraft approach routes.
- An Unmanned Aerial System (UAS) for automated self separation of Unmanned Aerial Vehicles (UAV), ensuring that each UAV can reach the desired destination with the optimal route without conflict with other aircraft or UAV and with no human intervention.
For each of these two case studies, this document provides a reference scenario plus three scenarios of growing degradation severity. The reference scenario, called Nominal scenario, describes the implementation of the system and its operational context. In that scenario, the system works properly, e.g. without failures or malfunctions. The other scenarios describe three defective functioning of the systems that generate three possible levels of degradation: confined, average and extended degradation. As for the AMAN Case Study, the three system’s defective functioning are a temporary interruption of the service, a permanent failure of the service, and the system providing wrong (and thus misleading) information to the controllers. As for the UAV Case Study, the malfunctions are the interruption of the communication channel between vehicle and remote pilot, the interruption of the communication channel plus a failure in the vehicle’s self-destroy system, and the interruption of the communication channel plus a failure in the self-destroy system and ADS-B system. ATM performances in the nominal scenario and in the degraded scenarios will be compared to assess the effects and the impact of the different levels of degradation. These scenarios will be used exploited internally in the project to provide references for the modelling activity. One of the objectives of this modelling activity is to use the models to assess the evolution of performance of the ATM system under consideration when facing automation degradation.
Scenarios will be reviewed and refined in collaboration with operational experts. Applying the models in federation on these scenarios we will evaluate their ability to model adequately the situation, adjust their performances, integrate them with other models if necessary, and calibrate them. The first two case studies keep the human operator in the loop (and sometimes over the loop) and will thus be used to model the potential human role in the containment of the degradation propagation.
Validating Models - We will be in condition to animate the Unmanned Aerial System (the second of the case studies) through an integrated set of simulators. These will be able to simulate and represent: the behaviour of each UAV; their separation algorithms; their communication and localisation devices. Simulators will also consider and show real traffic using an ADS-B ground station, and merge within it the simulated UAV. Using these simulators we will be in condition to reproduce a significant portion of airspace with both UAV and piloted aircrafts. UAV will operate in complete automation, with human intervening only in case of system degradation to activate predefined containment and recovery strategies. The initial version of these simulators will be provided by the European project ARCA in the framework of a scientific collaboration between SPAD and ARCA. Simulators will be adapted to be used as test-bed for the federation of models, developed in the first part of the project.
Through evaluation runs at different degrees of degradation severity we will evaluate the following abilities of the Federation:
- Ability to describe properly the degradation propagation
- Ability to take into account containment and recovery strategies (limited to a pre-defined set only)
- Ability to estimate the degradation effect on the overall system operational performances (limited to capacity and flexibility)
A validation report will contain the results of the validation together with recommendations and all the practical indication to refine the Federation and the constituent models on the basis of the simulation outcome and of the related analysis.
Developing the simulator - The set of simulators produced in ARCA will be adapted to our project and completed with a tool to monitor and measure degradation and estimate its propagation and reduction of performances in the UAS system. The tool will also identify opportunities for effective countermeasures and responses and early warnings, which can be used as a basis for possible reconfigurations.
This tool will exploit the prediction ability of our federation of models. The federation of models will be executed having as input data from the ARCA simulators regarding the operational conditions. When a degradation will occur the models will provide information about its possible evolution and propagation, and estimation of the influences this degradation will have on system performances. The functional modeling will also serve as the basis for identifying opportunities for possible countermeasures and responses, and early warnings to be used as basis for reconfiguration.
Since our models will be either relatively simple because related to single systems (e.g. Tropos), or based on a few simple principles and recursive (e.g. FRAM), it will be quite easy to implement them by means of software. The architecture of the tool integrated with the ARCA simulators is shown in Figure 3. This tool will represent the implementation in a prototype of the SPAD approach, demonstrating how it could be used in real systems for monitoring and estimating degradation propagation and reduction of performances and to facilitate an effective intervention in the degradation lifecycle.
Figure 3 - Architecture describing the integration of SPAD tool with ARCA Simulators
Figure 4 - Screenshot of SPAD demonstrator
Major findings and results
The aim of SPAD is to support the understanding modeling and estimation of automation degradation and its consequences in ATM. Significant advancement has been achieved in this direction but with some limitations in the real time application. The approach adopted by SPAD is to use a Federation of Models tuned and adapted through a set of case studies. The Federation is based on a set of synergetic and complementary state of the art models for system analysis and evaluation. These models require a significant human contribution, and the interactions between the models shall be managed by a human analyst. The significant human (non automatable) effort required and the limited automated support by application tools make difficult to use the Federation for real time purposes.
To overcome this real time limitation, the Federation of Models has been used off line to explore in advance a limited number of possible future events, of the ATM system under study, and estimate their possible consequences. Realistic conditions for these events were generated using the project Simulator. The functioning of the system was then simulated and monitored in real time, and if there was evidence that one of the explored events was going to happen, the estimate about the possible consequences was used to manage the event. This strategy is very effective to monitor and manage possible degradations but is expensive in terms of application effort. The analysis becomes gradually less informative for each variation of the operational conditions of the event, and each variation requires additional human analysis.
The approach developed in SPAD is more cost effective to support the analysis of systems, for example as a support to safety assessment and safety analysis. In such a case, the Federation of Models can support the interaction between the analyst and the operational experts, and the representations and preliminary analysis of the Federation can be used to elicit the opinion of the operational experts in a structured and stimulating way. When the complexity of the system under analysis grows, the application effort can be significant because of the different instantiations required for each possible set of events to be investigated. In such a case, the analysis shall focus on the most relevant parts of the system and choose the right combination of levels of granularity for its parts.
The use of a realistic Simulator, generating data about possible normal and abnormal operational scenarios offered the opportunity to apply the Federation of models to different possible realistic cases. In this way the Federation of models was "tested in practice" and improved on the basis of the feedback gathered from its application. This use of the Simulator was key for achieving an effective and realistic validation of the project outcome and to allow an objective evaluation of the results achieved.
- [Approved] Deep Blue, University of Toulouse. D1.1 - Case Studies Scenarios. SPAD Deliverable 1.1, 15/11/2011
- [Approved] University of Toulouse. D5.1 - Joint Event Review. SPAD Deliverable 5.1, 01/12/2011
- [Approved] University of Toulouse. D5.2 - Communication and dissemination material. SPAD Deliverable 5.2, 01/12/2011
- [Approved] Deep Blue. D1.2 - Degradation lifecycle analysis. SPAD Deliverable 1.2, 30/03/2012
- [Approved] ARMINES. D2.1 - Federation specification. SPAD Deliverable 2.1, 18/06/2012
- [Approved] ARMINES. D2.2 - Federation of models. SPAD Deliverable 2.2, 23/04/2013
- [Approved] Deep Blue. D3.1 - Revised UAS simulator. SPAD Deliverable 3.1, 10/04/2013
- [Approved] Deep Blue. D3.2 - Validation report, 18/07/2013
- [Approved] Deep Blue. D4.1 - Functional specification of the Demonstrator, 02/05/2013
- [Submitted under approval] D4.2 - Prototype of the Demonstrator,12/01/2014 part 1 - part 2)
- [Approved] University of Toulouse. D5.3 - Joint Event Review. SPAD Deliverable 5.3, 11/12/2012
- [Approved] University of Toulouse. D5.4 - Dissemination and External Coordination. SPAD Deliverable 5.4, 14/12/2012, part 1, part 2
- [Approved] University of Toulouse. D5.5 - Joint Event Review 3. SPAD Deliverable D5.3, 13/12/2013
- [Approved] D5.6 - Dissemination and External Coordination (split in 3 parts due to size management policy of the website - part 1 - part 2 - part 3)
- Sara Silvagni, Martina Ragosta, Alberto Pasquini & Stefano Mastrangelo. SPAD Demonstrator 3rd SESAR Innovation Days, KTH Royal Institute of Technology in Stockholm, Sweden, 26th - 28th November 2013 (Poster)
- A. Pasquini, M. Ragosta, S. Silvagni, M. Sujan, E. Rigaud, E. Hollnagel. Modelling of Automation Degradation: a Case Study 3rd SESAR Innovation Days, KTH Royal Institute of Technology in Stockholm, Sweden, 26th - 28th November 2013 (Paper)
- Célia Martinie, Philippe Palanque, Alberto Pasquini, Martina Ragosta, Mark Alexander Sujan and David Navarre. Understanding functional resonance through a federation of models: preliminary findings of an avionics case study 32nd International Conference on Computer Safety, Reliability and Security (SAFECOMP 2013), September 24-27, 2013, Toulouse, France.
- Eric Rigaud, Erik Hollnagel, Célia Martinie, Philippe Palanque, Alberto Pasquini, Martina Ragosta, Sara Silvagni, Mark-Alexander Sujan. Modeling trade-offs consequence propagation and their impacts 5th Resilience Engineering Symposium 2013, June 25-27th, 2013, Soesterberg, The Netherlands
- David Navarre, Célia Martinie, Philippe Palanque, Alberto Pasquini and Martina Ragosta. Model-Based Dynamic Distribution of User Interfaces of Critical Interactive Systems 3rd International Conference on Application and Theory of Automation in Command and Control Systems (ATACCS 2013), Napoli, Italy, May 28-30, 2012.
- Eric Rigaud, Erik Hollnagel, Célia Martinie, Philippe Palanque, Alberto Pasquini, Martina Ragosta, Sara Silvagni, Mark-Alexander Sujan. A Framework for Modelling the Consequences of the Propagation of Automation Degradation: Application to Air Traffic Control Systems. SESAR Innovation Days, Braunschweig, Germany, 27/11/2011-29/11/2012.
- Célia Martinie, Philippe Palanque, Alberto Pasquini, Martina Ragosta, Eric Rigaud, Sara Silvagni. Using Complementary ModelS-Based Approaches for Representing and Analysing ATM Systems' Variability 2nd International Conference on Application and Theory of Automation in Command and Control Systems (ATACCS 2012), London, UK, May 28-31, 2012.
- Célia Martinie, Philippe Palanque, Martina Ragosta. SPAD Demonstrator Workshop on End-user Interactions with Intelligent and Autonomous Systems. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI 2012), May 5-10th, Austin, Texas. - only distributed at the conference
- Erik Hollnagel, Celia Martinie, Philippe Palanque, Alberto Pasquini, Martina Ragosta, Eric Rigaud, Sara Silvagni. System Performances under Automation Degradation (SPAD) SESAR Innovation Days, Toulouse, ENAC, 29/11/2011-01/12/2011.
- Celia Martinie, Philippe Palanque, Eric Barboni, Martina Ragosta. Model Based Assessment of Automation Levels: Application to Space Ground Segments IEEE International Conference on Systems, Man and Cybernetics, Anchorage, 09/10/2011-12/10/2011, IEEE Computer Society - Conference Publishing Services, 2011.
- Celia Martinie, Philippe Palanque, Eric Barboni, Marco Antonio Winckler, Martina Ragosta, Alberto Pasquini, Paola Lanzi. Formal Tasks and Systems Models as a Tool for Specifying and Assessing Automation Designs International Conference on Application and Theory of Automation in Command and Control Systems (ATACCS 2011), Barcelona, Spain, 26/05/2011-27/05/2011, IRIT Press, 2011.