Dependability Challenges in Safety-Critical Systems: the adoption of Machine learning
Machine Learning components in safety-critical applications can perform some complex tasks that would be unfeasible otherwise. However, they are also a weak point concerning safety assurance. We will illustrate two specific cases where ML must be incorporated in SCS with much care. One is related to the interactions between machine-learning components and other non-ML components and how they evolve with training of the former. We argue that it is theoretically possible that learning by the Neural Network may reduce the effectiveness of error checkers or safety monitors, creating a major complication for safety assurance. An example on automated driving is shown. Among the results, we observed that indeed improving the Controller could make the Safety Monitor less effective; to a limit where a training increment makes the Controller’s own behavior safer but results in the vehicle to be less safe. The other one regards ML algorithms that perform binary classification as error, intrusion or failure detectors. They can be used in SCS provided that their performance complies with SCS safety requirements. However, the performance analysis of MLs relies on metrics that were not developed with safety in mind and consequently may not provide meaningful evidence to decide whether to incorporate a ML into a SCS. We analyze the distribution of misclassifications and thus show how to better assess the adequacy of a given ML.
Andrea Bondavalli is a Full Professor of Computer Science at the University of Firenze, previously he was a researcher of CNR in Pisa. His research activity is focused on Dependability and Resilience of critical systems and infrastructures. In particular he has been working on designing resiliency, safety, security, and on evaluating attributes such as reliability, availability and performability. His scientific activities have originated more than 250 papers appeared in international Journals and Conferences. He received a Doctor Honoris Causa award from the Budapest University of Technology and Economics – in 2019. Andrea Bondavalli since more than 20 years supports as an expert the European Commission in the selection and evaluation of project proposals. He founded a spinoff – Resiltech – which employs currently 45 people and consults a few companies. He led various national and European projects and coordinated a few. He participates to (and has been chairing) the program committee in several International Conferences including DSN, SRDS, SAFECOMP EDCC, LADC. Finally he is a member of the IEEE and of the IFIP W.G. 10.4 Working Group on "Dependable Computing and Fault-Tolerance.
Andrea Bondavalli, Professor, University of Florence, Italy
Security vs. Reliability: Is one more difficult to achieve than the other?
The security versus reliability debate is an old one. While many reliability researchers may see security as a subset of the reliability problem, many researchers that are solely focused on security topics would disagree, and might not give reliability problems the importance that they deserve. In this talk, I will elaborate on the security versus reliability debate and will talk about some of experiences in trying to answer if one of these issues is more difficult to achieve than the other.
Engin Kirda is a professor of computer science at Northeastern University. Before that, he held faculty positions at Institute Eurecom in the French Riviera and the Technical University of Vienna, where he co-founded the Secure Systems Lab that is now distributed over five institutions in Europe and the United States. Engin’s research has focused on malware analysis (e.g., Anubis, Exposure, and Fire) and detection, web application security, and practical aspects of socialnetworking security. He was a co-founder of Lastline, Inc., a Silicon-Valley based company that specialized in the detection and prevention of advanced targeted malware that was acquired by VMWare in 2020. Engin was the program chair of the International Symposium on Recent Advances in Intrusion Detection in 2009, the program chair of the European Workshop on Systems Security in 2010 and 2011, the program chair of the well-known USENIX Workshop on Large Scale Exploits and Emergent Threats in 2012, the program chair of the security flagship conference Network and Distributed System Security Symposium in 2015 and USENIX Security in 2017.
Engin Kirda, Professor, Northeastern University, Boston, USA
Self-Driving Cars – Challenging Reliable Distributed Systems
The automotive industry is working full speed on self-driving cars. They must be reliable, even ultra-high reliable, to ensure the safety of their passengers and must not pose a threat to others around. They are also distributed systems because they must be fail-operational. Unfortunately, despite vast economic investments, we have not reached a sufficient level of safety. In this talk, I will review the major challenges in designing self-driving cars and discuss various ongoing activities that may lead to a solution.
Wilfried Steiner received a degree of Doctor of Technical Sciences and the Venia Docendi in Computer Science, both from the Vienna University of Technology, Austria (in 2005 and 2018, respectively). From 2009 to 2012, he was awarded a Marie Curie International Outgoing Fellowship hosted by SRI International in Menlo Park, CA. His research is focused on dependable cyber-physical systems for which he designs algorithms and network protocols with real-time, dependability, and security requirements. Wilfried Steiner has been the SAE AS6802 (Time-Triggered Ethernet) editor and served multiple years as a voting member in the IEEE 802.1, standardizing time-sensitive networking (TSN). He is the Director of the TTTech Labs, which acts as the center for strategic research within the TTTech Group.
Wilfried Steiner, Director, TTTech Labs, Vienna, Austria