This article shows how bad design creates lethal operating conditions. Two examples are provided as unfortunate results of such bad design. This may serve as elements of reflection for designers and manufacturers of critical systems.

As part of my activity in the newly formed Innovation Center in Dublin in 2016, I had the chance to attend and coach several Design Thinking sessions.

Design Thinking, or Innovation with design in mind, is a great way to create a fruitful emulation between subject matter experts, designers and developers to lay down rapid prototypes. So thank you Accenture and Luma Institute that provided great training.

In these sessions, designers are having the best share, as their sketching abilities really boot-up the joint effort by providing visual common reference to the whole team.

Yes, sketching is the most wanted ability in the first two days of a five days Design Thinking session. It is not only about giving the bird’s eye view of the problem (aka draw on a napkin) but also to sketch the workflow and visualize transitions between phases or communications between systems. So “Bravo” to the designers, they are by far necessary and great minds with gifts we really need.

But, designers are not the users. When real life catches up with design, we do have the same problems we face with AI. It lack common sense.

Design primary motivation is not to ensure safety. It’s core inspiration is to provide a new user experience, a smoother system understanding, a more user friendly interface.

While those goals are all worthwhile, they fall short of capturing the essential safety need that should preside to ANY system. Yes, design should follow Asimov’s robotics laws and first and foremost produce systems that should NOT HURT human beings. Then only they should OBEY ORDERS and eventually PROTECT THEMSELVES.

Let’s take to recent and unfortunate deadly accidents to illustrate these.

On August 21st, 2017, the USS John S. McCain collided with the Alnic MC, a Liberian oil tanker, off the coast of Singapore. The report provides a detailed overview of the actions that led to the collision: when crew members tried to split throttle and steering control between consoles, they lost control of the ship, putting it into the path of the tanker. The crash killed 10 sailors and injured 48 aboard the McCain.

The report says that while fatigue and lack of training played a role in the accident, the design of the ship’s control console were also contributing factors. Located in the middle of the McCain’s bridge, the Ship’s Control Console (SCC) features a pair of touch-screens on both the Helm and Lee Helm stations, through which the crew could steer and propel the ship. Investigators found that the crew had placed it in “backup manual mode,” which removed computer-assisted help, because it allowed for “more direct form of communication between steering and the SSC.” That setting meant that any crew member at another station could take over steering operations, and when the crew tried to regain control of the ship from multiple stations, control “shifted from the lee helm, to aft steering, to the helm, and back to aft steering.”


Source: The verge

As a consequence, The US Navy will replace its touchscreen controls with mechanical ones on its destroyers.

Here is now a second example that illustrates a really scarry design related flaw. That really scares me to the bones when I realize that it actually exists in every airliner. Yes, all airplanes are now equipped with joysticks (I even played with them extensively when I was working for the Flight Simulator Division of Thales in 1988-1989 and integrated the lessons plans in A320 sims), well “Side Sticks”. Today’s commercial pilots use a passive side-stick controller to transmit their commands, but they don’t get any direct feedback from the airplane, nor do they get any feedback from the actions of the co-pilot.

This was inherent to fly-by-wire systems and simply vanished when the new generation of passive sidesticks happened. Why ?

On June 1, 2009, Air France Flight 447 crashed into the ocean on its way back from Rio de Janeiro. 216 passengers and 12 crew died on impact. The official investigation has concluded with “human error” as the culprit–pilots making mistakes that forced the plane to crash. But evidence unearthed by The Telegraph tells a different story, that the pilots of the Airbus A330-200, and everyone else on the plane, were really victims of bad design.

The last words of Pilots on flight 447:

02:13:40 (Robert) “Climb… climb… climb… climb…”
02:13:40 (Bonin) “But I’ve had the stick back the whole time!”
02:13:42 (Dubois) “No, no, no… Don’t climb… no, no.”
02:13:43 (Robert) “Descend… Give me the controls… Give me the controls!”

Source: Fastcompany

That should never happen, we all share that gut feeling.

They are fortunately researches and developments undergone to compensate such widespread design errors.  

Safran has recently introduced the SSU, an active sidestick unit that simulates control forces, and allows pilots to rediscover the real feeling of flight. Pilots enjoy better overall situational awareness, whether concerning actions by the other pilot, feedback from the autopilot, or the aircraft’s general behavior. Offering all the efficiency and accuracy of electrical/electronic systems, the SSU gives the pilot very realistic flight control sensations.

Source: SAFRAN Electronics and Defense

These considerations lead to a simple conclusion, Design, as interesting a tool and a concept should never be considered in abstraction of the elementary safety procedures and human factors. The safety rules should always prevail and the system should strive to provide as much information as possible to the human operator to enable a fully informed decision to be taken. 

Moreover, specifically bearing in mind the above example of US Navy, we should be aware  of our growing dependence from electronic systems that not only are battery dependant but also rely on critical infrastructures to provide their day-to-day services. It is vital to protect those infrastructures and prepare contingency and continuity plans. But it’s a different story…

Montpellier, November 2019