Slow is not safe

By Dr. Dan Gould, Solutions Engineer

To be honest, there are no good answers when you are standing in front of leadership and asked, “So exactly how far over the maximum allowable temperature did you let our million-dollar jet engine get?” But “Uh, well, I don’t really have that data” is about the worst one there is. Unfortunately, that was exactly the answer I had to give after one particularly disastrous day as a young propulsion test engineer about 8 years ago. So how did I get into this situation in the first place?

The runaway engine

Well, we were working with an engine that was known to sometimes quickly overshoot its max allowable temperature during startup. I had thus created an algorithm to warn the operator if this was about to occur, and allow him to abort the start. The algorithm worked perfectly, but I’d made a mistake in the UI programming. So when we started the engine, the indicator that should’ve warned the operator in time to abort the start didn’t light up.

Still, a few seconds later it was obvious that things were getting too hot, too fast. So we made the abort call and pulled back the throttle to shut off the fuel.

But nothing happened. Engine temperature continued to climb, and engine speed started to ramp up as well.

This was surprising and rather confusing, but no matter; we had a giant red physical emergency stop button that would close a separate fuel-shutoff solenoid between the fuel tank and the engine, placed there for this exact kind of situation. So the order was given. The button was pressed. And still nothing happened. The engine continued to spin, and its temperature climbed even higher.

Now we were beginning to worry. We initially hoped that the isolation valve had actually closed and the engine was just burning through some remaining fuel in the line. But this hope quickly died as the engine stayed alive and its temperature continued rising. If we couldn’t stop it, the best case was that the engine would continue running for an hour or two until it exhausted its fuel tank, at which point we would be left with a stopped, but likely scrap, million-dollar engine. Worst case, the temp would keep climbing until part of the turbine assembly decided to let go, sending shards of metal outward at 500 mph.

But perhaps the E-stop was just wired incorrectly and the shutoff valve was still getting power from somewhere else. As a last-ditch effort, we turned the key to completely remove electrical power from the entire test stand. For a moment, it seemed like it worked: engine temperature dropped! But we soon realized that this was only because the power to our data acquisition hardware was also controlled by that switch. All of our displays, including engine temperature, instantly dropped to zero. The engine kept on spinning. Its temperature, however, was now completely unknown.

Eventually, through a combination of shouting and frantic waving of arms, it was communicated to a nearby technician that something was wrong, and the technician wisely ran over and closed the manual hand valve that acted as the last isolation between the fuel tank and the engine. This thankfully worked, and the engine soon began to spool down.

Postmortem

In the end, although we had avoided creating any shrapnel, the high temps we did see combined with the missing data we didn’t have meant that all four of the engine’s turbine wheels had to be replaced. This cost about $100k in parts alone, and also resulted in about 5 months of downtime while we waited for the replacements to arrive.

So why did this happen? It turned out that several separate things had to go wrong at the same time for that engine to get away from us, and one of those things was indeed my fault. But a list of who did what, and when, is perhaps the least useful thing a postmortem can produce. The better question was why none of these mistakes were caught before we lit the engine, and the answer pointed back at our tools:

1. The visualization software could not control outputs.

Our operator screen was a collection of ActiveX controls that had many options for customizing their appearance, but zero ability to command any outputs. All abort logic had to be constructed within the limitations of both human memory and reaction times. Optimal abort limit values that might better prevent false positives were traded against alerting early enough for a human to see and act upon an indicator. Even worse, because the software had no control ability, the throttle on the million dollar engine was controlled with a finicky, partially-broken, lever-actuated-potentiometer. This potentiometer was used to generate the throttle actuator control signal, but it had become slightly out of cal, and its minimum output voltage shifted above that required to command the engine to fuel cutoff. Fuel was therefore introduced into the engine too early during startup, and then never stopped flowing even when the throttle lever was pulled all the way back.

2. There was no way to simulate any of it.

The visualization software could only be fed from live or historical data. So there was no easy way to throw in a simulated hot start to confirm that the correct indicator turned red when it was supposed to.

3. The hardware interfaces were too rigid to bench-test a part.

We later discovered that the E-stop solenoid valve had been installed backwards, and unfortunately it was a flow-assisted design that couldn’t close against the flow of fuel. A flow test with real fluid going through it would have immediately caught this issue. But doing that test would have meant removing and completely reconfiguring the flight DAQ hardware we were using for the actual engine test, plus building an entire software configuration, all for one valve. So instead we had simply connected the valve to a power supply, powered it on, heard it click, and called it good. Given the available tools, this seemed like a reasonable call at the time — and that’s exactly the problem. Contrary to the common aphorism, in testing, slow is not safe, because testing slowly means testing less.

4. And the system punished you for changing it.

Adding or adjusting a sensor on the flight DAQ we were using generally required power-cycling the unit. Thus it was intentionally hooked up to the main test stand power, where it could be easily power-cycled remotely.

Defining a system to avoid these problems

None of these problems were unique to that test stand. I’ve run into versions of every one of them on nearly every stand I’ve worked on since. Flip each one around, though, and you get a fairly good specification for what an optimal test control system should do instead:

+ Deterministic, integrated control and visualization

It should be one system that acquires data from a variety of sources, aligns it all in a rich visual interface, and uses it to make real-time calculations and outputs, all on a fixed and known cycle. If an indicator on the screen says the engine is too hot, the system behind it should be able to do something about it. Additionally, robust data simulation capability is needed so that both UI and control programming can be tested without having to take the added time and risk of connecting to physical hardware before software verification has been completed.

+ Usable at any scale

It should be just as reasonable to use the same system for one sensor on a bench as for a giant test stand with thousands of channels from dozens of different data acquisition devices. That means hardware flexibility: it should be able to talk to anything from a single LabJack to multiple NI PXIe systems to dozens of racks of PLC chassis. It also means a highly powerful system, but one possessing a shallow learning curve and a very small minimum required setup. Reading a flow meter, putting the value on a screen, and setting a digital output when flow rises above a limit should be as valid a use case as running an entire rocket test stand with thousands of inputs and hundreds of precisely sequenced outputs.

+ SAFE TO CHANGE

First, the system should be simple for test engineers to modify and experiment with. The engineers who wrench on the hardware should also be able to write the sequence that drives it. That calls for a simple, high-level language built for control logic, not an object-oriented C++ framework that only a dedicated software team can safely modify. Modifying sensors, outputs, programming, and visualizations should be possible with the minimum necessary interruption to the system as a whole. Second, there must also be methods to lock down the system on test day. Configuration, code, and UI should all compile down to version-controllable text files, so that on test day the exact, tested, known-working configuration is what runs, rather than whatever version an engineer was last experimenting with.

Enter Revel

Historically, finding a system that met all of these requirements was, to put it mildly, difficult. Graphical programming languages could offer a shallow learning curve, but at a cost. Programming complex logic with graphical methods often resulted in an unmaintainable mess of binary spaghetti, impossible to version-control or review.

Alternatively, very large companies sometimes have dedicated teams of software engineers capable of building internal tools with some of these features. However, those tools were usually written, by necessity, in lower-level languages like C++. This meant that while the test engineers could use the tools, they could never hope to modify them without putting in a ticket to the software team that often took weeks to turn around.

And even among the largest engineering companies, such teams were the exception rather than the norm. The jet engine incident happened while I was working at a very large company myself, but its software teams were, quite reasonably, focused on flight software and flight hardware, and so ground testing got the scraps.

As for everyone else, well, I personally know of nationally renowned aerospace test facilities that, to this day, use an Excel spreadsheet as their UI and VBA code as the backend to control their hundred-million-dollar facilities.

Revel, however, meets each and every one of the aforementioned requirements. It is deterministic. It is scalable. It is easy to change and yet safe to use. Remember the backwards-installed solenoid valve? That mistake wasn’t caught because setting up and programming a control system to conduct a flow test was deemed too difficult. But with Revel, things are different. The entire code required to conduct an automated flow test on the solenoid is shown in the 10 lines below:
‍

const CYCLE: float = 0.01

hw out |fuel.shutoff_open|: bool = False    # solenoid, via relay on a digital output
hw in  |fuel.flow_gpm|: float               # flow meter on an analog input

sequence shutoff_valve_flow_check:
    """Open the valve and confirm flow; close it and confirm the flow stops."""
    |fuel.shutoff_open| = True
    wait 5.0
    check |fuel.flow_gpm| > 1.0 "no flow with valve open"
    |fuel.shutoff_open| = False
    wait 2.0
    check |fuel.flow_gpm| < 0.05 "valve failed to close against flow"

Hook up a LabJack, a relay, a flow meter, and a hose, type these 10 lines into the provided code editor, and the backwards valve would have been caught on the bench, saving a million-dollar engine.

What about my personal mistake, the custom prediction algorithm undone by an incorrectly programmed indicator light? With Revel, both the alert and the automated command to cut fuel are completed in only 7 lines of RevelCode:
‍

alert turbine_overtemp:
    """Turbine inlet temperature is above the allowed limit."""
    severity = Severity.critical
    arm = |turbine_rpm_n1| > TURBINE_START_RPM
    trigger = |turbine_inlet_temp_f| > TURBINE_MAX_TEMP_F

loop overtemp_abort:
    if |alert.turbine_overtemp.fired|:
        |fuel.shutoff_open| = False

Because the alert and the control logic live in the same program, test sequences are no longer limited by human reaction times. The fuel valve closes in the same cycle the alert fires, so the abort threshold can be tuned to what the engine actually needs. And if a programming mistake causes the dashboard indicator to remain green, the valve closes anyway.

The Revel platform is also much easier to change than anything I’ve ever used. Modifying a sensor config means a 5 second software redeploy rather than power-cycling any hardware, so there’s no longer a reason to wire the data acquisition system to test stand power. At the same time, the code, device configuration, and dashboards are all version-controllable text files, so on test day you run a tagged version you know passed all its tests.

Revel records every channel at the full cycle rate the entire time the runtime is running, without having to be monitored or commanded to do so. It also logs every time an alert fires, so there’s a record of it even if nobody was looking. So, had I been using Revel, I could have said exactly how far over the limit the engine got and for how long, which would have made for a much shorter — and likely much more pleasant — meeting with leadership.

Conclusion

To conclude, I’d like to return to that cold February morning and the runaway engine. The below picture is of a turbine wheel from that very engine. I’ve kept it ever since, partly as a souvenir, but mostly as a reminder. Nobody in that test cell was careless that day. I made a mistake in the UI, another individual installed a valve backwards, and someone else made a perfectly sensible decision about where to plug in the DAQ.

Each one of those things is the kind of small mistake that happens on every test campaign. But what turned these small mistakes into an expensive disaster was a set of tools that made it too hard to catch these minor errors ahead of time. Now, even great software can’t prevent all possible mistakes. But it can make the mistakes cheap to find, on a bench or at a desk, long before anyone is standing next to a running engine. And when something does still go wrong, using the correct tools can make sure you have the data to understand exactly what happened, which, and please trust me on this, is a good deal more comfortable than the alternative.

It is often said that good writers don’t have to explicitly state the main point they are trying to make; their writing makes their purpose clear on its own. But I’m a mere test engineer rather than a syndicated columnist, so please forgive me for just bluntly stating my claim here: the Revel platform is the best control system I have ever seen, bar none. I’ve spent a good part of my career building control systems, fighting with them, and working around them, and Revel is the first one I’ve ever used that has me spending more time smiling than swearing. And yes, I work for Revel, and I’m aware that this sounds exactly like what someone who works here would say. But really, that just means I put my money where my mouth is: I took one look at RevelCode and the dashboards, and quit my previous job testing rockets because I wanted to be a part of something even bigger. This, of course, is still a rather bold claim. So don’t just take my word for it. Schedule a demo. Let us prove to you and your team that Revel is everything I say it is.