A fighter pilot looks down at the display and sees a symbol. One symbol, in one place, with a speed, a heading and a label saying what it is and whose side it is on.

Behind that single symbol, several entirely different machines have reported entirely different things. The radar has an accurate range and bearing but no idea what it is looking at. An infrared sensor has a very precise bearing and no range at all. A passive receiver has heard a radar transmitting and can name the type of radar, but cannot say how far away it is. A fourth report has arrived over a data link from an aircraft eighty miles away, measured from a different place at a slightly different moment.

Deciding that all of those describe one aeroplane, and not four, is the thing people mean when they say sensor fusion. It is not a display technology and it is not a marketing word. It is a hard mathematical problem that was not solved on the first attempt, and in the case of the F-35 Lightning II the public record of how badly it went is unusually specific.

Data fusion and sensor fusion are not the same thing

The terms get used interchangeably and they should not be. The discipline has had a formal structure since the late 1980s, when a US Joint Directors of Laboratories subgroup built what everybody still calls the JDL model. Alan Steinberg, Christopher Bowman and Franklin White revised it in 1998 and 1999, and that revision is the version still cited today.

It sorts the work by what is being estimated rather than by which box does it. Level 0 is feature and signal assessment — patterns inside a single sensor's raw output. Level 1 is individual entity assessment: deciding how many objects are out there, where each one is, and what it is. Level 2 is situation assessment, the relationships between them. Level 3 is scenario and impact assessment, what is about to happen. Level 4 is the system looking at itself — sensor alignment, performance, whether the models still fit. A Level 5 for the human in the loop was added later.

Sensor fusion in the everyday sense is almost entirely Level 1, and the reason it is hard has nothing to do with how many sensors you bolt on. Adding a sensor adds reports. Reports do not combine themselves.

What each sensor can and cannot give you

Every sensor on a fighter is good at something and blind to something else, and the pattern of those blindnesses is what makes fusion worth doing in the first place.

A radar measures range directly and measures angle poorly — an electronically scanned array narrows the beam but a bearing is still far less precise than the range that comes with it. It is also the only one of the three that announces your presence by transmitting. An infrared sensor is the mirror image: the angular precision is excellent, there is no range information in a thermal image at all, and nothing leaves the aircraft. A passive electronic-support receiver gives you a bearing to an emitter and, from the pulse characteristics, a strong hint of what type of emitter it is, which is identification the radar cannot provide.

Put the radar's range together with the infrared sensor's bearing and you get a position better than either could produce. Add the receiver's judgement about what is transmitting and you get an identity. That is the prize. The catch is that you only get it if you have correctly decided the three reports belong to the same object.

Correlation is the whole job

The technical name for the decision is data association, and in a Lockheed Martin paper on the F-35's mission systems released for public distribution in 2018, it is described exactly that way: an association process decides whether incoming data update an existing track or start a new one, and the track's position and identity estimates are then updated accordingly. The paper's stated aim is that “multiple detections of an entity are combined into a single track instead of multiple tracks”.

Stated like that it sounds procedural. It is not, because every report comes with uncertainty attached, and the uncertainties are different shapes. The radar's error is a thin wedge, long in angle and short in range. The infrared sensor's error is the opposite, a narrow line stretching away to infinity. The algorithm has to ask whether those two error regions overlap enough to be the same object, and the answer depends on how much error you are willing to allow — the gate.

Set the gate too tight and genuine reports about one aircraft fail to associate, so a second track is created alongside the first. Set it too loose and two aircraft flying close together collapse into one track, which is the more dangerous error, because the second aircraft is then simply not on the display. There is no setting that avoids both. Every fusion engine is a compromise between producing ghosts and hiding real aeroplanes, and the compromise has to hold while targets manoeuvre, while sensors drop in and out, and while the geometry changes.

It went wrong in public, and the record is specific

Most writing about F-35 fusion is either manufacturer material or hostile commentary. There is a third category, and it is the useful one: the annual reports of the Pentagon's Director, Operational Test and Evaluation, which describe problems in dated, concrete terms because that is their function.

The FY2016 report is the one that set it out in full, under the flat heading that avionics sensor fusion performance was still unacceptable.

Read that again slowly, because it is the exact failure the diagram above describes, found in a real aeroplane. The reported workaround was to switch sensors off. A fusion engine that works better with fewer inputs has not been fused at all.

The same report went further. Multiple false ground tracks appeared when only one threat emitter was transmitting. Tracks that timed out and dropped off the display could not be recalled, which matters against surface-to-air systems whose engagement radars deliberately transmit only in short bursts. And the errors did not stay in one cockpit: sharing erroneous tracks over MADL between aircraft in the formation, the report said, multiplied the problems. The following year's report listed deficiencies in the fire control thread and fusion functionality among the pacing items holding up weapons integration events.

Lockheed Martin's own 2018 paper is candid about the same thing from the other direction, naming the assurance that displayed tracks were real and not duplicated as a major early fusion problem, and singling out MADL data as where it bit hardest.

Why sharing tracks makes it harder, not easier

It is tempting to assume that four aircraft pooling their sensors must see four times as well. Sometimes they do. But a track arriving over a data link is a fundamentally different object from a radar return, and treating it as though it were the same is how formations end up looking at phantom aeroplanes.

A track from another aircraft was measured from a different position, so any error in where that aircraft thinks it is becomes an error in where it thinks the target is. It was measured at a different instant, so it has to be propagated forward before it can be compared with anything. And it is not raw data but someone else's conclusion — a track that has already been through a fusion engine once, with its own gating decisions baked in. Associating two already-fused tracks is a separate problem from associating two detections, and getting it wrong does not produce one extra ghost; it produces a ghost on every display in the formation at once.

This is the real distinction between the F-35's MADL and the older Link 16 that a generation of fighters shares. Link 16 was built to exchange a tactical picture between many different platform types. MADL is a directional, low-probability-of-intercept link between F-35s designed to pass the fused picture itself. The ambition is higher, and so is the cost of a correlation error.

What the pilot is actually being given

The point of all of this was never the sensors. It was what the pilot does not have to do.

In a fourth-generation cockpit the correlation was done by the pilot. Radar contact here, radar warning receiver strobe there, a call from a controller, a visual tally — and a human head decided which of those were the same aircraft, under g, at night, with a finite amount of attention. Pilots were good at it. They were not always right, and being right took capacity that was then not available for anything else.

Moving that job into software does not make the problem go away. It moves it somewhere the pilot cannot see, which is why the display convention matters so much: a symbol with no uncertainty drawn around it looks equally confident whether it rests on three agreeing sensors or on one ambiguous return. The honest version of the single-picture claim is not that the aeroplane knows more. It is that the aeroplane has already made the decisions a crew used to make out loud, and the pilot is now trusting those decisions rather than taking them.

That is also why the fusion story is a better measure of a modern fighter than the stealth story. Low observability decides who gets to shoot first. Correlation decides whether the thing being shot at is really there, and whether the second aeroplane behind it is on the display at all. One of those is a shaping problem that was largely solved decades ago. The other is still, every single sortie, a judgement call being made several times a second by software.

Sources: Director, Operational Test and Evaluation, FY2016 and FY2017 Annual Reports, F-35 Joint Strike Fighter; Lockheed Martin, “F-35 Mission Systems Design, Development and Verification” (approved for public release, JSF18-502); Steinberg, Bowman, White, Blasch and Llinas on the JDL data fusion model, Journal of Advances in Information Fusion; The Register interview with Squadron Leader Andy Edgell, January 2018.