*** Clue Detection Probabilities ***

From: Jack Frost [Jack_Frost@Soza.com]
Sent: Saturday, June 06, 1998 12:25 PM
To: 'sarrd-l@HUSKY1.STMARYS.CA
Subject: Clue Detection Probabilities

Here's the response to David Tate's note I promised earlier in the week.

In general, I'm questioning whether PODs obtained from exercises are
accurate, realistic, unbiased estimators of the PODs being achieved
under actual operational conditions. However, before elaborating on
what I was trying to say, let me please let comment on what I was not
trying to say.  

First, properly trained and motivated searchers will almost certainly do
better than untrained "do-gooders." I have no argument there. However,
there is a limit to how long even a well trained and initially well
motivated searcher can remain alert in the face of large amounts of
visual stimuli ("noise") that is not the search object. It also seems
plausible that a searcher who has been on many searches and found
nothing will have a lower expectation of finding something on a real
live search than on an exercise where he knows with absolute certainty
that there are objects to be found. Searchers, after all, are only
human. The question is to what degree expectations, perhaps largely
unconscious ones, based on previous experience can be overcome by
training and a conscious motivation to find the subject or a clue and
whether that training and motivation can produce an effect anywhere near
equal to the expectations which come from the certainty of the presence
of objects in an exercise and the incentive of knowing that someone else
will know how well you performed. I'll have to let the psychologists
answer that one.

Second, I wasn't suggesting that searchers not be provided any
information about subjects or potential clues. I was merely suggesting
that if allowed to view the actual object being used in an exercise,
they are being given far more information than they would normally get
on a real search. Much more on this below.

Third, I think many who do not regularly engage in maritime searches
have a greatly oversimplified view of what the Coast Guard does on a
search and of the maritime search problem in general. I did not suggest
Coast Guard-style parallel track search patterns were necessarily
appropriate for ground searches. I also did not suggest a "one size
fits all" detectability (sweep width) number would be suitable for all
inland situations. Even for maritime searching, we have extensive
tables of values and correction factors to account for various
combinations of: environmental factors that affect detection, search
object characteristics that affect detection, and sensor/platform
characteristics that affect detection. What I did say was that the
general principles of search theory apply to virtually all search
problems. This is like saying the general principles of internal
combustion engine operation apply to all internal combustion engine
designs. That statement does not imply we can come up with a single
engine design which will be useful for all applications, just that any
design must use and account for the principles of operation if engines
built from that design are to work properly.

Fourth, I think many people, at their first exposure to the idea of
measuring detectability, overestimate the difficulty of obtaining
reasonably accurate, objective approximations of detectability for a
reasonably comprehensive set of environmental situations, search objects
and sensors, and underestimate the accuracy and objectivity having and
knowing how to use such detectability measures will bring to their POD
estimates.

Now on to the "good stuff" I learned in Portland. 

Ken Hill made a number of important points in his presentation, and I
must admit I was still operating in that context to some degree.
However, since many on this mailing list probably were not able to
attend his talk, I should have brought those points out. I'll try to be
as accurate as I can, and I hope Ken will jump into this discussion at
some point.

In his talk, Ken described the visual detection process. As a person
scans the nearby surroundings, he/she will focus momentarily on some
area. (Koopman calls these "glimpses.") The amount of time spent
looking at particular spot is called "dwell time." The longer the dwell
time, the more likely an object of interest will be detected if it is
present.  Dwell time is terminated by a decision that there's nothing of
interest and it's time to move on. However, that doesn't mean that the
brain has completed all its processing. Sometimes an initial decision
to look elsewhere is reversed a few seconds later as realization comes
that there might have been something of interest after all. When that
happens, the searcher takes a second, more careful look, to see if
anything of interest can be recognized.

The word "recognize" is key. It isn't just "seeing" the object that
counts.  It is seeing it and recognizing it that counts. (Koopman's
definition of "detection" explicitly includes recognition.) How often
has a person spent some time looking for an item, (e.g. misplaced car
keys, jewelry, or other item) only to find it right out in plain sight
in the area they were searching.  Although the object was almost
certainly "seen" during the search, it wasn't recognized for some
reason. Often this is due to the angle at which it was viewed. Ken
also described "canonical positioning" (I think that was the term). A
key ring with keys on it is easily recognized in a "normal" position
where the ring and keys are essentially flat with most of their surface
area showing. However, turn them so that the keys and ring are viewed
on edge, and not only has the detectable area decreased, but it is an
unusual view making the keys and ring harder to recognize.

Ken pointed out that the function of camouflage is not to prevent a
person from "seeing" the camouflaged object. The function of camouflage
is to delay recognition so that in scanning, the searcher will notice
nothing recognizable (i.e. "interesting") that will capture his/her
attention within a normal or possibly even an extended dwell time. The
desired result from the camouflager's point of view is for searchers to
pass the object without detecting its presence because, even though they
may have looked directly at it, they did not recognize it in the
interval of time they spent looking at it.

Having all this in mind, it came to me that if, for exercise or POD
experiment purposes, searchers are shown the very object they will be
searching for, an effect exactly opposing that of camouflage may occur.
When a person looks at an object they know they are going to have to
find later, they probably pick up and remember tens, perhaps hundreds or
even thousands of detailed cues and clues which will decrease
recognition time should they see the object again in the near future.
I'm not a psychologist, but it at least seems plausible that this would
happen and that the searchers would be largely unconscious of doing it
since it is a very natural process. So, if you show me a knapsack and
tell me that in an hour I'm going to have to find it in the nearby area,
I'm going to note, consciously and/or unconsciously, the exact size,
shape, color, texture, style, any distinguishing markings, and a host of
other characteristics which will tend to make recognition nearly instant
should I see it again.  If this effect exists, then PODs obtained from
exercises, at least as I've heard them described, must be significantly
biased toward the high end of the scale.

Along these lines, Ken used the children's game of "can you find Waldo"
as an example. Now, I have trouble with this game and Ken confessed
that he does also.  However, I had no trouble at all with Ken's example
slides. In the first, he showed Waldo alone on a neutral, flat
background. The second slide was a portion of a page from one of the
Waldo books. Waldo was in the same posture and attitude (standing and
vertical) and also in about the same position on the screen as he was in
the previous slide, and I believe the image was also roughly the same
size in both cases. Detection was easy and instant because of these
visual cues.

I think Ken is going to write a paper or article on his findings soon,
and I encourage all to read it when it becomes available. I just hope
he isn't upset with me for this "sneak preview." (I also hope I got
everything right!)

Mark Fowler made points similar to some of those above in his comments
last Wednesday. He also raised the very interesting question of just
how close the appearance of the object shown to searchers needs to be to
that of the actual object in order to avoid subverting the essential
recognition process. If I'm shown a bright blue knapsack and the actual
one being carried by the lost person is brown, has my ability to detect
(see and recognize) the actual knapsack been enhanced because I was
shown a knapsack or degraded because I was shown a blue one? That may
be an extreme example, but it illustrates the point. Again, I'm not a
psychologist, but it seems likely that searchers use whatever "search
image(s)" they have in their minds, whether generated from verbal
descriptions, past experience or recent viewing of similar objects, as a
"filter" to remove as much of the visual "noise" as possible as they
scan their surroundings. If so, then it seems at least plausible that
using the wrong "filter" could degrade detection.

As Mark points out, even if the exact make and model of a potential clue
can be discerned, its actual appearance may be significantly altered
from an "identical" item taken off the store shelf. The actual item may
be faded, torn, stained, dirty, have patches (of either the functional
or souvenir variety), have been altered in some other way by the owner,
etc. I'm sure that awareness of these possibilities are part of any
good searcher training program. The question is how effective that
awareness and training is in terms of improving detection performance.
It seems certain to improve performance, but by how much? I think that
before we put too much faith in the POD values now in vogue, we need to
take a hard-nosed, dispassionate, take-nothing-for-granted, scientific
look at our most cherished assumptions about detection and POD to see
where we are on the mark and where we are not.

Regarding parallel track searching: The Coast Guard uses parallel track
search patterns in rectangular areas (segments) because it is the most
efficient way to cover a large patch of ocean. However, if the target
can be localized, other patterns may be used, such as the vector search
where the legs look like spokes of a wheel covering a circular area, or
the expanding square pattern (where the legs are parallel, but
approximate an expanding spiral about a point.) Until recently,
aircraft navigation wasn't all that accurate in terms of staying on
track. In addition, the added POD benefit of parallel search legs
accrues only when the legs are parallel relative to the search object.
Most maritime search objects are adrift, following only a somewhat
predictable path based on local winds and currents, with a large random
component in their motion. No matter how parallel the assigned search
legs may look when plotted on a chart, they are never followed that
accurately and often their appearance and the area they cover when
plotted relative to the moving target are significantly different from
the "perfect" chart display.  For these reasons, among many others, I've
long held that the Coast Guard's POD curve tends to be optimistic, since
it is based on an assumption of perfectly navigated, equally spaced,
parallel tracks relative to the target. Only now with very accurate
navigation and search patterns that at least partially compensate for
mean target motion is the CG probably getting close, on a good day, to
the PODs given by the curve they've been using. On a not-so-good day
they're probably down near the "lower bound" curve Mark Fowler mentioned
(more below).

As Mark points out, search theory is sufficiently robust that it covers
search techniques and situations other than parallel track searching.
Unfortunately, Mark misused a term in his description. The correct
measure of search effectiveness is POS (POS = POA x POD), and the only
lower bound on POS is zero. Zero POS can come from either not searching
(POD = 0), or searching in the wrong place (POA = 0). What Mark was
trying to say is this:  If a certain amount of search effort (i.e. a
given number of searcher-hours expended at some known searcher speed for
an object of known detectability (sweep width)) is applied uniformly
over an area (segment) of known size, then there is a theoretical lower
bound on the POD that effort should produce. That lower bound is given
by the so-called exponential detection function. There is also a
theoretical upper bound given by the so-called definite range detection
function. I can't go further than that without launching into a
treatise on search theory complete with a lot of mathematics most people
don't care to look at or even know about. What I can tell you is that
many of the published sets of POD values I've seen in the inland search
management books are inconsistent in the sense that at some point, some
of the values violate the upper or lower bounds given by these two
detection functions. That is, many times some, but not all, the values
in a given set can be correct. (The problem usually lies in either the
data analysis technique or experimental measurement error.)
Unfortunately, a number of false conclusions about the relationship
between POD and effort expended have resulted from these
inconsistencies.

Another thing that Mark correctly observes is that we need to identify
only those factors which have a significant impact on detection, and
then measure what those effects are. For example, we probably won't
need separate detectability values for blue, black, brown, green, red,
.. knapsacks.  We can probably get by with a rough estimate of contrast
with the prevailing surroundings like high, medium, or low. A limited
number of search object sizes will also probably do.

One more point. PODs, POAs and POSs are all "soft" numbers. We will
never have exact values to work with and such a thing as an exact value
probably doesn't exist anyway. Some have seized on this fact to argue
that one estimator or estimation technique is as good as another. For
anyone who believes this, I can only suggest dart boards and/or dice.
An unbiased, objective "soft" number is infinitely better than a WAG or
an estimator that's likely to be biased, even when the uncertainty
("softness") is large. PODs cannot be observed or measured directly on
an actual search. I think we should, to the best of our ability,
experimentally determine the effects of significant
measurable/observable factors, tabulate the results, and use those
results to infer PODs from measurements/observations of the same factors
at the scene of an actual search. Just asking searchers, in effect,
"What do you think your POD was?" seems to provide more opportunities
for bias and inaccuracy than we need.

In response to Mark's query about potential significant factors
affecting detection, things that come to my mind fall into three broad,
often interdependent, categories. The categories are search object
characteristics, environmental factors, and sensor characteristics.

Search Object Characteristics (Visual Search)
Size (small clues are generally harder to detect than subjects, for
example), color contrast with surroundings, brightness contrast with
surroundings, potential for attention-getting movement (waving, jumping
up and down, shaking vegetation, etc.), ability to make fire, smoke,
etc., day/night visual signaling devices carried (mirror,
flashlight/torch, etc.), likely behavioral factors (e.g. tendency to
crawl into brush, parked cars, etc. This may also affect POA
estimates), etc.

(FLIR) - Temperature contrast with surroundings, angular coverage
(e.g. 7 degrees).

(Audible search) - Ability/likelihood of subject responding to
searchers, sound signaling devices carried (e.g. whistle, game caller,
firearms), etc.

Environmental Factors
Visibility/Weather (fog, rain, falling/blowing snow must hamper ground
searches just as it does maritime ones), vegetation (type, density of
ground cover), terrain, season (esp. where there's significant seasonal
variation in density and color of vegetation, snowfall, etc.), lighting
conditions, etc.

Sensor/Platform Characteristics
Type of sensor (visual, aural, FLIR, etc.), search speed, type of
platform (on foot, snowmobile, aircraft (helicopter, fixed wing, single
or multiengine, high or low wing), etc.), search light power, night
vision goggles, searcher/sensor operator training/experience level,
navigational capability (i.e. ability to search where the search manager
wants them to search in the manner he/she wants them to search), time on
task, general fatigue level (did searcher start out tired?), etc.

Well that's more than enough of my palavering for one e-mail. I hope
this provokes some thoughts and generates some more feedback! I'll be
out of circulation for a little while, but don't let that stop anyone!

Jack Frost.