Problems with Existing Prediction Studies and Future Research Needs

TLDR: In criminology, neither clinical nor statistical methods of prognosis and prediction can avoid the two classes of error and these two kinds of error will be present no matter how success or failure are defined.

Abstract: Most decisions involve risk and are thus subject to two kinds of error. Prognoses, estimates of future conditions, and all probability statements are subject to the same two kinds of error. In criminological prognoses, the decisionmaker can be in error in that: (1) the individual who is predicted to fail may succeed, or (2) the individual who is predicted to succeed may fail. These two kinds of error apply to all decisions or estimates irrespective of the means by which such decisions or estimates are derived. In criminology, neither clinical nor statistical methods of prognosis and prediction can avoid the two classes of error.' In addition, these two kinds of error will be present no matter how success or failure are defined. The first kind of error is

Full text

Journal of Criminal Law and Criminology
-*2+#
002# Summer
/1'!*#
2++#/
Problems with Existing Prediction Studies and
Future Research Needs
Leslie T. Wilkins
-**-41&'0,"""'1'-,*4-/)01 &9.00!&-*/*6!-++-,0*4,-/1&4#01#/,#"2(!*!
/1-$1&# /'+',*4-++-,0 /'+',-*-%6-++-,0,"1&# /'+',-*-%6,"/'+',*
201'!#-++-,0
8'06+.-0'2+'0 /-2%&11-6-2$-/$/##,"-.#,!!#00 6-/1&4#01#/,,'3#/0'16!&--*-$4!&-*/*6-++-,01&0 ##,!!#.1#"$-/
',!*20'-,',-2/,*-$/'+',*4,"/'+',-*-%6 6,21&-/'7#"#"'1-/-$-/1&4#01#/,,'3#/0'16!&--*-$4!&-*/*6-++-,0
#!-++#,"#"'11'-,
#0*'#'*)',0/- *#+04'1&5'01',%/#"'!1'-,12"'#0,"212/##0#/!&##"0/'+/'+',-*-%6

9901-4169/80/7
102-0098S02.00/0
THE
JOURNAL
OF
CRIMINAL
LAw
&
CRIMINOLOGY
Vol.
71,
No.
2
Copyright
©
1980
by Northwestern
University
School
of
Law
Printed
in
U.S.A.
PROBLEMS
WITH EXISTING PREDICTION
STUDIES
AND
FUTURE
RESEARCH
NEEDS
LESLIE
T.
WILKINS*
Most
decisions
involve
risk
and
are
thus
subject
to
two
kinds
of
error.
Prognoses,
estimates
of
future
conditions,
and
all
probability
statements
are
sub-
ject
to
the
same
two
kinds
of
error.
In
criminolog-
ical
prognoses,
the
decisionmaker
can be
in
error
in
that:
(1)
the
individual
who
is
predicted
to
fail
may
succeed,
or
(2)
the individual
who
is
predicted
to
succeed
may
fail.
These
two
kinds
of
error
apply
to
all
decisions
or
estimates
irrespective
of
the
means by
which
such
decisions
or
estimates
are
derived.
In
criminology,
neither
clinical
nor
statis-
tical
methods
of
prognosis
and
prediction
can
avoid
the
two
classes
of error.
1
In
addition,
these
two
kinds
of
error
will
be
present
no
matter
how
success
or
failure
are defined.
The
first
kind
of
error
is
usually
termed a
"false
positive,"
or
"overpredic-
tion."
Overprediction
will
tend
to
increase
in
pro-
portion
to
the
decrease
in
the
number
of
individ-
uals
who
fit
the
"fail"
category.
It
is
important
to
stress
that
false
positives
cannot
be
avoided.
Clinical methods
usually are
not
able
to
estimate
the
magnitude of
the
errors, whereas
statistical
methods
do
so
as
an
integral
part
of
the
process
of
calculation
of
the
likelihood
of
success
or
failure.
But clinical
ignorance
of
the
size
of
the
errors
does
not
mean
that
those
studies
produce
a
smaller
number
of
errors
than
studies
applying statistical
prediction.
Where
comparisons
have
been
made
of
the
false
positive
rates,
the
clinical
rates
have
been
larger
in almost
all
instances.
2
THE
USE
OF
PREDICTIVE
STATEMENTS
At
present,
statistical
predictions,
and
also,
by
inference,
all
clinical
predictions, produce
a
large
proportion
of
false
positives.
The
proportion,
to
*
Professor,
Graduate
School
of
Criminal
Justice,
SUNY Albany;
currently,
Visiting
Professor,
Simon
Fraser University.
IIn
industrial
decisions,
the
two
classes
of
error
are
often
termed
"producer"
and
"consumer"
risks,
and
es-
timates
of
the
magnitude
of
the
two
classes
are frequently
written into contracts.
2
P.
MEEHL,
STATISTICAL
V.
CLINICAL
PREDICTION:
A
THEORETICAL
ANALYSIS
AND
A
REVIEW
OF THE
EVIDENCE
(1954).
some degree,
depends
on
the
frequency
of
the
phenomenon
predicted.
Violent
crimes,
for
exam-
pIe,
occur
less
frequently
than
property
crimes
and
thus
tend
to show
lower
levels
of
predictability.
Some
law
and
criminal
justice
writers have
sug-
gested
that
because
of
the
presence
of
false
positives
in
predictive
statements,
statements
and
decisions
having
a
predictive
basis
should be
avoided
in all
dealings
with
offenders.
In
other
words,
these
writers
assert
that
reference
should
be
made
only
to
the
past;
it
is
improper
to
give
consideration
to
the
possible
outcome
of
a
decision.
This
is
the
position
taken
by
advocates
of
the
"just
deserts"
theory.
It
must
be
realized,
though,
that
despite
the
fact
that
prediction
is
believed
to
have
been
avoided
if
it
is
not
directly
invoked
in
the interpre-
tation of
previous
events,
it
may,
nonetheless
be-
come
involved
(in
some
ways
not
completely
un-
derstood)
in
the
definitions
of
"culpability."
Whether
prediction
really
is
avoided
is
not,
of
course,
the
major
issue
of
concern.
What
is
of
concern
are
the
qualities
of
the
decisions,
whether
or
not
they
make
use
of
"predictions."
Humans
seem
to
be
anticipating
creatures,
and
there
is
difficulty
with
the
"just
deserts"
proposition
that
this faculty,
for
ethical
reasons,
should
be
inhibited.
EVENTS AND
PERSONS
Individuals
cannot
validly be
classified
as
"dan-
gerous"
or
"not
dangerous,"
but
their
crimes
can
be.
The
commission
of
a
crime
is
an
historical
event
that
can be
analyzed
with
precision.
On
this
basis,
one
might argue
that
thought
should
be concen-
trated
upon
definitions
of
those
kinds
of
behaviors
(not
persons)
that
require
restraint.
Of
course,
the
person
(who-prediction
or
not-will
continue
into
the
unknown
future)
will
be involved
in
any
such
restraining
situation.
But
the
logic
underlying
the
disposition
of
the
case
would
be
independent
of
judgments
about
personality.
Although
a
crime
may
be
contemplated,
legally
defined,
and
discussed
independently
of
the
of-
fender,
in
operational
terms
the actor cannot
be
separated
from
the
act.
However,
one
can
discrim-
inate
between
micro
and
macro
models
in terms

CAREER
CRIMINAL
SYMPOSIUM
closely
analogous
to
those
of
economic
theory.'
One
could
propose
methods
of
dealing with
the
issues
of
crime
that
do
not
involve
a
clear
concept
of
the
individual
offender.
The
focus
instead
would
be
placed
upon
the
features
of
situations
common
to
many
crimes.
In
this
manner,
it
might
be
possible
to
take
action
about
crime
(macro
models)
inde-
pendently
of
action
attempting
to deal
with
the
individual
offender (micro
models).
However,
this
type
of
approach
remains
relatively
unexplored.
MEASURES
OF
THE
SERIOUSNESS
OF
CRIMES
Researchers who
have
attempted
to
provide
scales
of the
seriousness
of
crimes
have
relied
upon
ratings
by
samples
of
assessors.
4
A
single
dimension
of
"seriousness"
has
been
postulated
in
which
the
act
is
described
without
any
reference
to
an
actor.
A
nebulous
generalized
actor
must be
assumed.
However,
it
is
doubtful
that
the
people
who
are
recruited
as
assessors
find
it
possible
to
imagine
a
disembodied
act. Raters
probably
fit
a
stereotyped
actor
to
each
instance
of
crime presented
in
the
sample
of
acts.
This
produces
an
unknown
sample
of
actors
(offenders)
who
have
been
supplied
by
the
imagination
of
the
assessors
making
the
ratings
of the
crimes.
For
example,
the
crime
stimulus
"theft
of
$I.00"
poses
the
image
of
the
kind
of
person
who
could
or
would
steal
$1.00;
and
the
stimulus, "theft
of
$10,000"
poses
the
image
of
the
kind
of
person
who
could
or
would
steal
$10,000.
The
two
offenders
"supplied"
to
match
the stimu-
lus
are
not
likely
to
be the
same kind
of
person.
Thus,
assessments
of
the
seriousness
of
acts
may
well
be
confounded
with
attributes
of
offenders
as
ascribed
by
raters.
The
implications
of
this
may
be
of
considerable
significance
for
all
seriousness
scales.
3The
distinction
between
micro
and
macro
models
may
be
illustrated
by
noting
that
the
skill
necessary
to
operate
a
village
grocery
store
is
not
an
adequate
quali-
fication
for
a director
of
the
budget.
The
economic
theory
that
leads
to
success
in
a
small
business
is
not
the
same
as
the
theory
that
is
required
to
guide
a national
policy.
The
domestic
detail
of
individual
purchases
(so
much
appre-
ciated
in
a
village
community)
does
not
rise
to
the
level
of
the
kind
of
sophisticated
abstraction
necessary
to
form
the
basis
of
a
national
marketing
policy.
Perhaps
too
much
debate
in
jurisprudence
is
at
the
level
of
the
village grocery
store
theory.
Consider,
for
example,
the
fondness
for
references
to
"little
old
ladies"
and
to
the
man
who
rides
the
local
omnibus.
This
sort
of
homeliness
characterizes
many
opinions
on
crime
policy
and
legal
philosophy.
4
T.
SELLIN
&
M.
WOLFGANG,
MEASUREMENT
OF
DELI-
QUENCY
(1964).
Theories
of
"just
deserts"
propose
that
the
sever-
ity
of
the
penalty
should
match
the
seriousness
of
the
crime,
but
even
according
to
these
theories,
the
measurement
of
seriousness
must
include
some con-
cept
of
culpability,
an
offender-related
character-
istic.
Indeed,
a
crime
ceases
to
be
defined
as
such
if
the
culpability
of
the
offender
is
reduced
to
zero.
But
such
reduction
of
culpability
does
not,
or
does
not
greatly,
modify
the
seriousness
of
the
harm
done.
The
transition
from
crime
to
harm
and
the
relationship
between
the
concepts
of
culpability
and
seriousness
need
greater
consideration
and
re-
search.
IMPROVEMENTS
IN
PREDICTION
There
are
many
reasons
why
the strategy
of
avoiding
predictive
statements
may not
be
realistic
or
even
desirable.
If
one accepts
this position,
it
is
necessary
to
confront
the
problem
of
false
positives
and
consider
what
is
ethical
under
conditions
of
uncertainty.
The
replacement
of
probability
by
some
subjective
certainty
is
not a
satisfactory
ap-
proach.
First, one
should estimate
the
magnitude
and
probable
impact
of
the
false
positive
prognosis,
both
upon
individuals
and
upon
the
social
and
legal
system.
The
present
position
is
that
for
every
person
correctly
identified
as
dangerous
(likely
to
commit
another
crime
against
the
person),
six
oth-
ers
will
possess
the
same
predictive
profile.
This
was
the
conclusion
of
a
very
thorough
and
intensive
testing
involving
a
large
sample
of
young
offend-
ers.
5
Given
this
background,
one
can
consider
whether
this
level
of
precision
can
be
increased
and,
in
addition,
what
other
modifications can
be
made
to deal
with
the
difficulties
arising
from
imprecise
judgments.
It
is
quite probable
that
the
precision
of
predic-
tion
methods
can be
improved significantly.
There
are
three
areas
where
it
would
be
necessary
to
invest
effort:
(a)
the
basic
data,
(b)
methods
of
input
of
basic
data
to
analytical
systems,
and
(c)
the
analytical
systems.
THE
BASIC
DATA
The data
used
in
making
predictions
can
give
rise
to
several
problems.
A first
problem
arises
from
the
fact
that
existing
case
papers,
the
basic
source
documents
for
information
used
in
predictive
stud-
ies,
do
not
seem
to
have
the
accuracy
necessary
to
5
Wenk
&
Emrich,
Assaultive
Youth,
9
J.
RESEARCH
CRIME
& DELINQUENCY
171
(1972).
1980]

LESLIE
T.
WILKINS
withstand
analyses
by
means
of
the
more
powerful
methods.
The
information
may
be
mixed
with
"noise"
as
well
as
redundancy.
It
therefore
may
be
worthwhile
to
explore
the
common
kinds
of
record-
ing
errors
and
how
they
influence
predictions
based
on
case
papers.
A
second
problem
arises
from
the
fact
that
some
items
of
information
may
be predictive,
but
"un-
desirable"
for
inclusion
in
an
analysis.
Race,
for
example,
may
be
such
an
item. Any
characteristic
that
the
individual cannot
change
voluntarily may
be
regarded
as
suspect
on
ethical
grounds.
The
correlations
may
reflect
the
position
of
the
individ-
ual
in
the
"real
world"
and
include
factors
quite
independent
of the
criminal
justice
processes.
Data
obtained
in
criminal
justice
research
may
reveal
injustice
that
cannot
be remedied by any
action
within
the
criminal
justice
system.
The
criminal
justice
system
is
not
operating
in
a perfect
society.
Where
data
problems
arise
from
these
sources,
one
must
ensure
that
legal
and
ethical considerations
are
permitted
to
outweigh
considerations
of
effi-
ciency.
A
third
problem
arises
from
the
fact
that
some
improvements
in
predictive
power
may
require
the
use
of
information
that
causes
an unjustifiable
intrusion upon the
personal privacy
of
an
individ-
ual.
But
it
is
also
possible
that
where
offenses
have
been
proven
against
an
individual
(prediction
of
recidivism),
some
of
the
rights
safeguarding
the
privacy
of
personal
information
are
diminished.
This
is
a
matter
that
jurisprudence
must
decide.
METHODS
OF INPUT
(CODING)
Methods
of
input
raise two
problems
for
predic-
tion.
First,
all
prediction
systems
have
used
data
with
a
fixed
time
base:
the
files
have been searched
for
information
in
one
operation,
and
these
data
have
been
coded
as
information
input
for
analyses.
Dynamic
procedures
of
data
recording
may
have
potential
for
more
efficient
prediction.
Where
in-
dividuals are incarcerated
(or
in
mental
hospitals),
it
is
possible
to
obtain
data
on
transitional
states.
The
problem
with
this
is
that
the
cost
of
such
data
collection,
as well
as
probable
ethical objections,
raises
concerns
that
may
outweigh
the probable
increments
in
predictability.
A
second
problem
stems
from
the
fact
that
some
important
information
may
be
lost
by coding
proc-
esses
which
tend
to
make implicit
assumptions
(e.g.,
A
+
B
=
B
+
A,
where
A
precedes
B
in
time).
Coding
stage assumptions
such
as
additivity
and
transitivity
are
not
removed
by
analytical methods
that
avoid
such
assumptions.
ANALYTICAL
SYSTEMS
Research
has been
directed towards
refining
the
statistical
methods
of
analysis,
and
work
in
other
fields
of
application
has
provided
new
techniques.
The
results, however,
have been
disappointing.
Little
or
no difference
can be
observed
between
the
power
of
quite
sophisticated
methods
(e.g.,
log
odds,
discriminant
function)
and
very
simple
sys-
tems
(e.g.,
point
allocation
as
in
the
Guidelines
of
the
United
States
Parole
Commission
and
unit
weights
as
used
fifty
years
ago).
It
is
possible
that
this
result
is
due,
in
part,
to
the quality
of
the
basic
data;
simple methods
are more
resistant
to "noise"
in
the
data
base.
Clearly, this problem
needs
in-
vestigation,
which
can
be
done
rather
simply
and
inexpensively
through
the
use
of
simulation
meth-
ods.
Although
it
is
known
that
all
methods
of
predic-
tion
provide
estimates
of
approximately
equal
pre-
dictive
power,
the
relationship
between
the
meth-
ods
as
they
apply
in
individual
cases
has
not
been
investigated.
The
different
methods
of
prediction
correlate
quite
highly
with
each
other,
but
not
perfectly.
Thus,
there must
be
some
proportion
of
cases
predicted
as
"failures"
by
one
method
but
as
"successes"
by
another.
It
is
possible
to
guess
at
the
nature
of
these
differences
through
the
use
of
math-
ematical
theory,
but
a
thorough
practical
exami-
nation
of
the
matter
seems
long overdue.
SOME
ADDITIONAL
PROBLEMS
IN
PREDICTION
AND
DECISION
The
prediction
of
recidivism
of
offenders
has
been
subject
to
a
wide
range
of
error.
But
a
much
greater
accuracy
has been
obtained
in
the
predic-
tion
of
the
decisions
of
authorities
in
the
criminal
justice
system
(e.g.,
judges,
parole
officers,
proba-
tion
officers).
Why
decisions
about
offenders
(such
as
sentencing
and
parole
revocations)
have proved
more
predictable
than
decisions
by
offenders
(to
commit
crime)
is
unknown.
A
cross-analysis
of
data
involving samples
of
these
two
classes
of
predictions
has
not
been
undertaken.
Indeed,
there
is
little
research
specifically
directed
at
methods
of
or
issues
in
prediction.
What
is
known
about
prediction
has
arisen
mainly
from
studies
in
which
prediction
methods
were
incidental
to some
other
focus.
Thus,
an
investigation
devoted
solely
to
methods
of
pre-
diction
might
prove worthwhile.
Many
authorities
assume
that
if
outcome
could
be
predicted
with
reasonable
accuracy,
it
would
be
a
simple
matter
to
determine
the
best
course
of
action.
But
it
is
questionable
whether
improved
prediction
techniques
would
provide
useful
infor-
[Vol.
71

CAREER
CRIMINAL SYMPOSIUM
mation
in
many
circumstances.
In
the
early days
of
prediction
in criminology,
it
was
thought
that
if
parole
boards
had
prediction
tables,
this
would
assist
their
decisions.
But parole boards
did
not
make
use
of
the
available
methods.
The
decisions
made
by
parole
boards
were
concerned
with
issues
other
than
"mere
prediction
of
recidivism."
This
became
clear
in
the
work
that
led
to the
develop-
ment
of
Guidelines
for
the
United
States
Parole
Commission.
While
the
guidelines
include
a
pre-
dictive
element,
it
is
the
lesser
of the
two
major
dimensions
considered;
the main
one
being
the
seriousness
of
the
instant
crime.
Modern
decision
theory
and
related
practical
methods
can
assist
in
decisionmaking
where
the
objectives
are
clearly
stated.
Prediction
methods
may
be
useful
or
even
essential
as
a
subset
of
other
analytical
techniques.
In
this
way,
it
may
be
pos-
sible
to
develop
more
efficient
and
ethical
rules
of
decision
and
procedure.
The
main
issue
today
is
not
how
to
make
predictions
(the
techniques
of
prediction
are
known,
and
there
already
are
good
ideas
for
improving
present
methods),
but
rather
why to
predict
and
when.
An
answer
to
the
ques-
tion
of
why
to
predict
will
help answer
the
question
of
what
to
predict.
It
also
may
be desirable
to
simulate
the
condi-
tions
of
decisionmaking
where
decisionmakers
are
required
to
respond
to
information
controlled
for
type,
probable
error,
and
situation.
Prediction
technique
should
not
be
seen
as
something
that
will
stand
up
on
its
own.
The
decision
environment
in
which
the
methods
of
prediction
are
to
be
embedded
is
also
an
essential
element
of
the
pro-
gram
of
research
and
assessment
that
is
now
re-
quired.
One
final
question
seems
to
be
of
outstanding
importance.
The
question
is
whether
prediction
should
be
considered
in
the
disposition
of
offenders.
This
question
is
related
to
many
subquestions
that
might
challenge
the
relationship
between
ethical
concerns
and
probability
or
degrees
of
belief.
But
in
the
final
analysis,
the
theory
of
"just
deserts"
does
not
appear,
either
practically
or
ethically,
to
have
disposed
of the
necessity
of
prediction.