Last year, Margo created a survey to gauge attitudes toward bullies
by their middle school peers. The
questions took months to draft and vet for bias. The Institutional Review Board
checked her protocol against NSF rules for the ethical treatment and informed
consent of human subjects. The survey
was tested on a dozen middle schoolers.
Three of them misunderstood the wording of question 17. Six more rewrites.
Next Margo sought guinea pigs. She needed a representative mix of
ethnicities and genders. Some urban
schools, some rural. Private and
public. Bible Belt South and affluent
Northeast. Six weeks of research,
e-mails, and snail mail. The begging
phone calls to sixty principals: “Yes
sir. Completely anonymous. I know, sir.
You’re absolutely right; the school day is precious. I promise. Fifteen minutes, max.”
Sixteen months after her thesis committee pitch, Margo rips
open a manila folder. Inside there’s raw data collected by the Survey Research Center from 600 7th graders. First up:
Grover Cleveland Middle.
Caldwell, NJ. She gets no further
than the demographic summary before the first wave of nausea overtakes
her. The profanity starts low in her
belly and works its way up.
The third respondent is seven feet tall, 400 pounds. A one-legged, Native American bisexual, he
has three kids. He’s served two
sentences for dealing crack. And he’s
only 14.
“An outlier. Please,
god, let him be an outlier,” Margo starts to pray. But the improbable drug dealer has
friends. Extraordinary friends. Hermaphrodites, award-winning actors,
schizophrenics, victims of incest, murderers, spelling bee champions,
mathematical savants – a playful group of young teens who have had fun at her
very considerable expense. They have
invented personae as comic exercise.
Darkened ovals on their response sheets form geometric patterns. “Dammit,
dammit, dammit,” the grad student
chants under her breath. “They’re useless.” All she can think about is how far in debt she
is, the price tag for a doctorate in Educational Psychology.
 |
| Oh, Grow Up! Tom Hiddleston as Loki, Norse God of Mischief (in your pick of pretty much all the recent Marvel movies) |
Maybe Margo calms as she scans the remaining responses,
gradually trusting that truth will emerge from time-tested methodology. Or maybe she becomes a poster child for the
research of Joseph P. Robinson-Cimpian from the University of Illinois at
Urbana-Champaign.
Last week Robinson-Cimpian published Inaccurate Estimation of Disparities Due to Mischievous Responders. His
common sense observation: adolescent
trouble-makers like to screw with people.
“Mischievous responders are ubiquitous in adolescent research using self-administered
questionnaires and can, even in small
numbers, lead to inaccurate conclusions that substantively affect research,
policy, and public discourse regarding a variety of disparities.”
The implications are
enormous [and obvious], particularly in a world that offers little objective
proof of achievement. Should high
schools continue instruction in the fine arts?
Should an assistant professor be promoted? Are Facebook bullies a threat? Do the assignments of a given class demand a fair
work load? Ask an adolescent.
As a group,
adolescents are not especially evil. But
they don’t generally feel the weight of surveys. They don’t know they’re used to hire and
fire, to fund and defund. Or maybe they do know.
Maybe anonymity frees a (sometimes-playful) show of power.
Races are won
against clock and competitor. Walls are
built to satisfy a carpenter’s level. Projects
are completed on-time and under-budget. The
success of many human endeavors may be measured by objective means. Where the measure is anonymous and self-reported,
however, it’s possible the resulting data and evaluation deserve greater and
more frequent scrutiny.
No comments:
Post a Comment