What Do We Learn from Narrow Randomized Studies?

(by Andrew Gelman)

Under the headline, “A Raise Won’t Make You Work Harder,” Ray Fisman writes:

To understand why it might be a bad idea to cut wages in recessions, it’s useful to know how workers respond to changes in pay—both positive and negative changes. Discussion on the topic goes back at least as far as Henry Ford’s “5 dollars a day,” which he paid to assembly line workers in 1914. The policy was revolutionary at the time, as the wages were more than double what his competitors were paying. This wasn’t charity. Higher-paid workers were efficient workers—Ford attracted the best mechanics to his plant, and the high pay ensured that employees worked hard throughout their eight-hour shifts, knowing that if their pace slackened, they’d be out of a job. Raising salaries to boost productivity became known as “efficiency wages.”

So far, so good. Fisman then moves from history and theory to recent research:

How much gift exchange really matters to American bosses and workers remained largely a matter of speculation. But in recent years, researchers have taken these theories into workplaces to measure their effect on employee behavior.

In one of the first gift-exchange experiments involving “real” workers, students were employed in a six-hour library data-entry job, entering title, author, and other information from new books into a database. The pay was advertised as $12 an hour for six hours. Half the students were actually paid this amount. The other half, having shown up expecting $12 an hour, were informed that they’d be paid $20 instead. All participants were told that this was a one-time job—otherwise, the higher-paid group might work harder in hopes of securing another overpaying library gig.

The experimenters checked in every 90 minutes to tabulate how many books had been logged. At the first check-in, the $20-per-hour employees had completed more than 50 books apiece, while the $12-an-hour employees barely managed 40 each. In the second 90-minute stretch, the no-gift group maintained their 40-book pace, while the gift group fell from more than 50 to 45. For the last half of the experiment, the “gifted” employees performed no better—40 books per 90-minute period—than the “ungifted” ones.

The punchline, according to Fisman:

The goodwill of high wages took less than three hours to evaporate completely—hardly a prescription for boosting long-term productivity.

What I’m wondering is: How seriously should we use an experiment on one-shot student library jobs (or another study, in which short-term employees were rewarded “with a surprise gift of thermoses”), to make general conclusions such as “Raises don’t make employees work harder.”

What I’m worried about here isn’t causal identification–I’m assuming these are clean experiments–but the generalizability to the outside world of serious employment.

Fisman writes:

All participants were told that this was a one-time job—otherwise, the higher-paid group might work harder in hopes of securing another overpaying library gig.

This seems like a direct conflict between the goals of internal and external validity, especially given that one of the key reasons to pay someone more is to motivate them to work harder to secure continuation of the job, and to give them less incentive to spend their time looking for something new.

I’m not saying that the study Fisman cited is useless, just that I’m surprised that he’s so careful to consider internal validity issues yet seems to have no problem extending the result to the whole labor force.

These are just my worries. Ray Fisman is an excellent researcher here at the business school at Columbia–actually, I know him and we’ve talked about statistics a couple times–and I’m sure he’s thought about these issues more than I have. So I’m not trying to debunk what he’s saying, just to add a different perspective.

Perhaps Fisman’s b-school background explains why his studies all seem to be coming from the perspective of the employer: it’s the employer who decides what to do with wages (perhaps “presenting the cut as a temporary measure and by creating at least the illusion of a lower workload”) and the employees who are the experimental subjects.

Fisman’s conclusion:

If we can find other ways of overcoming the simmering resentment that naturally accompanies wage cuts, workers themselves will be better for it in the long run.

The “we” at the beginning of the sentence does not seem to be the same as the “workers” at the end of the sentence. I wonder if there is a problem with designing policies in this unidirectional fashion.

Advertisements

0 Responses to “What Do We Learn from Narrow Randomized Studies?”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




About

The Statistics Forum, brought to you by the American Statistical Association and CHANCE magazine, provides everyone the opportunity to participate in discussions about probability and statistics and their role in important and interesting topics.

The views expressed here are those of the individual authors and not necessarily those of the ASA, its officers, or its staff. The Statistics Forum is edited by Andrew Gelman.

A Magazine for People Interested in the Analysis of Data

%d bloggers like this: