Judge and jury

One side effect of submitting papers to conferences is having to referee other contributions. Since Åsne and I are writing about consumers in the electricity market, we were asked to referee a paper on the possibility for reducing energy consumption through low-level behavior changes. After looking at it, I’m not sure what to do.

The authors are trying to calculate the possible range for energy conservation by identifying a whole range of possible measures (changing lightbulbs, inflating the car tires, turning down the thermostat and so forth), identifying a “range of potential savings” by surveying the literature (22 different studies), and then calculating the “likely range of participation rates”[1]. This will give a measure of the expected savings (the answer? “23 % expected savings”). To test this measure, they then ran a Monte Carlo simulation with a 1000 iterations to see if the measure found in the previous point was more or less on target (end result: 22 % savings with a +/- 29 % interval). Conclusion: If people change behavior, we can save a fourth of our energy. This is then presented as proof that it is not only technology innovation that drives energy conservation.

I find this whole paper to be a bit baffling. To take first things first, it’s very brief, only laying out its test, method, rationale and discussion in sketch form. Still, they do explain what they have done and what their framework is, so I guess the rest can be filled in in time for the conference. The strangest thing is more the idea itself: calculating not the upper limit of conservation, but to the exact percentage how consumers will behave in energy conservation. I just don’t see how this is expected to give probable results. Even if all the assumptions are nicely laid out in the final paper, I still don’t understand how aggregating behavior in this way can produce meaningful results. In addition, they present it as a major finding that there is something to be gained from behavior change. To their credit they do mention that further research into how people are “more than economically rational actors” (hey, that’s what we’re working on!. Is this really dynamite?

So what do we do? I mean, it seems harsh to reject it (the main author, who goes by his nick-name in the references, also work for the US Environmental Protetction Agency), but I don’t see the value in this. Plus, referencing some percentages, asking your colleagues and slapping together a Monte Carlo test seems kind of lazy. But I guess more work can be done to improve it a lot…

[1] How? “In review with [energy policy NGO] professional staff”. They asked their colleagues for their, no doubt professionally founded, best guesses. But still…


One Response

  1. I am sure you have found some kind of solution for this by now. But my gut instinct is that if something makes you go “WTF?” then it will likely get lots of other people to go “WTF?”.

    Since I am still a conferance n00b I wont claim to know how they work, but it seems to me that already too much sub-quality work is presented and there is no need to add to it.

    On the other hand: Scratch my back and I will scratch yours. I am sure I am going to have to rely on fellow cooworkers to let me perform with fail on my occations in the future.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s