ENOVA seminar

In addition to getting my name in the paper last Thursday, I also attended a seminar by Norway’s national energy conservation advisory board ENOVA. The theme was how to identify the correct instruments for changing habits and behaviour with regards to saving energy. The whole day was devoted to a presentation by a researcher from the Dutch version of ENOVA, SenterNovem, who presented their tool for assessing the correct instrument for affecting change, the Instrument Planner.

The basic idea is that you follow a set procedure, by identifying a target group and a desirable outcome. Then, after answering a questionnaire about different aspects of your target group, the Instrument Planner gives you a list of possible policy instruments with recommendations to choose from. To be honest, I found the Instrument Planner itself to have some serious flaws, but there is some interesting insights to be had from this. I’m going to illustrate the process for you, but first I need to explain the underlying assumptions of the setup.

The government has a number of ways to influence our behaviour, ranging from direct laws and regulations to “softer” approaches like coaching or demonstrations. Naturally, these instruments have different impact on the people to be influenced, and appeal to different parts of behaviour. For example, financial incentives help to enable behaviour by freeing up resources, but might not be a strong motivating factor in the same way using more moral instruments might. So our Dutch researcher has done a literature review on how different instruments affect different parts of what makes up behaviour, and has come up with a chart of what works and what doesn’t:

Coloured fields work, blank don't

Coloured fields work, blank don't

Notice how this is now a binary relation, meaning that if a field is blank, an instrument has no effect at all on that determinant of behaviour. We’ll come back to that shortly. This nice chart has been converted into a handy web site, where people can answer twelve questions about the target group, where each possible answer is given a value. We see the first question here:

Answers range from 0 to 3 points

Answers range from 0 to 3 points

After the twelve questions have been answered, the Instrument Planner sums up your scores:

Here the answers are summed up

Here the answers are summed up

…and comes out with a list of instruments you can choose from, rated after their effectiveness.

The final results

The final results

Now, there’s nothing wrong with a lot of this. It’s standard quantitative procedure to assign values to answers, and try to analyze the relationship between them through this. And in fact, the answers are not half bad. However, there are two major problems with this approach. The first is methodological: there are some questionable choices here. As we saw above, the first operation of the designers of the Instrument Planner was to break down motivations and determinants into binary relations. This means that from here on, there is no room for other influences. After this, they go on to assign numerical values to statements that the user of the Planner check off in the questionnaire. This is a source of possible error, since the points are simply summed up when all the questions are answered. Is “No, [the target group] is not aware of [the goal]” worth exactly three times as many points as “Yes, but [the target group] is not well informed”?

The second problem with this approach is the rhetorical effect it has. In spite of any methodological problems related to the Instrument Planner, the final result will prove almost impossible to dispute once it has been produced. Numbers and “hard facts” have a powerful grip on planner and policy makers, and any tool that promises to deliver answers to complicated relations will be used, regardless of any more theoretical issues. The question is if this is a challenge to the less directly policy oriented among us. Should we perhaps adopt a more clear-cut, less problematizing attitude towards policy recommendations in order to gain influence (because we do, after all, have ideas about what works and what doesn’t), or is that sacrificing academic integrity for power?

Advertisements

2 Responses

  1. Hm, interesting. Another methodological point is that many of the “instruments” are highly qualitative. Coaching, for instance, or “personal advice”, I would guess, are dependent on the pedagogical skills of the the one providing it, so quantifying past experiences with that “instrument” sounds, at least, more difficult than quantifying experiences with, say, subsidies..

    I Guess there is no quick answer to your last question, but in my opinion, yes, we should definitely try to adopt a more “clear cut” approach, perhaps even through developing tools similar to this (although this would leave us vulnerable to attacks from other academics…). It would probably increase our direct influence on actual policy, and indirectly I think it could be a good source of funding for more “proper” academic research. (“…this project will help us increase the accuracy of our already highly acclaimed KULT-O-MATIC policy bot…”).

    The downside, of course, is the chance of being proven wrong. If your suggested policy fails, it’s tough to explain, so I guess it’s easier to just stay out of it…

  2. You’re quite right about the other methodological point. I think one of the reasons that moving towards more policy recommendation is so hard lies in exactly this. In order to provide the kind of advice policy makers want, we would have to ignore a lot what to me at least constitutes good science. Of course, you can hold your nose and do it anyway, but I somehow have problems envisioning myself making the methodological shortcuts in the first place. But who knows?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s