A group of academics is stirring up trouble in the world of energy efficiency. Researchers at the University of Chicago and University of California Berkeley recently released a controversial working paper that highlights the apparent cost-ineffectiveness of the federal Weatherization Assistance Program (WAP) in Michigan. The research found that energy efficiency upgrades in homes that participated in the program were poor financial investments—costing roughly twice the value of energy saved over time. This finding contradicts the long-held belief that efficiency is one of our best financial investments.

The research has been critiqued soundly by ACEEE, NRDC, Martin Holladay from Green Building Advisor and discussed in detail by the Illinois Institute of Technology. Some criticisms of the work point to the research itself; other criticisms rightly focus on the broad assertions made from the study (by the press and the researchers themselves). Many of the criticisms are fair, but I believe we can still learn from this work.

What Exactly Did the Study Find?
The study’s core findings are these:

1. Homes that participated in the WAP program reduced energy consumption by an average of 10 – 20%. This is important information, as the impact from weatherization efforts is not often quantified rigorously. A 10 – 20% reduction in energy consumption is not trivial, and is consistent with a previous study by researchers at Lawrence Berkeley Labs.
2. The costs of the weatherization interventions were approximately double the value of the energy saved over time. The researchers determined that weatherization in this particular program was a poor investment even when carbon-reduction costs were considered.
3. There was no evidence of the often discussed “rebound effect” for homeowners that participated in the program; creating more efficient envelopes and heating systems did not lead home occupants to turn up the heat and negate savings.
4. It was difficult to get homeowners to participate in the weatherization program, even though it was free. Average participation rates in the study were 1%, while homeowners receiving encouragement in the form of house visits and phone calls participated at a paltry 6% rate.
5. The energy models that underlie the energy savings projections for the program (in this case NEAT, the National Energy Audit Tool) significantly over-estimated energy savings associated with efficiency interventions. The savings projected by the models were approximately 2.5 times the savings actually observed.

It’s important to remember that these findings are for a single federally funded program focusing specifically on low-income households. The study examined a small slice of a much larger and more complex system. It would be foolish indeed to assume that all efficiency efforts have poor returns and are not worth the initial investment—especially when we have research to the contrary from industry experts, government and academia. Please refer to the critiques referenced above for more on this theme.

With that said, we can learn from this research. As someone that has studied energy efficiency and works in the industry, my first instinct was to criticize the paper and simply dismiss it. Fortunately my second instinct was to read the study in detail, with a mind as open as it would allow. What I discovered is that there is value in this work, although perhaps not exactly what the authors intended.

We Need to Measure and Verify Actual Performance
The research highlights the need for us to objectively measure the performance of our buildings and the impact of efficiency improvements. It’s not enough to weatherize a home and congratulate ourselves on a job well done. We need to measure the actual energy consumed over time to ensure homes are performing as expected. We can’t do this with modeled energy consumption or projections of energy use. If our goal is to reduce energy consumption, we need to measure it.

And speaking of models, we would do well to remember that they are necessary simplifications of the real world and never perfect. Model inaccuracy is nothing new. Systems expert Donella Meadows wrote in Limits to Growth (The 30-Year Update) that “All models, including the ones in our heads, are a little right, much too simple, and mostly wrong.” This particular study found that NEAT models overestimated natural gas consumption by 25%, and several other studies have found that models designed to predict energy consumption based on the physical characteristics of the home are often inaccurate. Models are useful tools, but we should not assume they can replace the measurement and benchmarking of actual home performance.

Subsidizing Efficiency Doesn’t Work
The researchers in this study randomly assigned homes to a group that was encouraged to participate in the WAP program. Encouragement consisted of over 32,000 phone calls and nearly 7,000 home visits. Even with this encouragement, participation in this free program was only 6%, at a cost of more than $1,000 per weatherized household. We see similar statistics in my home state of Massachusetts, where the Mass Save program allows residents to sign up for free energy audits and steeply subsidized insulation and weatherization work. Statewide participation rates in 2013 and 2014 were approximately 2.9% and 3.1% respectively. It’s reasonable to assume that low participation rates are at least partially responsible for proposed state legislation that would require an energy audit for every home sold in the state. I fully support this legislation as I do not believe subsidizing energy efficiency is working quickly enough.

Why is it that we can’t give away free efficiency upgrades? These aren’t bad investments as the economists suggest (efficiency is without question a good investment if it’s free), it’s simply that energy prices aren’t high enough to motivate most of us to take action. The price of energy in the U.S. is an incomplete form of feedback—one that doesn’t include the social or environmental costs of fossil fuel extraction. We take action based on the information we have. If energy is affordable, why spend effort or money to conserve?

We need to provide clear and timely feedback to consumers and leverage the market to increase demand for efficiency. In order to do this, subsidies on fossil fuels need to be removed. A price on carbon must be implemented. We should measure and benchmark the performance of homes in near real-time. And every home in this country should have an efficiency rating to help home buyers and renters objectively assess the efficiency of different homes. Any combination of these actions would drastically increase demand for efficiency.

Let’s Work Together
The research presented by the economists in this paper is not perfect, but is valuable. We can learn from it, just as the academics can learn from some of the many criticisms of the paper if they so choose. Let’s hope that efficiency experts and academics can work together in the future to improve energy efficiency research. Personally, I want to know where we can improve and where we should focus our efforts—even if it makes me a little uncomfortable to hear it.