Did you ever receive a grade of “B” in a high school math class? How would have felt if you had? Would you have changed your behavior, maybe studying more (or less)? The answer is likely different for each of us; some might have felt happiness or relief, others disappointment and a determination to work harder. The grade would have affected each of us differently.

Might this be true for home energy scores as well?

Here at Resynergy Systems, we’re not only interested in sharing OPEN operational efficiency ratings with homeowners to help them understand how their homes are performing, we’re also interested in understanding the effect of sharing a rating with a homeowner—does the score itself impact behavior?

We had a theory—based on research from MIT—that sharing an operational rating during a home energy assessment would influence a homeowner’s decision to invest in efficiency upgrades. In order to test this theory, we decided to conduct an online experiment based on self-reported intention.

There were some surprises in our findings, but the results strongly supported our hypothesis that homeowners who received an operational rating would be more likely to invest in energy efficiency.

What did we do?
Our main hypothesis was that homeowners who receive a less than stellar operational score for their home would be more likely to invest in efficiency upgrades (e.g. air sealing or insulation work) than those who do not receive a score. We also thought this effect might be stronger for homeowners who were concerned about their energy consumption for environmental or cost reasons. We wanted to conduct a randomized control trial to confirm or reject our theory.

We discussed our ideas with Trisha Shrum, an environmental economist from Harvard (and co-founder of DearTomorrow) who provided feedback and guidance on structuring the research. She also pointed us to Amazon’s Mechanical Turk, an online marketplace used by academics and businesses for research and virtual work services. “MTurk” is a global network of web-connected workers, who review and can decide to accept Human Intelligence Tasks (HITs)—virtual work requests—for pay. In our case, we paid Turk workers a small amount of money to participate in a survey we constructed for this research.

The survey walked each participant through a hypothetical home energy assessment (i.e. audit), with a real-time randomization process that allowed a subset of participants to receive an OPEN score for their home as part of the assessment. We settled on sharing a conservative, above average score of 64 (out of 100, where 100 indicates most efficient) for the study, which we presented visually:

OPEN rating comparison graphic

Only U.S. residents were allowed to participate, and our goal was 400 responses. We had 401 responses in total, with 16 that were incomplete or failed to meet our data quality standards. This left us with unique and clean responses from 385 participants in 47 different states (if you are curious, we missed Idaho, Nebraska and South Dakota). Just under half (48%) of our participants received the OPEN score during their virtual assessment (all receiving the same score of 64, as shown above). These respondents were also asked a series of questions to help us understand how they felt about the score. The remaining participants (52%) did not receive a score and were not aware that the other participants had received one. All respondents were asked the same set of questions about their willingness to invest in efficiency after the audit, and all were presented with the same efficiency upgrade options with deeply subsidized costs: leave the home as is ($0), air seal the home ($100) or insulate and air seal the home ($500). We then asked a series of questions about energy use attention and concern, along with a handful of demographic questions.

OPEN rating research structure

What did we find?
We first looked at the entire population of respondents. As expected, we found that homeowners receiving an OPEN score were more likely to say they would invest in air sealing or insulation work, with a moderate increase of 4.2%. Although we could not confirm our hypothesis as this result was not statistically significant (p=0.433), it was promising.

We then analyzed three subgroups within the overall population based on their responses to our survey questions. These groups represent respondents who indicated they:

• Pay close attention to utility bills (the “attention” group)
• Are very concerned about reducing energy use (the “concern” group)
• Feel utility bills are expensive (the “expensive” group)

We believe these subgroups represent homeowners that are concerned about their energy consumption for economic or environmental reasons. When we investigated the effect of a score on these groups, we found that participants in each were over 9% more likely to say they would invest in efficiency when they received an OPEN score. However, the results were only statistically significant (α level .05) in one group: those that identified as paying close attention to utility bills.

A little analysis revealed that the “attention” group encompasses most of the other two subgroups. Of the 250 respondents falling into the “attention” group, 83% also identified as being very concerned about reducing energy use at home, felt their utility bills were expensive, or both. (This is slightly higher than the 75% of respondents in the “expensive” group that belong to either or both of the other two groups.) We believe it is these homeowner concerns that are driving the attention paid to utility bills.

Breakdown of respondents concerned about their energy use

The effect of an OPEN score on the “attention” group was statistically significant and compelling. This group represents 65% of our total sample population, and these homeowners were 11.8% more likely to say they would invest in efficiency upgrades when receiving a score (p=0.046).

group	% more likely to invest when OPEN score received	p-value	size (n)	% in Attention group
Attention	11.8%	0.046	250	100.0%
Expensive	11.0%	0.085	250	72.4%
Concern	9.4%	0.169	126	91.2%

These results provide preliminary evidence that receiving an OPEN score may motivate homeowners to invest in efficiency—especially if they pay close attention to their utility bills (i.e. are concerned about energy consumption) to begin with.

The survey provided other important insights:

• 72% of respondents wanted to know how their energy use compared to other homes. This suggests that people are both interested, and do not currently know, how their energy use compares to others.
• 82% of respondents indicated they were completely comfortable (57%) or somewhat comfortable (25%) sharing utility information with an energy auditor. This contradicts the belief that homeowners are concerned about privacy and reluctant to share energy data.
• 90% of respondents who received a score indicated they would take action to improve their OPEN rating if they received a score of 25 or less. Lower scores likely provide a stronger incentive for homeowners to reduce consumption and invest in energy efficiency.

Does it matter?
These findings demonstrate that simply sharing an efficiency score with a homeowner may encourage investment in energy efficiency. Although our experiment was based on an operational score, it is not unreasonable to think we would see similar results from an asset score if shared in a context meaningful to homeowners. One benefit of an operational score, however, is that it is a reflection of a homeowner’s actual energy use and not simply the result of energy modelling of the physical structure.

These results may be especially significant for residential efficiency programs, as the people most affected by the OPEN score indicated they pay close attention to their bills; these are likely the homeowners requesting energy audits and taking advantage of efficiency program offerings. Efficiency programs should consider targeting this population, as sharing an operational score with this group may have substantial benefits with relatively little cost.

Our findings also point to the idea that an OPEN score does not have the same meaning for all people. Although this needs further study (and data), our research suggests that receiving an OPEN score may not have the same effect on homeowners who are unconcerned about energy consumption. In fact, people who indicated they do not pay close attention to their utility bills and received the OPEN score of 64 were 10.9% less likely to say they would invest in efficiency in this study. These results were not statistically significant (p=0.3), but in the same way that a “B” grade has different significance for individual students, it is possible that less-concerned homeowners view a score of 64 as a positive rather than negative. Again, this needs further study, but these results are interesting and unexpected.

Summary
Our initial research results are promising, and we plan to continue our study by providing a randomized set of homeowners their actual OPEN ratings, based on their own energy use, during home energy assessments. We expect to find that homeowners receiving an unsatisfactory energy rating will be more likely to spend real money on efficiency improvements to their home, and that the determination of unsatisfactory is subjective. The actual score received will strongly influence the likelihood of taking action, but the magnitude of the effect will depend on the mindset of the person receiving it.

The idea that sharing an operational efficiency rating may influence homeowner behavior has implications beyond home energy assessments. Other related research questions we want to answer:

• Will sharing OPEN ratings as part of outreach efforts motivate homeowners to request home energy assessments?
• What is the effect of sharing an OPEN rating with a homebuyer or renter as part of a real estate transaction?

More to come.

 

Author’s note: If you have questions, ideas for further research or would like to collaborate with us on research in the field, please contact us.