Comments on new working paper ‘Do Energy Efficiency Investments Deliver? Evidence from the Weatherization Assistance Program’

By Brent Stephens on June 24, 2015

Economists Meredith Fowlie and Catherine Wolfram at the University of California, Berkeley and Michael Greenstone at the Energy Policy Institute at the University of Chicago (EPIC) have published a new working paper on the economics of residential energy efficiency retrofits, in which they present some discouraging results:

I first read the short research summary from EPIC, “Costs of Residential Energy Efficiency Investments are Twice their Benefits: Implications for Policy,” and found some surprising conclusions that led me to dig deeper into the actual working paper, “Do Energy Efficiency Investments Deliver? Evidence from the Weatherization Assistance Program.”

In this paper, the authors, somewhat amazingly, conducted a randomized controlled trial using more than 30,000 homes in the state of Michigan that were taking part in the country’s largest residential energy efficiency program, the Weatherization Assistance Program (WAP). For those who aren’t familiar, the Federal WAP provides funding to states, who then provide funding to local agencies and nonprofits, to coordinate the application of energy saving retrofits to low-income homes. The stated goal of WAP is to “reduce the burden of energy prices on the disadvantaged.” While it’s technical an energy policy program, it’s largely a means to reducing the burden of energy bills on those whom they most affect. Homeowners typically receive about $5000 worth of retrofits (e.g., improved air sealing, insulation, window replacements, and furnace replacements), and most engineering analyses predict substantial energy savings will result, saving homeowners money each year.

The researchers were curious about two main questions: (1) How do energy efficiency investments perform in the real world compared to model predictions?; and (2) Do residential energy consumers change their behavior in response to energy efficiency improvements (i.e., is there a rebound effect)?

To answer these questions, the authors took a sample of 30,000 WAP-eligible homes in Michigan and randomly selected 25% of the households to serve as a treatment group wherein they were strongly encouraged to apply for the retrofit program (it sounds like these homeowners were hounded on the phone and by in-person visits to get people to sign up — it can be tough to recruit for these things!). Workers went out to the recruited treatment homes and collected energy bills and other data. The remaining 75% of homes were grouped into a control group wherein they were able to apply for the program if they chose to, but they weren’t hounded and didn’t get any additional help in completing their application.

They were able to gather monthly natural gas and electricity consumption data over a period of about 6 years, making sure to collect at least 2 years of pre-retrofit data for all weatherized homes. This data set allowed them to address question #1 about the energy and cost savings due to WAP retrofits. As a side note, I was pleased to see that the authors focused on both gas and electricity in the heating-dominated climate of Michigan, as opposed to the recent work on the impacts of building codes in California by Erik Levinson at Georgetown and NBER, which received a lot of press after his appearance on Freakonomics in February 2015 — one of the key issues with that work was that he focused only on electricity savings, which is something that early California building energy codes didn’t really even address, as described here by David Goldstein at NRDC).

The authors also randomly selected a subset of weatherized and un-weatherized homes to collect a bunch of spot measurements of thermostat set points and indoor temperatures during cold days (below 45F) in 2013 to address whether or not occupants were keeping their homes any warmer. This was designed to answer question #2 about the rebound effect of energy savings (i.e., are people now just keeping their homes warmer because they can afford to?). This was a clever addition to the study in my opinion.

When the authors looked at the overall data set, they found that participation in WAP reduced energy consumption by about 8-10%, on average (lumping both natural gas and electricity use together). Most of these savings appeared to stem from reductions in natural gas usage (data are buried in the appendix). When they multiply these savings by natural average residential retail prices of gas and electricity, the authors estimate that the average energy savings of WAP retrofits is about $155 per year. Given that the national average residential energy bill (electricity + gas combined) is about $175 per month (I couldn’t find a better source of these data at the moment) or $2100 per year, a savings of $155 (or about 7%) sounds about right. Interestingly, the authors also found that there was no difference in thermostat set points or measured indoor temperatures in the weatherized or un-weatherized homes, suggesting that there was no rebound effect for indoor heating.

The authors also used a different form of the same dataset to explore energy savings. In what they call their ‘quasi-experimental design,’ they looked only at homes that had already applied for WAP funding but either (a) had already received retrofits and were thus considered as a treatment group, or (b) had not yet received retrofits and were thus considered as a control group. This allowed for a closer look at the energy savings achieved by WAP retrofits exclusively in homes that were clearly in need of retrofits. The savings due to participate in WAP were larger for this group: approximately 19% among these homes. There may be other explanations for this large discrepancy, as the authors explore in the appendix, but the long and short of it is that savings from WAP retrofits in homes in Michigan appear to be between about 8% and 19% in total energy savings, depending on how you look at it. These numbers sound reasonable to me and are in line with at least one previous study of which I am aware.

What the authors explore next is where most of the controversy appears to reside. When they compare the average energy savings attributable to participation in WAP (using the $155 per year estimate) to the costs of applying the retrofits (about $5000 in upfront costs, on average), the authors conclude that the costs of energy efficiency investments were approximately twice the benefits. This is an important finding. In the last couple of days, this finding has been turning heads and garnering lots of news coverage (here, here, here, and here). NRDC also has a cautionary blog post here.

It’s important to dig deeper into how they came up with this estimate.

Although they aren’t exactly clear with their methods here (no equations shown), on page 21 of their paper they show how they get to this conclusion: they calculated the net present value (NPV) of energy savings over the useful life of the improvements. This is a common and well-accepted approach to valuing any investment.

They estimate a range of NPVs based on the $5000 in average upfront costs and the first findings of 8-10% annual energy savings, ranging from about $1000 in the short-term assuming a high discount rate to about $2300 in the long-term assuming a low discount rate. Either way, within the analysis period, they estimate that the initial costs of the program are never recouped.

Using their second data set, they perform a regression to estimate about a 10% energy savings (similar to the first data set — they basically had a bunch of issues with the initial 19% estimate of savings and decided to use a more conservative estimate from their regression). They then calculate a central estimate of NPV of about $2400, with a maximum NPV of about $3500, both substantially less than the initial $5000 investment. This is a crucial finding: engineering analyses largely estimate NPVs of the same investments of about $10,000, or a 2:1 benefit-to-cost ratio over the analysis period. Instead, the authors turn that value on its head and report a benefit-to-cost ratio of 1:2!

Looking at these data another way, the authors also calculate a private internal rate of return, IRR (the simplest measure of return they investigate), of somewhere between -10.5% and +0.3%, depending on different assumptions. (They also calculate the same return after factoring in avoided CO2 emissions and other societal benefits, but I won’t focus on that here).

Now, let’s dig a little deeper into the assumptions that the authors made to arrive to these conclusions, because they’re critical for understanding the outcomes.

First, they use three different discount rates of 3%, 6%, and 10% in their NPV analysis, apparently just to test the sensitivity to the parameter. However, it is most common to use a discount rate of 3-3.5% for residential energy efficiency improvements, as described here in an ORNL report, citing OMB, as well as in this NIST guidance document. That’s what I used in a recent paper on residential energy efficiency investments in Energy and Buildings, as well as what the residential building energy optimization program, BEopt, uses as their default discount rate. So it’s probably best to ignore the larger discount rates of 6% and 10% in the working paper because those are really applicable only to scenarios in the private sector (when you have higher cost of capital and where you’re not simply given this money upfront). If I were the authors, I would use 3% as my central estimate of discount rates and use any other values only to test sensitivity. I can’t tell what they used in Table 7 for the IRR analyses (I think 6%, but I couldn’t find an explicit statement). If it was 6%, I would re-do this analysis with 3% and my sense is the conclusion may be quite different, particularly for longer lifespan assumptions.

Second, speaking of lifespan, they assume a lifespan of 16 years in their NPV analysis, taken as the savings-weighted average lifespan of retrofits ranging from attic insulation to furnace tune-ups (although they also explore 10, 16, and 20 year lifespans just to be safe). This is a fairly reasonable assumption in my opinion, although BEopt and other resources typically assume a longer 30-year lifespan given how we seldom retrofit our homes. I don’t mind the choice of a 16-year timeframe and appreciate the sensitivity analysis, but I would also consider extending to 30 years just to explore the sensitivity in more detail. I really don’t think 10 years is even worth considering for investments in home repairs such as attic insulation or air sealing, which could easily have lifespans on the order of 30 to 50 years or more. They considered 10, 16, and 20 years in their Table 7 estimates of IRR and found that it made a huge difference: in fact, moving from 10 years to 20 years turned the IRR from a large negative (-10.5%) to slightly positive (0.3%). This is obviously an important assumption.

Finally, and perhaps most importantly, the authors don’t appear to have factored in any increases in future energy prices, neither for electricity or gas. After reading a Vox story on this paper, I see that at least one energy pundit agrees that this is a big limitation:

On the other hand, I can also understand why the authors might not want to bother projecting future energy costs. I have previously used a 0.3% increase in residential electricity retail prices (in real dollars), as residential electricity prices tend to follow a relatively stable trajectory. However, natural gas prices have bounced all over the place in the last 40 years, including steep drops seen in the last 5-10 years (see Figure 12 in my report here for a look at historical natural gas prices). So I can understand in a sense, but since this is such an important assumption, I think it is certainly worth testing the sensitivity of their results to assumptions for future energy costs, particularly if you’re already going to test the sensitivity to the assumption of lifespan and discount rate! 

Overall, I applaud the authors for their attempt to answer these important questions regarding energy efficiency improvements and the Federal WAP. I am not that surprised by their findings of 10% or so energy savings; the $5000 spent per home can only go so far and I think 10% energy savings is pretty realistic and in line with other studies. I am also not too surprised that this number is less than what engineering estimates find. We tend to perform our analyses considering only the physics and not the people involved (guilty as charged!). However, I do think they seriously need to consider expanding their scope to focus primarily on a 3% discount rate, test the sensitivity to longer lifespans, and consider future escalations in both electricity and natural gas costs. Doing so would provide a more robust assessment and would help place their ‘central estimates’ more in line with more reasonable assumptions.

Ultimately, even after addressing these issues, the findings may still hold. Investments in WAP energy retrofits still might not be a great investment. But I actually don’t find that to be too alarming. For one, I think it is well known that many residential energy retrofits take a long time to pay off. It’s hard to achieve drastic savings in older buildings. Simple payback periods of 25 years or more are not uncommon.

But perhaps even more importantly, the WAP may simply be more of a social policy tool than an energy policy tool. The goal is to help lower-income homeowners save money on their energy bills, simple as that. It actually might not be the best model for testing energy efficiency investments. The investments come from the government, not from private investment where incentives to save may be different. At the end of the day, if the U.S. government chooses to spend $5000 per home (or whatever the magic number is) to help lower low-income homeowners’ energy bills, they have a couple of options. One would be to simply give them $155 per year to cover the savings they could have achieved with WAP, which over a period of about 30 years would add up to $5000 or so in investments per home. But that would essentially be a pass-through of funds from the U.S. government to utility companies. On the other hand with WAP, the U.S. government invests the same $5000 today, which then achieves the goal of lowering energy bills for individual low-income homeowners by $155 per year, on average, while simultaneously lowering greenhouse gas emission (slightly; about 10% along with energy savings) and stimulating a whole host of industries such as energy auditors, contractors, and material suppliers. Maybe that’s not such a bad policy.



Filed under: Energy efficiency | Energy policy | Local to Chicago | News