Marco.org

Apple’s official response to Consumer Reports’ poor 2016 MacBook Pro battery test:

We appreciate the opportunity to work with Consumer Reports over the holidays to understand their battery test results. We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab. After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life. We have also fixed the bug uncovered in this test. This is the best pro notebook we’ve ever made, we respect Consumer Reports and we’re glad they decided to revisit their findings on the MacBook Pro.

Apple’s tone and framing here, and in most recent PR statements where they’re on the defensive, rubs me the wrong way.

Consumer Reports has a spotty history with calling Apple out on product flaws. They’re usually written overly sensationally, and they often overstate the importance of minor issues.

But almost every time, the problem they’re reporting is real — especially in retrospect, after everyone’s defensiveness has passed and we’ve lived with the products for a while. It’s just debatable how big of a deal it is in practice.

The iPhone 4 antenna design really was flawed. The iPad 3 really did get uncomfortably warm. And the 2016 MacBook Pro really did have poor, inconsistent battery life in their test.

Apple’s framing here is almost Trumpian, evading responsibility for the real problem — Apple’s bug — by attempting to insult the test (“does not reflect real-world usage”), discredit and imply malice by Consumer Reports (“a hidden setting”), and disregard the bug as irrelevant (“obscure and intermittent bug”).

It reframes the story to be about Consumer Reports’ own failings and Apple helping them see the right way forward.

But disabling the browser cache during a battery test to make results more consistent is reasonable, Apple’s browser offers that feature, and it’s neither very well hidden nor unused by any customers.¹

Nothing about a battery-life test truly reflects “real-world usage.” Battery tests are approximations, designed to mimic the most common tasks but in an artificial, automated, repetitive way for hours on end to get reproducible results.

Real-world usage is so varied, with such wildly different software and usage patterns, that nobody’s battery test is relevant to much of anything except relative comparisons of its own results. A test that lasts 5 hours on a 13-inch and 6 hours on a 15-inch tells you that the 15-inch probably has better battery life than the 13-inch, but that’s about it — it certainly doesn’t mean that your 15-inch will get 6 hours the way you use it.

Reloading web pages without a browser cache is no more or less valid than whatever Apple uses to tell me that my laptop will get 10 hours of battery life when my actual workload usually gets about half of that.

The real story here is that Consumer Reports did get very poor and inconsistent results from their battery test, which was a reasonable and valid test, due to a real bug in Apple’s web browser.

With the bug now fixed (in beta, at least), the MacBook Pros deserve a retest, and Consumer Reports is conducting one.

But their previous results were real, and Apple’s bug was to blame.