Consumer Reports announced today that they approve of the iPhone 4S, after famously refusing to recommend the iPhone 4 because of that antenna issue that turned out to affect almost none of the millions of people who bought it.
But the iPhone 4S was defeated by three Android phones in CR’s scoring criteria:
These pluses were not enough, however, to allow the iPhone 4S to outscore the best new Android-based phones in our Ratings. Those top scorers included the Samsung Galaxy S II phones, the Motorola Droid Bionic, and several other phones that boast larger displays than the iPhone 4S and run on faster 4G networks. […]
Other phones that topped the iPhone 4S include the LG Thrill ($100 on AT&T), which has the ability to capture stills and videos in 3D, as well as display them on its 4.3-inch 3D display, and the Motorola Droid Bionic ($300 on Verizon), which also has a superb 4.3-inch, high-resolution (540 x 960) display, with excellent keypad readability under most lighting conditions, even in bright light.
I’m looking at their full test results (I’ve been a CR website subscriber for six years), and I’m really not confident in the metrics and priorities that they seem to be using. Even some of the measurements seem suspicious to me:
- CR rated the iPhone’s battery life lower than all three Android phones that “beat” it in the scoring (Galaxy S II, Infuse 4G, Thrill 4G). But most other reviews have claimed that using 4G, one of the biggest selling points that CR uses against the iPhone, results in terrible battery life. And I don’t think any reasonably objective reviewers would claim that Android phones generally get better battery life than iPhones, so I’m suspicious of their testing method that showed the 4S being worse than all three Android models.
- The iPhone’s “Highs” include “Dynamic, intuitive iOS operating system provides superb multimedia functionality and easy access to most features”, “The best MP3 player we’ve seen in a phone”, and “Built-in voice-controlled personal digital assistant”. But it’s scored identically to the Android phones in “Ease of use”. Maybe this is my iOS bias, but I’ve never heard regular non-geeks talking about how easy-to-use their Android phones are or how they’re equally easy to use as an iPhone.
- CR rated the iPhone’s camera higher than the LG Thrill’s “3D” cameras and noted that viewing the 3D images on the Thrill caused eyestrain, yet they cited this 3D capability as a reason why the Thrill was better than the iPhone in the summaries. They also say the iPhone has “one of the best cameras we’ve seen on a phone”, but that wasn’t enough to earn the “excellent” dot-circle rating.
Screen size seems to be as important as quality, with the same “excellent” rating given to all of these screens. They even, again, use screen size as a reason to recommend these Android phones more highly than the iPhone. But then, in the “Lows” of each Android phone, they include “Larger than many of the smart phones we’ve tested”.
Does CR consider a huge screen, which results in a huge phone, a positive or a negative? Seems inconsistent.
- “Easy access to Apple’s online stores, with an unmatched selection of apps, games, music, and more” is significant enough to include as a “High” for the iPhone, but it’s not a ratings category and does not appear to contribute to the score. Not considering the software and media ecosystem as a relevant scoring metric on a modern smartphone seems negligent at best, considering how many people are using their smartphones as media players and pocket computers.
Consumer Reports is a good place to get an overview of objective specs and measurements for a product category. I’ve often consulted their ratings to guide many purchases, mostly appliances, without any major regrets.
For their ratings to be useful to my purchase, their priorities and criteria need to approximately match mine. This is easy for most of the products they review: most people want a dishwasher to be able to quietly, effectively, and reliably wash dishes. A dishwasher that’s quieter is objectively better than a louder one. An air conditioner that uses less energy for the same cooling is better than a less efficient model. You can assign numbers and scores to factors like these.
Smartphones have too many subjective criteria, and even the measurable stats don’t always yield a definite answer on what’s better. If you want a huge screen, you’ll get a huge phone, so is a larger screen size a good thing or not? Fast 4G network access kills battery life, so is 4G a good feature for you? Do you want the best normal camera, or a lower-quality 3D camera? Do you want any particular apps or games that are only available on one platform? Do you need a kickstand? (Do they still make those?) These all depend on your priorities.
A product as complex and multifaceted as a modern smartphone is beyond Consumer Reports’ ability to rate in a way that’s useful to most buyers.