Because modern smartphones are a computer, camera and telephone rolled into one, they require a series of rigorous tests. But they also need to last.
We’ll only recommend appliances you’ll love to own, that work well and keep working well for a long time.
Our independent lab testing measures a phone’s quality when it’s new. It still forms a core part of our lifetime scoring because a slow or poorly-designed mobile isn’t going to live a fruitful life.
A robot arm recreates a “typical user day” using the phone until the battery is completely discharged. Each day is mostly stand-by (19 hours) with intermittent internet use, calling and photography. Each device is tested both at maximum brightness and at 300 nits.
Phone battery life is even more important than it seems. Batteries have a finite number of recharges before they degrade, so a shorter cycle means less total usage before the battery needs replacing. Because many manufacturers make that hard to do, some owners choose to throw out the whole phone. Either way, a long battery cycle means you don’t have to make that decision as quickly.
A repairable phone can both save you money and reduce your waste footprint. This score is based on the availability and price of spare parts, how easy it is to disassemble the phone, and what repair documentation and support is available. It’s not localised to New Zealand, but you can help us demand a local repairability label.
It’s reasonable to expect a new phone to remain fault-free for five years. Our predicted reliability won’t tell you whether a device will stop working tomorrow, but it does show which brands make models that are less likely to fail.
Test performance breakdown
The cameras are tested with a variety of scenes, including moving subjects and a test pattern. Scenes are tested in both daylight and low-light (with and without flash). We cover HDR, face detection, bokeh and zoom. Each camera mode is tested, including the selfie camera. For video, image stabilisation effects are measured as well as zooming while recording and sound quality.
All tests use a stable tripod, except for the image stabilisation test where the phone’s locked in a stable rig with a known rate of vibration added.
The resulting images are compared using analysis software. We also do a subjective analysis. In addition to image quality, the lab tests shutter delay, autofocus speed and start-up time.
We measure the size of the display and its resolution. We test its brightness, contrast, smudge resistance and readability from various angles and in different lighting conditions.
We use standardised benchmark tests to measure performance. Each test is run multiple times, back to back. We also penalise the phone if it gets hot during benchmarking.
We check how easily phones break and whether they can be operated afterwards. For obvious reasons, this test is done last. We buy two samples of each phone in case one breaks early.
Scratch resistance: The tester attempts to scratch the display (and camera glass) with a test “pencil” set to five different levels of force. We find the maximum load that doesn’t lead to permanent scratches.
Tumble test: Simulates dropping a phone from waist height. Phones are turned on and placed in a tumbling Z-shaped drum with a drop height of 80cm on to a stone base.
We check for damage after rotating the tumbler 25 and 50 times. At each check, the functionality of the phone is tested.
Water resistance tests: The phone is switched on, connected to a network and placed under a rain simulator for 5 minutes. Devices that claim water resistance of IPX7 or higher are submerged to the stated depth for the stated time. After each test, the phone’s functionality is assessed immediately, then each day for three days.
The music quality test is conducted using high-quality headphones with the same three pop and classical MP3s. We measure the sound balance of the built-in speakers by calculating the ratio of low-frequency energy compared to mids and highs.
We also find the maximum undistorted loudness of the speakerphone at a distance of 1m in an acoustic chamber.
Ease of use
The touchscreen is rated for design, shape, usability with small and big fingers, blind accessibility and feedback (sound or haptic touch). The test includes swiping with fingers, using multi-touch gestures (e.g. zooming), swipe writing, stylus use, and copy-pasting. We also rate any physical buttons.
Our experts browse to some popular websites and complete tasks including navigating, zooming and using bookmarks. Web performance is tested against two benchmarks programs.
To determine ease of typing, we time how long it takes to write a passage from a Robert Louis Stevenson poem.
An assessment of the phone’s features.
To test GPS navigation, our testers record their position while driving a 15km test track. We compare the recorded track to the actual track, also considering signal strength in tunnels. Finally, we check which phones can receive signals from alternative satellite systems, such as GLONASS.
Security and privacy
Databases that track vulnerabilities are consulted for each device.
We evaluate password creation, including whether:
- both letter cases, digits and symbols are encouraged
- there’s a lower or upper limit on length
- password strength is indicated, explained, and checked against dictionaries
- the manufacturer protects against brute force methods, such as limiting incorrect guesses or using a cooldown period.
During set-up, we check for:
- unencrypted connections, to ensure sensitive data like unique serial numbers aren’t being shared
- how much personal data is requested when creating an account
- whether options are opt-out or opt-in, buttons deceiving or submenus confusing
- the length of Ts and Cs and privacy policies and how easy they are to understand
- whether the risks of using fingerprint or face unlocking are adequately explained.
- We check whether the user can remove or disable preinstalled apps (bloatware) and, finally, how easy it is to reset the phone and whether data is deleted properly.
The phone is attached to a dummy head with an artificial ear in an anechoic chamber. We record the incoming and outgoing sound on initiated and received calls, both with and without ambient noise (which makes the measurement more realistic).
Phones are placed in a shielded chamber and connected to a 4G network. We calculate the signal strength at the phone divided by the signal strength at its source. The measurement is repeated using five frequency bands and four orientations to minimise the influence of the antenna setup.