Mobile phones

We try to get a cross-section of brands and models available in the New Zealand market. Where a model is released in different storage sizes, we usually test the smallest capacity only.

Overall score breakdown

Camera quality (25%)

The cameras are tested with a variety of scenes, including moving subjects and a test pattern. Scenes are tested in both daylight and low-light (with and without flash). We cover HDR, face detection, bokeh and zoom. Each camera mode is tested, including the selfie camera. For video, image stabilisation effects are measured as well as zooming while recording and sound quality.

All tests use a stable tripod, except for the image stabilisation test where the phone’s locked in a stable rig with a known rate of vibration added.

The resulting images are compared using analysis software. We also do a subjective analysis. In addition to image quality, the lab tests shutter delay, autofocus speed and start-up time.

Battery life (15%)

A robot arm recreates a “typical user day” using the phone until the battery is completely discharged. Each day is mostly stand-by (19 hours) with intermittent internet use, calling and photography. Each device is tested both at maximum brightness and at 300 nits.

Display (15%)

We measure the size of the display and its resolution. We test its brightness, contrast, smudge resistance and readability from various angles and in different lighting conditions.

Performance (10%)

We use standardised benchmark tests to measure performance. Each test is run multiple times, back to back. We also penalise the phone if it gets hot during benchmarking.

Durability (10%)

We check how easily phones break and whether they can be operated afterwards. For obvious reasons, this test is done last. We buy two samples of each phone in case one breaks early.

Scratch resistance: The tester attempts to scratch the display (and camera glass) with a test “pencil” set to five different levels of force. We find the maximum load that doesn’t lead to permanent scratches.

Tumble test: Simulates dropping a phone from waist height. Phones are turned on and placed in a tumbling Z-shaped drum with a drop height of 80cm on to a stone base.

We check for damage after rotating the tumbler 25 and 50 times. At each check, the functionality of the phone is tested.

Water resistance tests: The phone is switched on, connected to a network and placed under a rain simulator for 5 minutes. Devices that claim water resistance of IPX7 or higher are submerged to the stated depth for the stated time. After each test, the phone’s functionality is assessed immediately, then each day for three days.

Audio (10%)

The music quality test is conducted using high-quality headphones with the same three pop and classical MP3s. We measure the sound balance of the built-in speakers by calculating the ratio of low-frequency energy compared to mids and highs.

We also find the maximum undistorted loudness of the speakerphone at a distance of 1m in an acoustic chamber.

Ease of use (5%)

The touchscreen is rated for design, shape, usability with small and big fingers, blind accessibility and feedback (sound or haptic touch). The test includes swiping with fingers, using multi-touch gestures (e.g. zooming), swipe writing, stylus use, and copy-pasting. We also rate any physical buttons.

Our experts browse to some popular websites and complete tasks including navigating, zooming and using bookmarks. Web performance is tested against two benchmarks programs.

To determine ease of typing, we time how long it takes to write a passage from a Robert Louis Stevenson poem.

Capabilities (5%)

An assessment of the phone’s features.

To test GPS navigation, our testers record their position while driving a 15km test track. We compare the recorded track to the actual track, also considering signal strength in tunnels. Finally, we check which phones can receive signals from alternative satellite systems, such as GLONASS.

Security and privacy (3%)

Databases that track vulnerabilities are consulted for each device.

We evaluate password creation, including whether:

• both letter cases, digits and symbols are encouraged
• there’s a lower or upper limit on length
• password strength is indicated, explained, and checked against dictionaries
• the manufacturer protects against brute force methods, such as limiting incorrect guesses or using a cooldown period.

During set-up, we check for:

• unencrypted connections, to ensure sensitive data like unique serial numbers aren’t being shared
• how much personal data is requested when creating an account
• whether options are opt-out or opt-in, buttons deceiving or submenus confusing
• the length of Ts and Cs and privacy policies and how easy they are to understand
• whether the risks of using fingerprint or face unlocking are adequately explained.

We check whether the user can remove or disable preinstalled apps (bloatware) and, finally, how easy it is to reset the phone and whether data is deleted properly.

Calling (2%)

The phone is attached to a dummy head with an artificial ear in an anechoic chamber. We record the incoming and outgoing sound on initiated and received calls, both with and without ambient noise (which makes the measurement more realistic).

Reception

Phones are placed in a shielded chamber and connected to a 4G network. We calculate the signal strength at the phone divided by the signal strength at its source. The measurement is repeated using five frequency bands and four orientations to minimise the influence of the antenna setup.