How to find the perfect mobile phone for your needs.
Because modern phones are a computer, camera and telephone rolled into one, they require a series of rigorous and standardised tests.
We try to get a cross-section of brands and models available in the New Zealand market. Where a model is released in different storage sizes, we usually test the smallest capacity only.
The cameras are tested with a variety of scenes, including moving subjects and a test pattern. Scenes are tested in both daylight and low-light (with and without flash). We cover HDR, face detection, bokeh and zoom. Each camera mode is tested, including the selfie camera. For video, image stabilisation effects are measured as well as zooming while recording and sound quality.
All tests use a stable tripod, except for the image stabilisation test where the phone’s locked in a stable rig with a known rate of vibration added.
The resulting images are compared using analysis software. We also do a subjective analysis. In addition to image quality, the lab tests shutter delay, autofocus speed and start-up time.
A robot arm recreates a “typical user day” using the phone until the battery is completely discharged. Each day is mostly stand-by (19 hours) with intermittent internet use, calling and photography. Each device is tested both at maximum brightness and at 300 nits.
We measure the size of the display and its resolution. We test its brightness, contrast, smudge resistance and readability from various angles and in different lighting conditions.
We use standardised benchmark tests to measure performance. Each test is run multiple times, back to back. We also penalise the phone if it gets hot during benchmarking.
We check how easily phones break and whether they can be operated afterwards. For obvious reasons, this test is done last. We buy two samples of each phone in case one breaks early.
Scratch resistance: The tester attempts to scratch the display (and camera glass) with a test “pencil” set to five different levels of force. We find the maximum load that doesn’t lead to permanent scratches.
Tumble test: Simulates dropping a phone from waist height. Phones are turned on and placed in a tumbling Z-shaped drum with a drop height of 80cm on to a stone base.
We check for damage after rotating the tumbler 25 and 50 times. At each check, the functionality of the phone is tested.
Water resistance tests: The phone is switched on, connected to a network and placed under a rain simulator for 5 minutes. Devices that claim water resistance of IPX7 or higher are submerged to the stated depth for the stated time. After each test, the phone’s functionality is assessed immediately, then each day for three days.
The music quality test is conducted using high-quality headphones with the same three pop and classical MP3s. We measure the sound balance of the built-in speakers by calculating the ratio of low-frequency energy compared to mids and highs.
We also find the maximum undistorted loudness of the speakerphone at a distance of 1m in an acoustic chamber.
The touchscreen is rated for design, shape, usability with small and big fingers, blind accessibility and feedback (sound or haptic touch). The test includes swiping with fingers, using multi-touch gestures (e.g. zooming), swipe writing, stylus use, and copy-pasting. We also rate any physical buttons.
Our experts browse to some popular websites and complete tasks including navigating, zooming and using bookmarks. Web performance is tested against two benchmarks programs.
To determine ease of typing, we time how long it takes to write a passage from a Robert Louis Stevenson poem.
An assessment of the phone’s features.
To test GPS navigation, our testers record their position while driving a 15km test track. We compare the recorded track to the actual track, also considering signal strength in tunnels. Finally, we check which phones can receive signals from alternative satellite systems, such as GLONASS.
Databases that track vulnerabilities are consulted for each device.
We evaluate password creation, including whether:
During set-up, we check for:
We check whether the user can remove or disable preinstalled apps (bloatware) and, finally, how easy it is to reset the phone and whether data is deleted properly.
The phone is attached to a dummy head with an artificial ear in an anechoic chamber. We record the incoming and outgoing sound on initiated and received calls, both with and without ambient noise (which makes the measurement more realistic).
Phones are placed in a shielded chamber and connected to a 4G network. We calculate the signal strength at the phone divided by the signal strength at its source. The measurement is repeated using five frequency bands and four orientations to minimise the influence of the antenna setup.