Electronics

How we test digital cameras

Updated 01 Jul 2011

Introduction

Ever wondered what goes on in the lab? We explain what's involved in testing a digital camera.

Camera testing is a complicated process. Find out how we arrive at our ratings and recommendations, the functions and features we appraise and how we decide what a camera will score in each category.

Our report will help to explain why our results sometimes differ from other reviewers, who rely on an individual opinion to appraise these complex products.

To see our test results in action, check out the digital camera report.

The test process

Testing digital cameras is expensive. To help with the high cost, test results are shared between a number of consumer organisations around the world.

All the tests and procedures described in this article are carried out or overseen by professional testers in consumer laboratories in Europe.

To try to keep up with the rapidly evolving digital camera market, we test batches of cameras 5 or 6 times a year. The lab is constantly testing cameras, but each batch can take some months to complete.

We only test products we can purchase from retail outlets, which means we're often a bit slower than places that take samples from manufacturers, but at least we know that what we're testing is representative of what you're likely to see on the shelves.

Before any testing begins, the lab makes an exhaustive list of each camera's features and specifications.

For the most part we don't rely on manufacturers specifications. For example, we take measurements that include all protruding parts when the camera is off and weigh them with their memory, batteries, neck straps, and so on, that come as standard. Manufacturer's specifications often leave these sorts of things out, but our intention is to try to give you information that relates to actual use.

All computer-related tests are performed on an up-to-date PC. The 5 monitors used are profiled and calibrated EIZO ColorEdge CG21 or EIZO S2433W and both monitors and video cards are adjusted to sRGB colour space.

In order to compare the test results to those of the previous tests using this method, in all relevant tests (especially those where a subjective score is given, such as the outdoor shots) samples from the last test are re-tested in certain aspects and used as anchor for scoring.

Unless otherwise noted subjective tests are evaluated by 5 people, among them at least 3 interested users of cameras who are trained in assessing images. Each panel member has their own monitor to provide best possible viewing conditions.

Image quality

All evaluations of picture quality are performed after the photos are transferred to the PC. Unless otherwise stated the cameras are set to full automatic function – including exposure, white balance and focus – with maximum resolution and image quality setting in JPEG . Test charts are adjusted to fill the full monitor display of the camera. If not stated otherwise a tripod is used.

Resolution

Resolution is determined by the evaluation of a black and white test pattern. This is a visual appraisal of the pattern in the middle and at the corners of the test chart. Additionally, there is a score for aberration and colour problems at the edges of the image, with comments where appropriate.

The pictures are taken in maximum and minimum optical zoom setting at ISO200 (high-end cameras take pictures at ISO1600 and ISO3200 and in both JPEG and RAW formats where possible). The test chart is evenly illuminated by 2 light sources with D65 light characteristic (daylight), illumination level is at 2000 cd/m².

Colour reproduction

Colour reproduction is tested using an IT8 colour test chart. The lighting is similar to resolution (above) and the camera is set to ISO200 and Auto or its equivalent. For high-end cameras the test is repeated at ISO1600.

There's a subjective score of the colour balance for the IT8 test chart for luminance, contrast, colour saturation and definition.

The pure white field of the test chart is evaluated for colour balance of the primary colours. This is done using a software calculation (histogram) of the test chart differences in brightness for the 3 primary colours in the white field (ideally they should be equal).

Low light performance

Picture brightness in very low light conditions is measured by the electronic evaluation of pictures taken of a plain screen with a dimmable white light source. The pictures are taken at a very low luminance setting of 1 cd/m². The sensitivity of the camera is evaluated by measuring the average brightness and the standard deviation of the evaluated pixels in the central part of the digital image.

Resolution in low light conditions is tested at the same light level (1 cd/m²) using the resolution test chart. This test chart is photographed at the maximum wide position of the camera's lens and scores are given for sharpness and image noise.

Low light performance is evaluated with a visual inspection of an image of a test chart taken in a low light environment. The test chart is placed on a turntable which rotates at 45 seconds per rotation. Photos are taken at an illumination level of 11 lux and evaluated for loss of sharpness and image noise.

Image noise (high-end models)
This is a measurement of the luminance and colour noise (red, green, blue) in the central grey field of the test chart at ISO 1600 and ISO3200 (JPEG and raw format). The camera is set to its minimum focal length and white balance is done manually.

Distortion

Measures vertical and horizontal grid lines on a target at 1/3 and at 2/3 image height and width in both JPEG and RAW format at ISO 200. The lens is set to minimum and maximum focal length.

Saturated colour reproduction (high-end cameras only)

We generate a continuous, optical spectrum from a white light source on a screen and add a bright invisible infrared light source. We take pictures at the highest JPEG quality and in sRGB colour space. These pictures are displayed on a profiled monitor and evaluated for their respective differences to the original spectrum.

This is done at ISO 200 and ISO 1600 and always with manual white balance (measured at the projection screen without dispersive element and infrared source) for all high-end cameras. Scores are given for the colour characteristics at low and high frequencies (infrared to ultraviolet) and the colour continuity (steps or holes in the spectrum).

Veiling glare (high-end cameras only)

This measures the amount of stray light in a lens caused by reflections of light between the surface of the lens element and the inside of the barrel of the lens. A lens that performs poorly will often show up as producing images with a washed-out look over most of the photo. This is often less noticeable than other types of glare which may leave a distinctive light spot on an image. Veiling glare can result in a reduction in the contrast of the image as well as the colour saturation.

Measured according to ISO 9358:1994 at maximum and minimum aperture and at normal focal length with ISO 200 setting and using manual white balance.

Reflections (high-end cameras only)

An image of a dark screen with a bright light source at 10° right or left of the maximum horizontal image angle is taken at maximum aperture (minimum F-number) and minimum and maximum focal length . We do not use a lens hood, even if one is provided.

Practical pictures / viewing tests

Practical pictures are performed using 5 images:

Images 1 and 2 are portraits under medium light conditions (D65 daylight, about 140 lux) in a room with a dark background, both with and without the use of the integrated flash. The tests are performed with the camera set to auto or its equivalent. The subject is 2 metres away and the focal length is adjusted to fill the screen with the subject. Scores are given for skin tone quality, colour quality in general, exposure level, sharpness and image noise.
The 3rd test picture is an outdoor landscape picture with sky, a building and a natural background of lawn and trees, taken in bright sunshine. The camera is focused at infinity. Scores are given for detail in highlights and shadows, colour quality, exposure level, sharpness and image noise.
The 4th test picture is an indoor image of different items such as skin, clothes, toys with artificial colours, high contrasts and a heat source at 450°C (with almost invisible infra red emission). Scores are given for detail in highlights and shadows, colour quality, exposure level, sharpness and image noise. The subject is 2 metres away and the focal length is adjusted to fill the screen with the subject.
The 5th test picture is for the evaluation of the face detection function. An image is taken of 2 people left and right of the image centre, in front of a bright white screen with some high contrast objects. The image is taken as close as possible to normal viewing angle (47°). The people are at a distance of 1 and 3 meters and the rear screen is at a distance of 5 metres. Scores are given for sharpness and the exposure of the faces, with preference given to the person closest to the camera.

Functions tests

Monitor and viewfinder

The quality of the monitor and viewfinder are evaluated by a panel for readability under different light conditions (sunshine, low light, indoor). Adjustments for users wearing glasses (diopter correction) is evaluated as well. In both cases we look for picture quality in terms of colour performance, brightness, contrast, resolution and the speed of the refresh rate (important for precise pictures when zooming or taking photos in panning situations).

Viewfinder

The evaluation of the viewfinder is performed by scoring the usability in bright sunlight conditions outside, the usability in poor light conditions inside, the quality of resolution and refresh rate at panning scenes, and the size of the image in the viewfinder. Basic cameras are not penalised for not having a viewfinder, but high-end cameras are.

For high-end cameras
We measure the difference of the size of the image in the viewfinder related to the width stored in the image. The camera is positioned as close as possible to a rectangular test chart. The test chart is adjusted to full viewfinder size by adjusting the distance to the test chart at the maximum wide setting and maximum telescopic setting of the zoom lens.

Monitor

The monitor's loss of readability (contrast, brightness, colour) is measured when viewed from an angle other than 90° from above (evaluation range is +/-30° to left and right and +/-20° to above and below related to view from right angle). This test is performed indoors in good light with no light from behind the monitor, and in poor light conditions. This is repeated outside in bright conditions.

The quality of the monitor in Playback mode is evaluated using a test image in JPEG format with fine details and high contrast. Each camera tested has the test file displayed in full screen mode and without additional information (if possible), which is photographed with a high resolution digital camera.

Each camera's monitor is shot using the same focal length of the recording camera, so that different sizes of the file image show the different size of the display. These shots of the tested camera's monitors are evaluated on a high quality PC screen for resolution, sharpness, contrast and brightness of the display, as well as errors such as a visible cell structure of the LCD panel.

In this test only the quality of the monitor itself is evaluated. In the tests previously described the cameras are in record mode, which means that the effects of the image sensor, the lens and the exposure electronics also have an influence on the result.

The following pictures show some examples of different monitor displays.

Image stabilisation

The test signal is recorded from acceleration sensors applied to the arms of 2 men holding a camera in their hands at a normal shooting position. This signal is averaged for the 2 people and measured over a lengthy period.

We test by mounting the camera on a shaker and applying the recorded movement in different directions and at different strengths while taking images. Half the images are taken at normal viewing angle (47°) of an 80cm target at around 1 metre distance. This is done with a light level of 11 lux and the camera's flash is turned off. Where possible the test is repeated with the lens set to a viewing angle of 12°, which puts the target at around 4 metres from the camera and the light level is adjusted to 1000 lux.

Focusing

A score is given for the autofocus behavior in a room with an illumination level of 11 lux, with flash suppressed. Scores are given for speed of focus, ringing, a stable final setting and sharpness of the images.

Autofocus sensitivity (high-end cameras only)

We take an image in automatic mode and at normal focal length (image angle 47°) of a test chart at a distance of 1 metre, illuminated by photo lamps. There's a pattern in the centre of the screen to allow the camera to focus automatically. The power of the photo lamps is reduced, which in turn reduces the contrast of the pattern on the test chart. We measure the luminance of the test chart in cd/m² when the auto focus doesn't reliably find the focus point any more.

Macro image

The macro function of the camera is performed by measuring the smallest sharp area which the camera can take in macro mode. We also measure the distance between test chart and lens front. The test is performed using automatic focus.

Flash

We check to see if the flash is "intelligent" (the power of the flash automatically adapts to the distance from the subject).

Flash quality

A picture of the test chart is taken in a dark room. The camera is adjusted to fit to the full test chart in the viewfinder/monitor of the camera by using the zoom lens. Pictures are taken at 1, 3 and 5 metres.

The resulting images are appraised for the amount of light put out by the flash, evenness of illumination and ability of the flash to cope with varying distances. If necessary we also note reflections/shadows from the camera's body.

Flash quality – long distance

A complete darkroom with white walls has distance marks (small grey cards) at different positions within the room. Their brightness is evaluated for the flatness of the distribution of the flash light across the scene.

Video tests

Video viewing test

Monitors, projectors and video cards are adjusted to sRGB colour space. All measurements and recordings are made using automatic exposure and automatic white balance settings. Two one minute video recordings are made of a scene featuring people with colourful clothes, other objects with fine details, and realistic artificial plants at different distances. The automatic focus system and white balance is switched on if possible.

The camera is mounted on a tripod. About a third of the take is recorded while panning and another third while zooming. Lighting for the tests is under the following lighting conditions at the targets:

Daylight conditions, 3500 lux (D65)
Low light, 12 lux.

These recordings are replayed on profiled and calibrated high-quality liquid crystal monitors or projectors with high resolution (full HD with 1920 x 1080 pixels and at least 24 frames per second). The test panelists evaluate colour fidelity and view the actual objects with appropriate illumination. The recorded images are rated according to the following criteria:

Sharpness (how faithfully structures are reproduced)
Colour fidelity (colour neutrality and colour noise)
Exposure and contrast gradation (also in highlights and in dark areas)
Image and focusing irregularities
Overall image quality.

Resolution

We take video recordings at maximum and minimum focal length (using the same test chart used for still images) and rate them for the resolution of the images at the centre and corners.

Macro recording

We take video recordings at maximum magnification and rate the images for resolution and overall quality. Particular attention is paid to any issues with focusing, exposure or contrast.

Audio quality

All measurements and recordings are made using the camera's automatic gain control settings. An audio stereo sample (classical music with high dynamic range) with a duration of about 30 seconds is played over two high-quality loudspeakers in a room.

First, a reference recording from these samples is produced on a digital recorder with a high-quality stereo microphone at a distance of 2 metres. The sound pressure level at the location of the microphone is at an average of 65 dB(A). Two audio recordings are made with each camera being tested, under the same conditions. Recording is with the built-in microphone in automatic mode and we create possible noise by using the zoom button or zoom drive.

We then listen to the recordings using high-quality headphones and rate the overall audio quality at the original volume. Problems with motor-, zoom- and button-noises are commented on separately.

Device ready for operation

The device is switched off for at least a minute then switched on again, and we measure the time it takes before the first video frame is taken.

Ease of use

Ease of use is evaluated by 5 people, at least 3 of whom have some formal photographic training. Scores range through a 5 point scale from bad to very good.

Manuals

The printed documentation and the manuals on supplied CD-ROMs are evaluated for:

Completeness of documentation with detailed explanations and examples for the use of the camera.
Appearance, paper quality, readability of text and sketches.
Structure, sufficient and easily understood content, non-technical explanations.
Are there safety hints for electrical danger or damages by static discharge?
Are there helpful hints in case of malfunction?
Quality of explanations for the use of digital cameras in general and the use of the camera itself.
Quality of the description for connecting to a PC and file transfer.
Is there a table of contents or equivalent available?
Are there hints for the care of rechargeable batteries?

Shutter delay

All the camera's functions are set to automatic and the lens focused at a target at 10 metres. Then, with the camera positioned 1 metre from a stop watch, the shutter release button is pressed in one motion and the time it takes for the shutter to open is measured.

A device is used to start the stopwatch at the instant the shutter is pressed and the resulting image shows how far the watch had progressed before the image was captured. We then repeat this process with the target set to 1 metre to help assess the camera's ability to cope with different circumstances. This process is repeated 10 times per camera and the results are averaged.

High-end cameras such as SLRs may have a choice of phase detection or contrast detection autofocus depending on their mode (usually LiveView for the latter). Where this option is available the test is carried out in both modes.

Camera ready for operation

The camera is switched off for at least 1 minute. Then we measure the time it takes before the first picture can be taken. The monitor is on and has to preview the live scene before the release button is operated.

Picture-taking speed (single shooting mode)

We measure the time taken from pressing the shutter release until the camera indicates it's ready to take next picture. It's set to its highest quality setting with the flash off.

Picture-sequence speed (continuous shooting mode)

Using a consistent normal illumination, with the camera set at medium focal length, a subject at 3 metres and the flash off we measure:

the number of pictures/second or the shortest possible time interval between 2 pictures.
the number of pictures until the buffer is full.

Inserting and removing memory card

The insertion and removal of the memory cards is evaluated:

How easy is it to grip when inserted?
Is the eject button accessible and easy to use?
Is there a clear indication of which way/side the memory card should be inserted?
Does the card compartment have a cover and is it easy to open?

Release button

Is the release button easy to handle? Does it need much force to depress? (If it's too stiff there's a danger of moving the camera while releasing.) Is there a clear indication when the shutter releases?

Autofocus

We give a rating for the quality of the feedback of the toggle point when using the focus lock by half pressing the shutter release.

Manual focusing

How easy is it to use the focus adjustment in manual mode? Are there easily readable scales or other information aids for determining the object distance?

Zooming

We give a rating for the adjustment of the optical zoom control, speed, responsiveness, number of steps and precision.

Battery change and control of the remaining power

We check how easy it is to change the battery and what indications there are when the battery is getting low.

Lens change

We check for precision, mechanical play and friction when changing larger and smaller lenses.

Video recording

We rate the following:

How the device fits into the hand and its comfort for taking video.
Delay of video recording starting after pressing the record button.
Feedback during video take.
Control of the zoom.
Control of the record button.
Control of other functions such as focusing.

Video playback

We rate the following:

Access of playback, pause and stop function.
Possibility of fast forward and rewind.
Video clip selection (including sorting, searching and filtering).

Versatility

The versatility rating is a combination of a number of feature and function scores. Scores are given for having the feature or function and how well it performs. Cameras that can be used in very simple auto modes as well as offering a lot of manual options will score better than those with only auto options.

Features covered include the following for both stills and video where appropriate:

Calculated resolution. This is taken from the image produced not the manufacturers specifications.
Lens – including zoom, aperture range and depth of field. Zoom range calculations are done by photographing two test charts in maximum wide and maximum tele settings. The visible width and height of the text chart is measured together with the distance between front of the lens and the test chart. The zoom ratio of the optical lens system is calculated from the measured image field angles not the focal lengths of the lens setting.
Image ratios and formats supported.
Memory – delivered, number of cards accepted and internal memory where applicable.
ISO variability (ISO is the International Standards Organisation rating for film speed).
Shutter speed – minimum and maximum.
Quality settings – number available.
White balance – manual and automatic and how many settings available.
Exposure settings – manual and automatic and how many settings available.
Monitor /viewfinder – type, adjustment and options such as grid guides, histograms etc.
Deleting options.
Flash types and connections.
Focusing and macro options (including closest focus points, face detection, focus assistance lamp, etc).
Battery type and charging options.
Audio and video – connections and options.
Other connection – PC, printer, etc.
Self timer options.
Tripod socket position.
Panorama options.
GPS options.
Shutter delay score.
Image stabilisation type.
Camera ready for operation score.