How We Test Digital Cameras and Lenses

We take a dual approach to reviewing digital cameras and lenses at PCMag, splitting work between standardized lab tests in a controlled environment and photo sessions out in the real world. Doing so allows us to evaluate image quality in a predictable way, as well as challenge modern autofocus systems with various types of subjects. Here’s a look at what we do in each set of tests.

How PCMag Tests Digital Cameras

How We Test Cameras

In the lab, we evaluate image quality across the entire ISO range to determine how photos will look in varying levels of light, as well as to determine whether lenses can deliver crisp images on modern, high-resolution sensors. With an interchangeable lens camera, we always use the highest-quality lens we have available at its best f-stop to evaluate noise and detail. We rely on an X-Rite Color Checker Classic swatch chart to get a quantitative noise value from out-of-camera JPGs, framed next to a foreign bank note that allows for a qualitative analysis of noise and detail for both JPG and Raw images.

We use an XRite Color Checker Classic to check noise in color swatches and a foreign banknote to see how much noise influences detail at a pixel level (Credit: Jim Fisher)

We can also confirm the accuracy of manufacturer-stated shooting rates and buffer capacities, as well as evaluate performance with multiple memory formats. We rely mainly on field tests to check subject recognition modes, but also see how well cameras handle on-screen images of people, animals, and other common subjects as necessary.

Image quality isn’t as huge a differentiator among digital cameras as it was in years past. The majority of the cameras we review match peers with the same size sensor in terms of picture quality. There are outliers, of course, which is why we still test everything. Autofocus intelligence, sensor readout speed, rolling shutter distortion, and video features are the areas in which today’s models differ significantly.

On that note, we now put more emphasis on testing cameras outside the lab. Autofocus performance has become particularly important. To that end, we look at how well a camera can track moving subjects and see if features like face and eye detection work as advertised. We sometimes use an Atomos Ninja V to record the viewfinder in the field, which helps us better compare autofocus performance between different camera systems.

Taking the camera out into the world also informs our impression of its ergonomics. The battery life, build quality, handling, viewfinder, and level of weather protection are all important factors to consider. If a manufacturer bills a camera as rugged or waterproof, we do our best to put it through the ringer. We drop tough cameras on the ground and take water-protected systems out in the rain. We’re not able to perform teardowns, but certainly take them into account when they’re available.

We make sure to get waterproof cameras wet during testing (Credit: Jim Fisher)

Field testing also allows us to determine how a camera handles different types of scenes, including those that push dynamic range. Capturing Raw images in these scenarios lets us see what photographers can expect when they lift shadows, curb highlights, and perform other edits in post. We use Adobe Lightroom Classic as our standard processor.

Video tests are qualitative. We record at the highest quality and utilize the various color profiles available on a camera. When a flat profile is an option, we see how well it grades and utilize a standard LUT in Apple Final Cut Pro for processing. We pan the camera to look for rolling shutter distortion, check to see how well autofocus works in video, and try any special features like slow-motion capture or Raw recording. We also record long-form clips to see if a camera is prone to overheating.

Finally, we look at the support system for a camera. If it’s an interchangeable lens model, we consider the cost and quality of both first- and third-party lenses. We also test how well a camera’s companion app works for image transfers or tethered shooting in the studio.

How We Test Lenses

Your choice of lens has more influence over the pictures you take than your camera. As such, we perform thorough reviews and picture analyses on optics, both in our test studio and in the real world.

Nikkor Z 135mm Plena lens (Credit: Jim Fisher)

We use one of two SFRplus test charts to check a lens’ resolution, as well as for any distortion and vignetting. Our large chart is ideal for high-pixel cameras, but our testing space requires us to use a smaller, less precise chart for extreme telephoto angles. We evaluate a lens at various points across its focal range (for zooms) and at every full-stop aperture available. We include the figures in our reviews, using a scaled metric to qualify them. After all, comparisons between various sensors of differing resolutions are not informative. Our reference scale follows below.

We run JPGs through Imatest software for analysis using an MTF50 baseline with line width per picture height (lw/ph) figures and refer to these numbers colloquially as “lines” in our reviews.

SFRPlus test chart (Credit: Jim Fisher)

We perform all tests on a tripod and enable a self-timer to reduce the chance of vibrations affecting results. We aim to run every lens lab test at the camera’s lowest ISO and with an electronic shutter, taking multiple samples to lessen the chance of a misfocus or vibration. Occasionally, we use a higher ISO with longer lenses in addition to the base ISO score since those optics are prone to interference even on a tripod. Our lead analyst has performed thousands of lab tests, and can recognize an oddball result from missed focus, shutter shock, or an unintended vibration.

The bokeh in this image shows the onion skin effect, it’s more noticeable here because of light refracting through a defocused droplet of water, just to the right and above the berry (Credit: Jim Fisher)

Resolution is an important factor in lens performance, though not a hugely differentiating one: Most modern lenses are very, very sharp. As such, we also look at other aspects of optical performance. We check lenses to see if they show any false color, for instance, both in high-contrast areas on the plane of focus (lateral chromatic aberration) and in transitions between the plane of focus and background blur (longitudinal chromatic aberration, or LoCA).

Soap bubble bokeh draws highlights with edges brighter than at center, an effect that can lead to busy, characterful backgrounds (Credit: Jim Fisher)

We also consider the quality of the background blur itself, or bokeh. We scrutinize the shape of defocused highlights to see if they are rounded or show a cat’s eye effect, as well as check for any onion skin texture, extra-bright soap bubble edges, or purple/green color shift (another form of LoCA).

False purple and green color (LoCA) is difficult to remove using software (Credit: Jim Fisher)

We also try and get sunstars from lenses. Sunstars are a result of optical diffraction, so they tend to show up at very small apertures, usually f/16-22. At these settings, the lens draws small points of light with multi-point starbursts, an effect that some landscape specialists find desirable. We look to see if the tines of the star are crisp or if they streak out into several points, as well as check for any false color effects. At small apertures, most color issues come from internal reflections in the lens optics. These appear as green or rainbow-color orbs (commonly called ghosts) in images.

An example of a landscape scene with a sunstar captured with the Sony FE 12-24mm F2.8 GM at 12mm f/11 (Credit: Jim Fisher)

Flare is another concern for lenses when you take photos with the sun or another bright light source in or just outside the frame. We check to see how a lens handles such backlit scenes in terms of contrast, false color, and internal reflections. If a lens comes with a protective hood, we make sure to properly attach it before testing.

Flare can cut contrast and exacerbate false color issues in lenses (Credit: Jim Fisher)

Lenses for mirrorless camera systems commonly rely on in-camera corrections to compensate for distortion and vignette. We consider this to be an integral part of their design, so we don’t knock them for these issues as long as the corrections don’t detract from an image. But we do report any distortion that’s visible in uncorrected Raw pictures, and call out instances in which distortion correction is so extreme that it affects picture quality.

You can see the barrel distortion and darkened corners in the uncorrected version of this image (left half) versus a version with the proper lens correction profile applied (right half) (Credit: Jim Fisher)

Finally, we look at focus performance for a lens. We test how quickly and accurately it locks onto a subject and drives focus across its range, and evaluate the manual focus experience too. For the latter, we consider the size and dampening of the focus ring, as well as whether it supports a linear or nonlinear response (or both). For cinema use, a linear response is desirable because it allows you to repeat takes from shot to shot. For video-friendly lenses, we test the throw distance, the angle of rotation a lens requires to move from close to distant focus.

We also take focus breathing into account since the angle of view for many lenses blooms or shrinks along with changes to the plane of focus. This matters for video shots that rack from one subject to another, and, for stills, this effect can make it difficult to stack multiple exposures for a larger depth of field or macro composition. Most photo lenses breathe to some extent, but we note when the effect is significant.

For our latest coverage, check out our camera page, or for more pointed buying advice, peruse the list of the best cameras we’ve tested.