9 December 2021

With “lifetime scoring”, we will only recommend appliances that last

It’s no longer good enough for products to just perform well when they’re new.

In the biggest change in our product testing since the 1960’s, we are incorporating a lifetime performance score to help shoppers buy appliances they know will last.

It’s no longer good enough for products to just perform well when they’re new. We’re the first international consumer organisation to only recommend durable appliances that work well for a long time.

We’ve tested products for more than 50 years. That testing shows many modern appliances work well when they’re new, but some of them aren’t built to last. This change doesn’t mean we’ve thrown out our heritage – we still assess ‘as-new’ performance. However, we’re now including measures of reliability, owner satisfaction and repairability.

Our new lifetime scoring system has been applied to washing machines and standard (canister and upright) vacuum cleaners on consumer.org.nz. In the first quarter of 2022, the new scoring will be rolled out to other products, starting with large appliances.

Only models that perform well and meet all three measures will be recommended.

Considering only “as-new” performance, we recommended 17 of 49 washing machine models we’d tested, from six brands. Using lifetime scoring, with emphasis on durability as well as “as-new” performance, 14 models from just four brands are recommended. Some of those models were not previously recommended.

Our scoring change comes in response to manufacturers building products that don’t last and can’t be repaired. This “take, make, waste” approach is unsustainable. Waste needs to be seen as a design flaw. End-of-life recycling is important, but it’s a last resort to divert materials from landfill. Products need to be designed to be repaired, reused and refurbished – preventing them reaching end-of-life so soon.

This change came from data we collect in our annual reliability survey that has been conducted for more than a decade. Consumers told us that appliances don’t last as long as they used to, and they are getting increasingly harder to repair. This means shoppers are paying for needless replacements and the planet pays by another appliance ending up at the tip

This is only the start for lifetime reporting. We’ll develop the scoring further and, in the future, we’ll also include an assessment of how easy it is to repair a faulty product.

Find our new lifetime scoring for washing machines and standard (canister and upright) vacuum cleaners

Member comments

Get access to comment

Dwayne B.
15 Dec 2021
Consumer introducing yet more arbitrary factors into scoring.

I made another comment on this thread, but this is a different point:
Consumer seems intent upon "reinventing" scoring by adding arbitrary factors, thereby obscuring the actual performance of items tested.
One example of this in recent years is the distortion of scoring for toilet paper, which resulted in inferior products being elevated to the front row due to environmental scoring.
Now we see another arbitrary factor added.
Consumer is increasingly less interested in delivering objective test results and more interested in injecting policy-related matters into testing. This is a slippery slope.
Display your survey-based reliability results and environmental policy scores, if you must, next to actual performance scores, but don't mix objective measures with subjective opinion.
It's disturbing that Consumer does not appear to comprehend the dangers of adulterating scoring methodology.

Ray S.
13 Dec 2021
Ask those who fix things

Recognising the inherent difficulties, the people who fix the things which can be fixed - the front line techos - could merit consideration when enquiring about reliability and design. Some work perhaps with single brands, others with multiple brands. These people and their experiences are 'gold'. Perhaps there is a practicable way to tap in to such resources.

Joanne C.
12 Dec 2021
Brilliant

Just what I’ve been waiting and hoping for

Hugh W.
12 Dec 2021
Lifetime Reliability Scoresd

Well done. I currently have a failed movement issue with a Lorus watch that is only four years and eight months old. The issue may yet be resolved when our negotiations are complete. This watch was replaced under warranty when it was quite new. Now that movement has failed and I was quoted $113 to replace the movement in this $150 watch. The $25 watch that i bought to tide me over the original warranty replacement is still going strongly and accurately.

Norman H.
11 Dec 2021
Landfill reduction

Well done consumer. The technology has been around for years to make products that last. Instead we fill our landfills with junk. I would like to see government mandating that products should last a minimum amount of time

Daniela B.
11 Dec 2021
Appliances that last!

Well done Consumer Institute! reliable, long lasting appliances are definitely worth searching for. Lower environmental impact, saving time, and money as well I'm sure.

Faye S.
11 Dec 2021
BOSCH Beware!

I was led to purchase a BOSCH washing machine after reading Consumer's testing reports. The Bosch I have now is just under 12 Years old. Yes older I know but isn't this what this is about? The machine has required repairs continually throughout its lifetime, during its warranty and out of warranty.

It has been frankly a nightmare of an appliance. I would have spent more on repairs, parts, and labour now than the actual machine itself when I originally purchased it. No repair is under $ 150.00 so you can see how this has added up over time.

The times I have had to deal with BOSCH Customer Service have not been particularly good experiences, being made to feel that I am just a 'whinger' when I have simply stood up for myself as a consumer. I would never purchase a BOSCH appliance again or recommend a BOSCH appliance either.

Interesting considering how well they do in 'Consumer' testing. Again, though it's the 'Have you bought in the last 12 Months' question. Not 'How has your appliance been since you purchased it'. That would be the better one to ask in my view.

John R.
14 Dec 2021
The concept of variation in outcomes

I strongly praise this "lifetime endorsement" rating during test outcomes. I have however had an entirely different outcome to "lifetime" performance from a Bosch washing machine. I admit the screwdriver I pierced the rubber door grommet with may have required a man come to make a replacement but the machine is going strongly still some 22 years of family use.

I bring this to your attention only to illustrate there can be some variations to the "lifetime" measurements.

John M.
11 Dec 2021
testing durability.

Paul says he would love to be able to test durability of appliances. The inherent problem with the testing that Consumer does is that they are always testing appliances NEW off the shelf, and then form their reports around how good that appliance is at performing a certain task. But every manufacturer makes sure their machine works at its best, brand new! But for most appliances the decline of performance with use and age is what is important. So Consumer need to be doing their testing on aged machines. How would this be possible? OK. So at present they go out and buy a new machine to test. Instead of testing that particular machine (or in addition to testing that machine) they need to search for a Consumer member who has owned and used the same model for a time. Consumer then swaps the new machine for the used one (so the member gets a newer item as compensation for their trouble) and Consumer gets a real feeling for the downward performance curve.
One of the appliances which I think has huge potential in comparative deterioration rates is vacuum cleaners. Vacuums loose their suck as a gradual process and so it is almost impossible for the user to notice. I recall from when I was a kid how the electrolux salesman would call and demonstrate to my mum how much more powerful a new model would be but the mechanism of the new model was exactly the same (same power but they would change the body shape to make it look different). The power difference would be as a result of inadequate seals to prevent dust getting into the bearings and other moving parts of the motor and there really wasn't any incentive for the manufacturer to improve this design as they would end up shooting themselves in the foot from not being able to sell replacement models as easily!
I know the swap idea would have its problems (larger items especially would be limited to local members only because of freight) but it is necessary to remove the emphasis of value away from "brand new" performance. I'm sure manufacturers target initial performance over durability. You think reputable manufacturers wouldn't do that? Think back to what that car manufacturer was caught out doing to their cars!

Mark H.
11 Dec 2021
Variability is higher for the new factors so they appear to influence the overall score more than features

I repeat that this new approach makes the overall score unusable for me.
Assuming you weighted four factors equally (25% each) then the overall ranking is mostly determined by the factor with the greatest variability. I can see this order of importance in the two reviews you link to for washing machines and vacuum cleaners:
1. Owner satisfaction has 80 point range: 10-90%, 19-90%
2. Predicted reliability has 80 point range: 10-90%, 10-90%
3. Efficiency/Usability has 28 point range: 50-80%, Ease of use 59-84%
4. Performance has 23 point range: 60-72%, 56-89%

Now, I know that the four factors aren't weighted equally but I can no longer see how they are weighted so you've just made it a little more difficult to analyse the effects resulting from differences in varaibility between the factors.

Anyway, if you want to do a fairer weighting between factors then you could do it by standardising the ranges i.e. making the lowest score in each range 0% and the highest 100%.  So, in calculating the overall score, the range for washing performance would not be 60-72% but 0-100%. Likewise, Owner satisfaction would change from 10-90% to 0-100% too.

Paul - Consumer staff
11 Dec 2021
Hidden detail

Kia ora Mark,

I'm sorry you feel the new scoring is unusable for you. I can't offer an answer that will satisfy you there. Durability needs significant weight in our reporting. Our testing over many years shows that most of these products function perfectly well. We think, and many consumers tell us, that durability is more important in a purchase decision.

You can still sort our results by "washing performance". That as-new testing hasn't changed. Our test data is still reported for each model - so you can drill down to see which are better at rinsing or spinning water from your load, for example.

There is detail behind our score calculations that you can't see. We've taken some time getting to this point and we have uncovered, considered and dealt with the points you raise and many more during its development. There's no doubt we'll find places to improve it as we go, that's why we've started by rolling it out to washers and vacs. We'll learn from this as we extend it to other large appliances early next year.

Ngā mihi,
Paul Smith
Product test manager
Consumer NZ

W M & C M D.
11 Dec 2021
Survey Data

When you survey for appliances the first question is have you purchased in the last 12 months. This has made me ineligible for washing machines for nearly 30 years.
We purchased a Maytag that is still as good at the job when we got it, it has had a couple of simple repairs. Unfortunately they have now moved to the same style of electronics and drive systems as the others and aren’t as durable.we chose that model on the advice of a person who simply said I am making a fortune repairing xxxxx washing machines, still a top ranking brand, still needing constant repairs and having a short life span.

Paul - Consumer staff
11 Dec 2021
Reliability of products bought in the past 5 years

Kia ora,

Our reliability survey asks for data on products you've bought in the past 5 years. I'm sorry if that wasn't clear in the survey question - I'll go and check that to make sure it's clear for future surveys. Some other satisfaction surveys we do limit the time to 12 months (such as the recent retailer satisfaction survey).

We would ask for reliability data going back further, but we've found that 5 years is about the limit for our memories. When we asked for data going back 10 years, we found people got vague about when they bought a product and what faults happened. Colleagues at the UK consumer organisation, Which?, showed me the results of a recent survey they did gathering data on much older products. It clearly showed that, after the first 5-7 years, we generally rounded up and reported products as 10, 15, 20, 25 or 30 years old. That made the data very unreliable. It was great to see that people like you own such durable appliances though!

Thanks,
Paul Smith
Product test manager,
Consumer NZ

J W.
13 Dec 2021
I'm with W M & C M D.

Pfft! "people got vague"! If you're relying on people's memory then the data you gather is pretty much useless. Perhaps the first question should be "Do you have receipts for the purchase and any subsequent repair?".

Dwayne B.
10 Dec 2021
Dubious.

The reliability data you gather from your surveys is itself unreliable, and often based upon far too few respondents to be statistically significant.

Paul - Consumer staff
11 Dec 2021
Statistical significance

Kia ora Dwayne,

I'll refer you to my much too long and rambling response to Mark's point below.

But to summarise as succinctly as you have here, we only report statistically significant results for reliability and satisfaction. We use a test of confidence to show which results are statistically better or worse than average.

I'm amazed at the number of responses we get to our surveys. For example, our last reliability survey (May 2021) gathered data from 2616 washing machine owners. It meant we could analyse 13 brands, covering all brands we've tested except Speed Queen.

There will always be some brands we can't analyse, because they are new to the market (so there is less data and that data doesn't show if the products become faulty as they get older) or those that have a small market share (there aren't enough owners out there). We're making sure those brands aren't penalised in our assessment. We're also working hard to increase the number of survey responses we get to increase the number of brands we can report on, and reduce any errors in the survey data.

Ngā mihi,
Paul Smith
Product test manager, Consumer NZ

Mark H.
09 Dec 2021
User reliability/satisfaction data has pitfalls when included in the overall rating

Yes, reliability and satisfaction are useful attributes but the usefulness of your overall rating for any product has declined for me. It now includes a lot more factors that I cannot check in the product showroom.
Reliability and satisfaction are factors that depend upon usage and the user. These are factors that I can't control for when making a buying decision.
I regularly read reviews on many consumer-focused websites where users have given negative reviews when they have not read the product manual and/or have not used the product as intended. Why would the consumer data be any/much different?
Consumer lifetime performance data tends to focus on problems which means that the other end of the spectrum is under represented.
Finally, lifetime data is often a lifetime old. We'd probably be better off with an assessment of current factors such as build quality, design, repairability, reusability, etc for the current product rather than including in the rating estimates based on older products.

Sarah T.
11 Dec 2021
Reliability- an essential

Whenever buy a major item reliability and longevity are paramount considerations. For 40 years I have relied on Consumer reports. Hence I switched to imac and get 10 years rather than an average of 4 years with 3 previous android based machines. I currently have a Bosch diswasher 10 years, going strong-1 minor repair caused by user blockage. My 23 year old Miele clothes dryer- no repairs fully functioning. CI always refer to consumer for my purchases and have consistently found them reliable save for a recently bought Breville toaster.
I suggest there could be one or two more questions in your surveys eg ist amout paid for each repair,specify type of repair and age when repair needed. Your bands for years of ownership need to be more precise esp after 5 yers- perhaps 2 year gaps till 15 and over.
Mark criticses your surveys for focusing on problems. They should do. His criticism that"the other end is underrepresented" is factually incorrect. The satisfaction surveys you have cover precisely that.

Paul - Consumer staff
11 Dec 2021
In response to the pitfalls

Kia ora Mark, thanks for raising a few points. Apologies now because this reply has become huge - so I'm back at the top here after writing it to warn you it goes on a bit!

We use a statistical analysis of consumer data, so our ratings don't rely on the opinions of one or two people. We also report the relative differences between brands, not the absolute number of faults - so we allow for the reality that someone with a faulty product is more likely to respond to a "reliability survey" than someone who owns a product that quietly gets on with its job.

Similarly, our survey data no doubt contains data from people who never read the manual, and those who follow every letter of it. But the nature of a statistical analysis accounts for those extremes - we report the results of the overall population. The challenge is to make the population of people in our survey match the population of everyone who owns that product. We can never survey everyone, so there will always be a margin of error. Reducing that error usually means gathering more data - something we work hard to do. I'm always amazed at how many responses we get to these surveys. For example, our washing machine analysis includes data from 2616 consumers. We have been able to analyse 13 brands with statistical confidence. We can still do better though.

I agree that people don't use products as the designers intended. We don't read the manuals, we find shortcuts to save time and effort, and many of us are pretty bad at maintaining stuff. But not following the manual or using a product exactly as you should is human nature. If a product is designed to be used in a particular way, and that's not how is is used in reality, that's a design fault - not user error. It's the job of the designer and manufacturer to allow for reasonable use, not to try to correct it through instructions buried in a manual.

That's why we have a Consumer Guarantees Act, and don't just rely on manufacturers offering a warranty. Whereas a warranty is filled with conditions specifying how it must be used, the CGA is defined by the actions and expectations of a "reasonable" consumer. It recognises that it's reasonable to expect that products won't be used to the letter of the instruction manual.

To your final point of lifetime data being a lifetime old. Yes, we are using reliability data for appliances up to five years old to predict the reliability performance of a product you can buy today. We've got data from over a decade of annual reliability surveys. We've analysed that and found that, as you might expect, the reliability of a brand changes incrementally, or not at all year to year. I plan to publish that data early next year - it's very interesting (if you're sort of data-nerd I can relate to :-)) We don't see, for example, a brand that makes very unreliable products suddenly finding the magic formula for durability. So we are confident that if, for example, washing machines made by brand X in the last few years have proved to be relatively reliable, we can say with some confidence that the machines you see from brand X in store today will be relatively reliable. Of course - we could never say that the exact product you buy will be reliable, reliability and statistics don't work like that.

I'd love to be able to physically test durability of products. As a design engineer, I get really excited at the thought of having a row of washers churning away until they break down (far too excited if I'm honest). But that sort of testing is of very limited use. We'd have to test a lot of machines to find faults (as most faults are relatively rare), then test more to prove that the fault wasn't just a one-off. That would cost a fortune (certainly more than the test budget of a non-profit consumer advocacy organisation). It would also take a long time, so we wouldn't be able to report on the models you see in the shops - they'd be gone by the time our testing showed anything.

Having said that - we are working with other consumer organisations to explore how we can test for certain critical failures. I'm very interested in testing electronic control boards - that's a common fault and expensive fix, and there's a way we can speed up the durability testing of those components.

Overall, we recognise that this is a big change. There is no easy way to measure and report durability (the manufacturers know that and use it to their advantage). We've made a start. We're confident in the robustness of our assessment and believe our approach is fair and transparent. However, we recognise its limitations. We're working hard to improve what we do, as we can't keep testing and reporting as-new performance alone. We hope manufacturers and importers put a similar effort into making their products more durable.

Paul Smith
Product test manager, Consumer NZ