There has been considerable comment, in some cases critical and often uninformed, of our research methods. We address some of the issues which have been raised.

Should mystery shopping be used to research the financial adviser industry?

Mystery shopping is a long-established research technique and it is often applied in the financial services sector. Regulators such as the Financial Services Authority in the UK (FSA) and the Australian Securities and Investments Commission (ASIC) have relied on mystery shopping research to inform their policy making. Mystery shopping of financial advisers allows regulators and others to examine industry practices, assess the consumer experience and decide whether policy settings are appropriate.

In a report released in November 2009 the Australian Parliamentary Joint Committee on Corporations and Financial Services, which held an inquiry into financial products and services, proposed that ASIC undertake annual “shadow shopping” of financial advisers – far from being an irrelevant research tool, mystery shopping is becoming an increasingly important part of the regulatory landscape.

It is not only regulators who rely on mystery shopping. Financial services firms in overseas markets, and here too we understand, use mystery shopping to research their competitors and their own staff. However, the results of this research are not released publicly.  Mystery shopping is a reality of the market place as we are sure most New Zealand-based advisers are well-aware.

Was Consumer NZ only interested in beating up financial advisers and looked for bad news to report?

Our purpose in undertaking this research was clear – to document the current consumer experience of the adviser industry. We undertook qualitative research which was capable of identifying issues (good and bad) present in the industry. We had no prior expectations of the outcome.

We felt it was important that relevant, factual information about the advice and information supplied to consumers be collected and reported.

We believed establishing a clear record of the current outcome for consumers would be helpful to other consumers as they seek financial advice. Providing objective, factually based information for consumers is our core business.

We also thought it important to gather factual information which we and others could use to inform ourselves during this important period of policy debate. To our knowledge, no other group or agency has attempted to provide objective evidence to the community, ensuring the policy debate is inclusive.

We, and the agencies that were approached before we began our research, also saw value in documenting the consumer experience in 2009 as it would provide a benchmark by which the effect of reforms could be assessed. Consumer NZ conducted the research and the findings are entirely its own. The research was supported by the Retirement Commission, the Ministry of Economic Development and the Securities Commission.

Did Consumer NZ investigate the industry in the right way?

The approach we took was objective, well-researched and appropriate for the industry being assessed. We asked two simple questions – are consumers being given advice that is likely to be in their interests and are they given sufficient information to make an informed decision. The expert panel answered these questions in each case.

We had no prior expectations of the outcome. That the research was objective is obvious from our use of real people seeking advice about their own situations. There was no pre-set scenario to test the industry. The industry was observed going about its usual business and each adviser had an opportunity to shine.

The approach was similar to that used by ASIC in 2003 in that we asked real people who were seeking advice about their personal situation to be our shoppers. Another important similarity is that the advice and information was assessed by a three-member panel made up of a mixture of advisers and others. Peer-review was an essential part of our approach and ASIC’s.

Despite the fact that our sample was smaller than has been used by ASIC the results are similar. ASIC reported that among the “deficiencies” it found were:

• failing to show how the recommended strategy and action was appropriate for the client
• plans were hard to read and ‘padded’ with reams of generic information
• higher-fee investments (including wrap accounts and master trusts) were recommended without showing why these were better
• planners recommended selling existing investments without showing how new investments or investment vehicles would be better.

ASIC also reported the judges commented many plans were “unclear, poorly written, swamped with generic material and difficult to follow”.

Our research turned up similar issues.

That there was similarity between our results and those of ASIC’s (much larger) 2003 study confirms the robustness and relevance of our research.

Should we have chosen different people for the panel?

The panel members selected from the industry represented the range of experience, opinions and business models in the industry today. On any one day we had one representative from the Institute of Financial Advisers, a client adviser from Gareth Morgan Investments and an expert from outside the industry.

While we could quite legitimately have chosen a consumer representative for the non-industry position, we elected to ask a highly-regarded academic with expertise in economics and finance. Complaints about our choosing an academic with this background reveal a poor understanding of the knowledge, analytical and technical requirements of financial planning, finance and economics. 

In each case the panel reached a consensus about the advice and information given. No one member could dictate the outcomes. The panel did an excellent job of identifying issues relevant for each plan and we would not hesitate to use any of the panel members again.

We have documented the failings of the material reviewed. No matter who was present on the panel, they could not have produced out of thin air clear cost information, missing analysis or a rationale for the recommended strategies. Many plans were simply missing the essentials. It is time the industry accepted the issues our research has raised. It is also time for the industry to provide a credible solution for consumers.