Experts upon Experts

It’s been a few weeks since the Ireland report regarding expert evidence in family proceedings was published. Apologies for not getting to it before. It coincided of course with a rash of articles about George Hibbert, one such expert who is the subject of outstanding complaints and disciplinary proceedings. I wrote about the George Hibbert coverage here.

Before we look at the substance of the report it’s worth giving some consideration to how it came to be written in the first place. The first reference I can find to it is in the Family Justice Review, who seem to have received a draft copy of the report. The FJR report says:

3.142.A report to be published shortly by the Family Justice Council has examined the quality of expert psychologist reports in a sample of private and public cases. It points to serious issues both with the quality of reports and the qualifications of those carrying them out. We are not surprised in view of the concerns we heard expressed throughout our work about the quality of reports generally. We recommend that studies of the expert witness reports supplied by various professions be commissioned by the Interim Board, subsequently the Family Justice Service.

3.143.Agreed quality standards for experts in the family courts are clearly needed and we recommend that they should be developed. The FJS should lead this work. Meeting the standards could be a requirement for payments to be approved by the LSC. Criteria could include adherence to set timescales, membership of appropriate professional bodies and completion of specified court focused training, peer review and continuing professional development. 

That was November 11, but the report was actually only published in mid March 12. It appears to have been part-funded by the Family Justice Council but published by the University of Central Lancashire.

The report’s author, Professor Ireland is a Forensic psychologist and academic at the University of Central Lancashire. Judging from the CV type information on the UCLAN website, Prof Ireland’s main area of experience appears to be around offenders and secure environments. I cannot see anything that suggests a particular experience in relation to family proceedings per se (although of course in many families there is much overlap between family and criminal justice). At the time Professor Ireland’s report was published she was a member of the Family Justice Council’s Experts Committee. I know this because I checked it at the time of publication, but I cannot provide a link because the Judiciary website no longer lists an Experts Committee (although there are various orphaned references to that committee, making me confident it is not a figment of my imagination – see minutes of FJC meetings in April and July last year here and here, and the Press release issued by UCLAN which quotes a Heather Payne, who is described as the chair of the FJC experts committee. Dr Payne is a Cons. Paediatrician and member of the FJC).

To the best of my recollection the FJC Experts Committee as listed on their website in mid March was comprised entirely of experts, some medical, with only one lawyer named (a female QC whose name I did not note). We’ll come back to that. Again, to the best of my recollection, at least one of the reviewers who assisted with the research was also a member of the FJC experts committee.

So that’s where it’s come from. But what have we actually got? The report that has been published is a summary report only. It is unclear if a more substantive report is in the pipeline. The author herself acknowledges the limitations of the study which was only ever intended to be a preliminary study to inform decisions about whether more robust work ought to be done (although you’d be unlikely to pick up on the qualified nature of its conclusions from the press coverage). But much of what Professor Ireland reports reflects my own experience of variable quality in the psychological reports that courts rely upon, and I agree with many of her sensible proposals.

I’m a bit puzzled though by a couple of things. In particular the assessment of experts reports by reference to the Civil Procedure Rules and something called the Daubert standard. I know that the Family Procedure Rules 2010 only came into force in the early part of 2011, but prior to that the Family Proceedings Rules applied and there was a previous PD on experts which was applicable from 2008 and was itself quite comprehensive and prescriptive (the files sampled were from between 2009-2011). Thankfully there is some commonality between the Civil Procedure Rules and the Family Procedure Rules, but I don’t understand why the author has assessed against what appears to be inapplicable criteria – the Family Procedure Rules and Practice Direction (even the previous incarnations) are far more detailed than the CPR and more tailored to the particular requirements and purpose of family proceedings). It suggests a lack of knowledge of the culture and practice of Family Proceedings (one could argue this might bring much needed objectivity from an observer not steeped in the prejudices and complacencies of a hermetically sealed environment, although I would tend to take a different view). We know that in November 2011 the report was still in draft stage, so I would have thought there was ample opportunity for the report to have at least made reference to the FPR – at the very least acknowledging that a new regime had recently been implemented which may have impacted upon practice since the timeframe from which the sample cases were drawn.

I wonder if the conspicuous absence of any reference to the governing framework that is applicable to this field is attributable to those involved in the commissioning and conduct of this research being more experienced in criminal and civil / PI fields than in family? I can go no further than wondering because the relevant information about the membership and makeup of the FJC experts committee is no longer publicly available, and it may well be that the CV available on the UCLAN website for Prof Ireland is not comprehensive. Regardless of which set of rules the reports were assessed against it is pretty clear that a significant number of reports assessed fell far short of either set of criteria . The report appears to question whether there should be more stringent rules and guidelines in place for the commissioning and admissibility of expert evidence, but I think that prior to that question comes the query as to whether or not the courts are consistently applying the guidance already in place, and if not why not? My experience and the qualitative data set out in the summary report suggests they are not consistently applying the guidance.

The Daubert standard is a reference to an approach to expert evidence applied in courts in (for example) the US. For the purposes of this post the wikipedia reference will do. In essence the Daubert standard says that if expert evidence is to be relied upon in court proceedings it must be based upon sufficient facts or data, be the product of reliable principles and methods, and of the application of the principles and methods reliably to the facts of the case. Although there have been prior calls for a test along these lines to be introduced in this jurisdiction (Law Commission Mar 11) this has not yet happened. Again, it seems odd on one level that the current practice in the courts is being assessed against criteria adopted only in other jurisdictions, although no doubt the author of the report would say that the findings of it demonstrate just why we need some kind of reliability test akin to the Daubert standard in this jurisdiction. Again, if one cross checks the previous and current PDs applicable to family proceedings these should go some distance towards that aim – if consistently applied and enforced.

The report states that:

Of those experts completing the reports, over one fifth were unqualified and were not registered either with the Health Professionals Council (HPC) or the British Psychological Society (BPS). Of the qualified psychologists, over two thirds were clinical psychologists, with the remaining either educational or forensic psychologists. Just under one fifth of the sample referred to themselves as consulting psychologists. Nine tenths of experts did not report a current assessment or treatment practice, with expert assessments their primary work. 

I carried out an entirely unscientific exercise myself earlier today to see how many of the psychological experts I could think of were registered with the HPC. Most were, a couple were not (although these were people I had not heard of recently so may simply have retired), and one was listed as an Educational Psychologist which came as something of a surprise given that my experience of him was as an expert assessing risk in relation to a parent. All of the experts I would “rate” were appropriately registered (I did not check the BPS). Most of the experts I regularly encounter are to my knowledge in clinical practice, but those who are best known are often also the most busy. It is not uncommon for them to be so booked up that an unknown name is put forward. I don’t doubt that there will be some who are not in clinical practice or who do probably far more of these expert reports than is healthy.

The problem with these striking figures however is that the report relies upon the data provided in the CVs on the court file, presumably separately indexed in the bundle (rare) or annexed to the body of the main report (rare) or set out in summary form as a paragraph incorporated into the start of the report (more common). There is no consistency of information provided on CVs, which in my experience range from the ridiculously lengthy to the totally useless. Often the focus is (understandably) upon the experts experience qua expert and it may well be that clinical or other expertise is not explicit or foregrounded in some CVs where perhaps it should be. It is highly likely that at least some of the files sampled will have included no CV – they often circulate at court but don’t make their way into the bundle. This is clearly an area for improvement and I wonder if there should be a template for CV information required prior to instruction of an expert in family proceedings, so that experts are able to set out their stall in a coherent way and courts and parties can make good choices about the appropriate expert at the outset. I agree that it should be a requirement that any party who wishes to instruct a psychologist should be permitted to instruct only an expert who is registered both with the Health Professions Council and the British Psychological Society.

The report also raises concern about the quality of the reports – inappropriate use of or reliance on psychometrics, insufficient analysis of data, failure to link data to opinion, and most concerningly a lack of actual psychology.

Fifty-three reports were indicated to have problems in linking the final opinion to the body of the report.  There were 126 reports in total, so this represents 42% of the sampled reports.

“To illustrate examples concerning problematic opinions, comments from raters on the assessed reports included:

  • –  “No backed up opinion, one paragraph for each instruction question”;
  • –  “Very little opinion; Thirteen lines of opinion and does not address thequestions”;
  • –  “Three pages of psychological assessment only”;
  • –  “Poor, just has no opinion and refers to facts so readers can make their own mindup”;
  • –  “Opinion very weak, report mainly a description of an interview that has littlefocus”;
  • –  “Fails to answer any of the instruction questions”;
  • –  “Huge assumptions expressed in the final opinion, not linked to facts e.g. sexualassault [unproven] may have led to angry acting out of Mrs X emotional turmoil”;
  • –  “Psychometrics used but not referred to”.
  • –  “Bases opinion on personality using the wrong measure”. “

These are highly concerning ratings, albeit qualitatively based. But for all that I’d quibble with the extent to which this is a problem I agree entirely that it is a problem. As has already been noted on the Suesspiciousminds blog there is a frightening diversity in the quality of psychological reports. Good reports – which I think are the majority – are enormously useful, but there are some real stinkers. You wait months and when they arrive they help no-one. Some seem to regurgitate everything they have read and then bolt on a paragraph of “opinion” at the end, whilst others operate on the basis of factual assumptions that are neither accepted nor proven. It is not uncommon for me to get to the end of a report and immediately go back to the start to see where I missed the actual psychology. Several thousand pounds and a few observations in the absence of the application of psychological theory is neither value for money, nor purposeful delay and nor does it aid the justice of the case. There are a few repeat offenders where the same stock paragraphs seem to reappear in each report, setting out general psychological principles.

That said I struggle slightly with some of the criticisms made by the rates that suggest experts have strayed into the remit of the judge by making a recommendation as to whether or not a parent can provide good enough parenting or on the question of rehabilitation. It’s unclear from the report whether or not the reviewers had letters of instruction (I would guess not if relying on the bundle, although most experts set out the questions in the main body of their report), but psychologists are often asked in terms to offer such recommendations.

I share the anxiety expressed about the use of expert witness commissioning companies, and think the suggestion that experts should be in clinical practice as well as undertaking expert work is sensible. I also agree that there should be further thought given to mechanisms to enable peer review and assessment of the quality of expert reports to be undertaken.

There is a recommendation for feedback to be given to experts. There is in fact a provision in PD 25 for experts to be notified of the outcome of proceedings and the use that has been made of their report, although how consistently this is applied I don’t know.

As ever, the issue for me is not in identifying the problem (inconsistent quality of expert evidence), or what needs to be done in order to remedy that (we already have a good template in the FPR and PD25), but in how we translate that into practice in the pressured environment of the family justice system. Nobody has the time or resource to implement this stuff. And whilst Ireland’s report has generated some punchy headlines and given grist to the mill of those who would happily condemn all experts, it can only go some distance in helping us in working out how to fix this on the ground. Everyone, from every branch of the system, has so much to fix – this is one more thing on a long old To Do List.

ISWs throw in the towel

Community Care reports on the prospect of Independent Social Workers quitting over the fees cap to be introduced by the LSC in October. What the article fails to disentangle (in fact it rather adds to the confusion) is the distinction between ISWs contracted by CAFCASS on a self employed basis to carry out Guardian or report writing work it does not have the in house resource to manage (and for which CAFCASS pays its own rate) and ISWs instructed as an expert by the parties with the permission of the court, and whose costs will be met (usually) by the LSC at its own rate. I know that there have been complaints about both the CAFCASS rate and the proposed rate via the LSC, and the thrust of the article is right: ISWs are finding it a real squeeze. Many are  quitting working for CAFCASS, or have already done so, only to find that their other source of self employed income i.e. expert instructions will also be severely limited.

.

It’s important to distinguish between a self employed CAFCASS Guardian and a social worker who is instructed as an independent expert in the case with the permission of the court. Their roles are quite different, and whilst each is independent in their own way it is not the role of a Guardian to conduct in depth assessments of a family or children. Where that work has not been completed by social services or has not been completed satisfactorily or with an independent and open mind the only means by which a parent can obtain a fair and thorough assessment is via ISW report. This type of report can turn a case around. If social workers flee not just CAFCASS, but refuse to undertake assessments at £30 an hour too, there will be injustices.