Best Practices in Mental Health Reform: Discussion Paper

• Return to Table of Contents
• Previous section

APPENDIX A - Summary of Research and Evaluation Directions from the Literature Review (ANMH)

Evaluation must be carried out at all levels of the mental health system: system wide, program and individual. Methodologies required for systems evaluation will differ from those used at program and individual levels. The following section briefly outlines the key principles, derived from the Phase I Literature Review, which should be observed in all future research and evaluation of mental health programs and services/supports.

• The importance of conducting evaluations using methodologies appropriate to the program/service studied was emphasized by various investigators.
• Wherever possible, the use of experimental or quasi-experimental designs with control or comparison groups is optimal. However, the use of participatory research (involving "subjects" in the design, conduct and utilization of research plus the scientist's involvement in action to improve group or program functioning [Chesler, 1991]) and qualitative designs in the study of self-help groups and consumer/family initiatives is recommended by most researchers in these areas.
• In all evaluation endeavours it is necessary to clearly define and describe the nature of the experimental intervention, including all program elements, to make it easier to relate specific elements of the intervention to outcome and to increase the generalizability of the findings.
• Similarly in relation to outcome, researchers must provide a clear description of the outcomes to be studied, and to use standardized, reliable instruments. The review of the research highlighted the difficulty in achieving experimental effects in the measurement of symptoms and functional outcomes which was ascribed to problems of measurement; the increased sophistication of community programs which reduces differences between experimental and control conditions; and the severity of the illness in patients being studied and limited gains which can be expected during relatively short study periods and typically short follow-up. Longer follow-up of at least two years is recommended.
• In order to determine the cost effectiveness of the program/service it is strongly recommended that cost data be included in all research. Cost effectiveness data must take into consideration the perspectives of clients/patients, family, society and health systems.
• Just as consumers and families must be involved in planning and delivering services/supports, so must they be meaningfully involved in the design and delivery of the evaluations of those programs/services.

Recommendations for future evaluations of specific core services/supports

Case Management

Case management programs and assertive community treatment (ACT) have far more research evidence than any of the other core mental health services/supports reviewed in Phase I. In particular the ACT model has demonstrated its effectiveness and acceptability to clients and families in multiple trials. However, there are still some areas which need further study. Research still needs to be conducted to determine which elements of ACT model programs are related to outcome. This is particularly important since new programs cannot always implement the model programs faithfully due to local or other circumstances. Research needs to be conducted to determine optimal length of intensive ACT service provision and methods of transitioning clients into less intensive community care programs. As mentioned above, all research requires longer term follow-up in order to gain maximal results.

Crisis Response

Research in this area is very sparse. Although the various components of a crisis response system are widely accepted and implemented, no systematic evaluations have been carried out.

Housing/Community Support

The literature contains numerous articles examining aspects of the experiences and outcomes of residents in specific settings or models, but the design and methods of the research remains weak, limiting our ability to draw inferences from the results. Although the supported housing model is gaining wide acceptance as the preferred model, there is sparse research evidence regarding effectiveness to support this choice. Carling (1990) proposed a step-by-step approach for evaluating supported housing:

• define the purpose of each evaluation
• operationalize the concepts involved and the program characteristics
• formulate questions that are relevant to the key constructs in supported housing (i.e., choice, types of living arrangements, and services and supports)
• look beyond traditional mental health outcomes (i.e., to quality of life, physical and material well-being, personal relationships, social, community and recreational activities, and personal development/fulfilment)
• systematically examine the characteristics of programs and residents, and the impact of programs on residents and families and
• study the process of implementing/adapting programs.

Inpatient/Outpatient Care

The British experiences with evaluating deinstitutionalisation provide ample support for the benefits of subjecting all similar endeavours to careful research, and of examining a broad range of client and systems-level outcome indicators in such investigations. Evaluations of new community-based psychiatric services often report that they are more cost effective than hospital care but seldom examine whether such options are more or less effective for specific types of individuals. Researchers are urged to include data and analytic methods which can generate predictive data to guide future policy and program development.

In addition, since much of the care of less severe mental disorders is carried out by primary health care providers, this area deserves more attention by researchers, program developers and policy makers. The promising results of evaluative work (eg., Ferguson et al., 1992 Kates et al., submitted) highlight the importance of developing and assessing new service delivery models which feature cooperation between mental health practitioners and primary care settings.

Consumer and Family Initiatives

Until recently there has been little systematic evaluation of the effect of self-help groups. Research on self-help approaches for individuals with mental illness and their families lags behind advocacy for and even operation of such programs. Early research is mostly descriptive, for example, presenting and typing various self-help approaches, describing members of self-help groups , or how mental health professionals feel about or interact with mutual support groups for psychiatric patients. A small but growing body of studies are evaluating the experiences and outcomes of individuals who participate in self help and consumer/family initiatives and these efforts should be supported and expanded. As discussed briefly above, participatory research and qualitative methods are appropriate for use in evaluating these types of programs/supports.

Employment

A number of well-designed studies have demonstrated the benefits of supported employment (SE) programs for individuals with serious mental illness (SMI). Researchers in this field have suggested further areas which require more investigation.

The issue of how long to provide employment support and the best types of support have not been adequately studied. Studies have shown that clients who receive long-term and continuous support keep their jobs longer than those receiving intermittent or short term support.

• Attention needs to be paid to the process of accessing SE services. Traditionally professionals have decided when clients are ready for vocational rehabilitation. This goes against current thinking which says that consumers have the right to decide which services they need and research findings which suggest that if clients are given adequate information about programs and services they will be able to make appropriate choices.
• Once access is assured, researchers should investigate the characteristics which lead to retention in the SE program. The support of a multidisciplinary case management team has been shown to increase retention. Bond and colleagues (1997) speculate that this occurs because services are time unlimited, readily available, include assertive outreach and are sensitive to fluctuations in the client's clinical condition. Closely related to this is the issue of duration of employment. The few studies which have looked beyond a two year follow-up suggest that longer periods of support will lead to higher employment rates. Job stability and satisfaction with the job are related areas which have not been carefully examined.
• Another little studied process is that of job development. Programs which require the client to take responsibility for finding a job do not appear to be satisfactory for most individuals with SMI. Staff roles and responsibilities in helping clients to find jobs and in the interview process need to be clarified. The issues of disclosure, contact between staff and employers and employers' obligations to make accommodations for SMI employees also require thoughtful study.