Measuring positive mental health in Canada: construct validation of the Mental Health Continuum–Short Form - HPCDP: Volume 37-4, April 2017

Correspondence: Heather Orpana, Public Health Agency of Canada, 785 Carling Avenue, Ottawa, ON K1A 0K9; Tel: 613-878-5011; Email: heather.orpana@phac-aspc.gc.ca

Abstract

Introduction: Positive mental health is increasingly recognized as an important focus for public health policies and programs. In Canada, the Mental Health Continuum-Short Form (MHC-SF) was identified as a promising measure to include on population surveys to measure positive mental health. It proposes to measure a three-factor model of positive mental health including emotional, social and psychological well-being. The purpose of this study was to examine whether the MHC-SF is an adequate measure of positive mental health for Canadian adults.

Methods: We conducted confirmatory factor analysis (CFA) using data from the 2012 Canadian Community Health Survey (CCHS)-Mental Health Component (CCHS-MH), and cross-validated the model using data from the CCHS 2011-2012 annual cycle. We examined criterion-related validity through correlations of MHC-SF subscale scores with positively and negatively associated concepts (e.g. life satisfaction and psychological distress, respectively).

Results: We confirmed the validity of the three-factor model of emotional, social and psychological well-being through CFA on two independent samples, once four correlated errors between items on the social well-being scale were added. We observed significant correlations in the anticipated direction between emotional, psychological and social well-being scores and related concepts. Cronbach's alpha for both emotional and psychological well-being subscales was 0.82; for social well-being it was 0.77.

Conclusion: Our study suggests that the MHC-SF measures a three-factor model of positive mental health in the Canadian population. However, caution is warranted when using the social well-being scale, which did not function as well as the other factors, as evidenced by the need to add several correlated error terms to obtain adequate model fit, a higher level of missing data on these questions and weaker correlations with related constructs. Social well-being is important in a comprehensive measure of positive mental health, and further research is recommended.

Keywords: mental health, positive mental health, surveys and questionnaires, factor analysis

Introduction

It is increasingly acknowledged that a state of health is not the "absence of disease," and that both physical and mental health are required for comprehensive well-being. Within this context, the role of mental health is gaining attention because it is associated with better functioning, physical health and ability to contribute to society.^{Footnote 1} The Public Health Agency of Canada (PHAC) defines mental health as "the capacity of each and all of us to feel, think, and act in ways that enhance our ability to enjoy life and deal with the challenges we face. It is a positive sense of emotional and spiritual well-being that respects the importance of culture, equity, social justice, interconnections and personal dignity."^{Footnote 2,p.3} This is similar to the World Health Organization definition which states, "Mental health is a state of well-being in which an individual realizes his or her own abilities, can cope with the normal stresses of life, can work productively and is able to make a contribution to his or her community."^{Footnote 3}

A number of measures of positive mental health have been developed. These include the Warwick-Edinburgh Mental Well-Being Scale (WEMWBS),^{Footnote 4} Diener's Flourishing Scale,^{Footnote 5} Huppert and So's scale,^{Footnote 6} the PERMA profiler based on Seligman's model of positive psychology^{Footnote 7} and Keyes' Mental Health Continuum (MHC).^{Footnote 8Footnote 9} Most have been developed in the context of understanding individual differences in well-being, with the exception of the WEMWBS, which was developed specifically to monitor population well-being.

The MHC is a measure of positive mental health developed by Keyes8 based on a three-factor model of well-being including emotional, psychological and social well-being. The emotional well-being component assesses positive affect, life satisfaction and interest in life, and is informed by the work of Bradburn^{Footnote 10} on positive affect and Cantril^{Footnote 11} on life satisfaction. The psychological well-being component assesses functioning in six domains-autonomy, mastery, personal growth, positive relations with others, purpose in life and self-acceptance-as developed by Ryff.^{Footnote 12} Finally, the social well-being component is based on the work of Keyes. It reflects individuals' appraisals of their experiences in society and includes the five concepts of social contribution, social acceptance, social coherence, social actualization and social integration.^{Footnote 13}

The original Mental Health Continuum tool included subscales for emotional, psychological and social well-being, and consisted of 40 items. Keyes used a distribution-based criterion to identify a high level of positive mental health, called "flourishing" and defined as scoring in the top tertile of at least one of the emotional well-being scales, and at least six of the 11 social and psychological well-being scales. Using this approach, 17.2% of the sample was considered to be "flourishing," 56.6% was moderately mentally healthy, and 12.1% was "languishing."8 Because the initial MHC tool was lengthy, Keyes subsequently chose "the most prototypical items representing the construct definition for each facet of well-being"^{Footnote 9,p.1} to create the 14-item Mental Health Continuum-Short Form (MHC-SF), which is widely used today and the focus of this study. It is unclear whether any analysis aside from investigator judgment was used to inform the choice of items for the short form of the scale. The MHC-SF was implemented in both the 2011-2012 Annual and 2012 Mental Health cycles of the Canadian Community Health Survey (CCHS).

While the MHC-SF can be used as a continuous scale score, most frequently it is used to report on the prevalence of "flourishing," which is defined as a respondent responding "almost every day" or "every day" to at least one of the three emotional well-being questions, and six of 11 psychological and social well-being questions (at least one plus at least six). Keyes proposes that this approach to identifying "flourishing" is "similar to the standard used to assess and diagnose major depressive episode."^{Footnote 8Footnote 9} We would argue that it is not necessarily appropriate to apply an approach to diagnosing a mental illness to the identification of positive states of mental health. Furthermore, this approach to identifying "flourishing" can result in an individual with only high scores on psychological well-being (six items), and only low scores on social well-being (five items), being described as flourishing, which is inconsistent with the theory that positive mental health requires high levels of emotional, psychological and social well-being.

A number of studies have confirmed the three-factor model and adequate scale functioning of the MHC-SF. Keyes et al.^{Footnote 14} conducted a confirmatory factor analysis (CFA) on scale responses in a sample of Setswana-speaking South Africans and confirmed the three-factor model. Lamers et al.^{Footnote 15} reported that analyses supported the three-factor model, but also found that the social well-being subscale had only moderate reliability, while the psychological and emotional well-being subscales had high reliability. A three-factor structure was also confirmed in a study with Dutch, South African and Iranian respondents.^{Footnote 16} However, in order to obtain adequate model fit, some item residuals were allowed to covary.

Studies that have measured flourishing using this approach (at least one plus at least six) have produced a range of prevalences of flourishing: 49.3% among American college students,^{Footnote 17} 37.9% among American youth,^{Footnote 18} 20% in a Setswana-speaking sample of South Africans,14 and 28.5% among Italian adults.^{Footnote 19} Using this approach to scoring the MHC-SF, the prevalence of flourishing among Canadians was 76.9% in 2012.^{Footnote 20} The markedly higher prevalence of flourishing in Canada compared to other countries has raised concern about the functioning of this scale on population surveys in the Canadian context.

The purpose of this paper is to describe the measurement properties of the MHC-SF in the Canadian context, to examine its factor structure, and to propose a conceptually and empirically consistent approach to reporting on positive mental health based on this set of questions. Criterion-related validity and internal consistency are also examined. Positive correlations with the MHC-SF subscales were expected with self-rated mental health, life satisfaction and sense of belonging to the community. Negative correlations were expected with the WHO Disability Assessment Schedule (WHODAS), negative social interactions and psychological distress. Cronbach's alpha coefficients were expected to demonstrate good internal consistency without redundancy of items, with scores in the mid 0.80 range.^{Footnote 21}

Methods

Data

The 2012 Canadian Community Health Survey-Mental Health Component (CCHS-MH) is a household sample of 25 113 Canadians aged 15 years and over in the 10 provinces excluding individuals living on reserves and other Aboriginal settlements, Canadian Forces personnel and residents of institutions.Footnote 22 Less than 3% of the Canadian population was excluded. A multistage sampling design based on the Labour Force Survey was used to ensure adequate coverage by age group and sex, in each province. Response was voluntary. The combined (household and person) response rate was 68.9%. The data collection period was from 2012 January 02 to 2012 December 31. Data were collected using computer assisted personal interviewing (CAPI) and telephone interviewing (CATI); 87% of interviews were conducted in person. Proxy interviews were not permitted for this survey. Statistics Canada calculated sample weights to ensure that weighted estimates represented the Canadian household population in the 10 provinces.

The Canadian Community Health Survey-Annual Component (CCHS) is an ongoing household sample of approximately 65 000 Canadians aged 12 years and over in the 10 provinces and three territories, excluding individuals living on reserves and other Aboriginal settlements; full-time Canadian Forces personnel; persons living in the Quebec health regions of Région du Nunavik and Région des Terres-Cries-de-la-Baie-James; and residents of institutions.Footnote 23 Less than 3% of the Canadian population is excluded. The CCHS uses three sampling frames to select the sample of households: an area frame based on the Labour Force Survey (40.5%), a telephone list frame (58.5%) and a random digit dialling frame (1%). Multiple cycles can be combined. Data used for these analyses were collected from 2011 January 01 to 2012 December 31. The combined (household and person) response rate for 2011-2012 was 68.4%. Data were collected using both CAPI and CATI. Statistics Canada calculated sample weights. We obtained data from Statistics Canada; only microdata from respondents agreeing to share their data with the Public Health Agency of Canada and Health Canada were included.

Measures

Variables common to both surveys

Positive mental health

The MHC-SF consists of 14 items reflecting emotional, psychological and social well-being. All items are asked using the following format: "During the past month, how often did you feel…?" Response choices include "never," "once or twice," "about once a week," "about two or three times a week," "almost every day," and "every day." In the CCHS, Item 6 is worded according to the original MHC-SF questionnaire: "society is becoming a better place for people like me." Since its addition to the CCHS, however, Keyes has suggested changing this item to "society is becoming a better place for all people" or "our society is a good place," but this change has not been implemented on the CCHS surveys.^{Footnote 9}

The original scoring method ascribed the values of 1 through 6 to response categories "never," "once or twice," "about once a week," "about two or three times a week," "almost every day," and "every day." However, we created an alternative scoring method that more accurately reflects the underlying metric of the response categories, converting the semantic content of the response category into days and a ratio scale. "Every day" was ascribed a value of 28 days (4 weeks × 7 days per week); "almost every day" 20 days (4 weeks × 5 days per week); "about two or three times a week" 10 days (4 weeks × 2.5 days per week); "about once a week" 4 days (4 weeks × 1 day per week); "once or twice" 1.5 days; and "never" 0 days. We used both scoring methods for the CFA in separate models.

Demographic variables

Sex, age group, best estimate of household income, marital status and employment status were all self-reported, except in the case of nonresponse on income, in which case Statistics Canada imputed missing data.

Variables on the CCHS-MH 2012

The following variables were used in analyses of criterion-related validity:

Self-rated mental health (SRMH) was measured by a single question which asks the respondents, "In general, would you say that your mental health is: excellent, very good, good, fair or poor?" This question was also dichotomized as "fair," "poor" and "good" versus "very good" and "excellent."

Life satisfaction was also measured with a single question: "Using a scale of 0 to 10, where 0 means 'very dissatisfied' and 10 means 'very satisfied,' how do you feel about your life as a whole right now?" This question is consistent with OECD recommendations on measuring life satisfaction.^{Footnote 24}

Sense of belonging was measured with a single question: "How would you describe your sense of belonging to your local community? Would you say it is...: very strong, somewhat strong, somewhat weak, very weak?"

Social support was measured through the Social Provisions Scale by Cutrona and Russell.^{Footnote 25} Ten questions were used to measure five types of social provision: attachment, guidance, social integration, reliable alliance and reassurance of worth. We used a continuous scale score ranging from 0 to 40.

The Kessler Psychological Distress Scale (K6) was developed in order to discriminate cases of serious mental illness from no-cases based on nonspecific psychological distress symptoms in the US National Health Interview Survey. We used the six-item version (K6) for these analyses, with a continuous scale score ranging from 0 to 24.^{Footnote 26}

The CCHS-MH 2012 implemented the World Health Organization Disability Assessment Schedule 2.0 short version (12 items).^{Footnote 27} Scores range from 0 (no disability) to 100 (full disability).

Negative social interactions were measured using four questions assessing exposure to negative social interactions based on work by Krause.^{Footnote 28} We combined responses on the four questions to create a scale score ranging from 0 to 12.

Analysis

Descriptive statistics

We calculated descriptive statistics and a correlation matrix for all items in the test sample (CCHS-MH 2012). We calculated Cronbach's alpha for the entire instrument and three subscales. We conducted our analyses in SAS Enterprise Guide 5.1 (SAS Institute Inc., Cary, NC, USA). All analyses were weighted to account for the multistage sample design. Because bootstrapping was not available for CFA or correlation analyses, we used rescaled weights for all analyses.

Confirmatory factor analysis

The CCHS-MH 2012 was used to test the original model. We conducted confirmatory factor analysis (CFA) with robust maximum likelihood (RML) estimation in Mplus version 7.3 (Muthen and Muthen, Los Angeles, CA, USA). RML estimation allows for the use of weights and also accounts for nonnormality of data when calculating errors. Then, we fit the final model against the CCHS 2011-2012 sample to test whether the respecified factor structure was also confirmed in a different sample. While there are disagreements in the literature about what constitutes adequate fit, we adopted those suggested by Hu and BentlerFootnote 29: a value of at least 0.95 for comparative fit index (CFI) and Tucker-Lewis index (TLI); a value less than 0.08 for standardized root mean square residual (SRMR); and a value less than 0.06 for root mean square error of approximation (RMSEA). Because Mplus does not currently have a method to incorporate bootstrap weights into CFA, we used rescaled weights divided by the average design effect for all CFAs after consultation with Statistics Canada.

Criterion-related validity

We examined criterion-related validity through correlation analyses between MHC-SF subscales and positively and negatively related concepts (self-rated mental health, life satisfaction, sense of belonging, social provisions, psychological distress, negative social interactions and the WHO Disability Assessment Schedule). Because we are reporting multiple correlations in validation analyses, we chose a conservative p-value of < .001, which is equivalent to a Bonferroni correction for 50 analyses.

Results

Confirmatory factor analysis

The CCHS-MH 2012 sample reflected the Canadian household population living in the provinces. About half of the sample was male, and the mean age was 47.2 years. Just over three-quarters of participants responded to the survey in English, 22% responded in French and less than 1% responded in another language. Almost two-thirds of respondents reported a somewhat strong or very strong sense of belonging to their community. The mean life satisfaction score was 7.95 on a 0-to-10-point scale. Seven percent of respondents had a level of distress on the Kessler Psychological Distress Scale (K6) greater than or equal to 9, which is consistent with levels of distress seen among people experiencing a depressive episode or anxiety disorder^{Footnote 26} (data not shown).

The CCHS 2011-2012 reflected the Canadian household population living in the provinces and territories. Men made up about half of the survey population. The mean age of respondents was 47 years. English was the most frequent language of response (74% of respondents), while 21% responded in French and 5% responded in another language (data not shown).

MHC-SF responses

Responses to MHC-SF items on the CCHS 2012 were generally high on a scale of 1 to 6 (Table 1). Mean responses on a 6-point scale ranged from a low of 3.78 on Item 6, "our society is becoming a better place for people like you," to a high of 5.44 on Item 11, "you had warm and trusting relationships with others." Univariate normality of responses using the original scaling was poor: kurtosis was greater than 3.0 for four variables. In contrast, kurtosis and skewness were acceptable for all variables when the days scaling was used. Missing data on the CCHS 2012 sample was less than 2% for all items, except for three items on the social well-being subscale. One of these items (Item 6) approached 5% of missing data (4.8%). Ninety percent of the sample had complete data on all 14 items.

There were substantially higher levels of missing data on the MHC-SF items in the CCHS 2011-2012 sample, ranging from 5.7% to 14.0%, but item means were similar. Missing data were highest on the items for the social subscale (7.7%-14.0%). Correlations between items ranged from 0.58 to 0.62 for the emotional well-being subscale; 0.28 to 0.55 for the social well-being subscale; and 0.36 to 0.51 for the psychological well-being subscale (data not shown). Correlations in these ranges indicate good factorability of items. Cronbach's alpha approached an acceptable level for the emotional well-being subscale at 0.82, the psychological well-being subscale at 0.82 and the social well-being subscale at 0.77.

We tested the three-factor structure proposed by Keyes through CFA using robust maximum likelihood (MLR) estimation in Mplus. First, we ran the model using the original scaling (1 to 6) (Table 2, model 1a). This model had inadequate model fit. We then ran the model using the scaling representing number of days per month (Table 2, model 1b). While this model had better fit than the first model and RMSEA (0.036) and SRMR (0.043) were acceptable, CFI (0.927) and TLI (0.911) still indicated insufficient model fit.

Modification indices suggested that allowing the following residuals of items to covary would yield the greatest improvements to model fit: 8 with 6, 8 with 7, 7 with 6 and 6 with 5. The model including these correlated errors (Table 2, model 2a) had adequate model fit: CFI, TLI, RMSEA and SRMR were all above accepted thresholds. Aikake information criterion (AIC) decreased by 3775.646 with 4 degrees of freedom. While this three-factor model had acceptable fit, this was only possible with the addition of four covarying error terms that could not be explained in a theoretically meaningful way. This model is presented in Figure 1.

Figure 1
Confirmatory factor analysis of the Mental Health Continuum-Short Form, standardized coefficients

Abbreviations: Emo, emotional well-being; MHC-SF, Mental Health Continuum-Short Form; Psych, psychological well-being; Social, social well-being.

Note: Items refer to questions 1 through 14 on the MHC-SF.

We fit this three-factor model against data from the CCHS 2011-2012, to confirm the stability of the model. In this sample, the three-factor model that included correlated error terms on the social well-being factor demonstrated adequate fit (Table 2, model 2b), suggesting that this three-factor model is stable across samples.

We computed a correlation matrix between the continuous subscale scores and related concepts as shown in Table 3. The correlation between the emotional and psychological well-being subscales was 0.62. Both subscales had positive and significant correlations with life satisfaction, self-rated mental health, sense of community belonging and the Social Provisions Scale. Life satisfaction and self-rated mental health were most strongly correlated with emotional well-being (0.57 and 0.47, respectively).

The subscales had negative and significant correlations with the concepts of psychological distress, negative social interactions and the WHO Disability Assessment Schedule, with stronger correlations between the subscales and psychological distress than for negative social interactions or the WHO Disability Assessment Schedule. All correlations were significant at a conservative p < .001 level.

In contrast, the social well-being scale had weaker correlations with related factors than either the emotional or the psychological well-being scales. In particular, the correlations with social provisions and negative social interactions with the social well-being scale were lower than expected, at 0.28 and −0.21, respectively. The correlation with sense of belonging, however, was moderate, as anticipated, at 0.40.

Discussion

Strengths and limitations

This study explored the factor structure and psychometric properties of the MHC-SF in the Canadian household population, and examined criterion-related validity of resulting scale scores. The CFA supported the theoretically based three-factor structure in the Canadian adult household population either with the original scaling (1 to 6) or when MHC-SF responses ranging from "never" to "every day" were scaled as days ranging from 0 to 28, as long as four correlated error terms were added to the model.

All of these correlated error terms were on the social well-being scale, which when combined with multiple lines of evidence (higher levels of missing data on this factor, lower correlations with related concepts), warrants caution in the use of this subscale. The poorer functioning of the social well-being scale is consistent with findings from other studies.^{Footnote 19} We are proposing that the emotional and psychological well-being subscales be used as measures of positive mental health in the Canadian population, noting that an alternative measure of social well-being may be needed.

We examined the criterion-related validity of the emotional, social and psychological well-being subscales by examining the correlations between these scores and a number of concepts that we theorized would be positively associated and a number that we theorized would be negatively associated. We observed significant positive correlations with both subscales for life satisfaction, self-rated mental health, sense of community belonging and social provisions. We observed significant negative correlations for psychological distress, negative social interactions and the WHO Disability Assessment Schedule.

Generally, concepts that were more closely related to emotional or psychological well-being (such as life satisfaction to emotional well-being) had stronger correlations than concepts that were less closely related to these concepts (such as WHO Disability Assessment Schedule to psychological well-being). These findings support the criterion-validity of the measures.

Other measures for convergent validity would have been ideal, such as self-esteem and self-actualization for psychological well-being; however, we were constrained by the existing content on the surveys used for these analyses. Future research could also examine the factor structure of measures of both mental health and mental illness simultaneously, similar to the approach taken by Lamers et al.Footnote 15

Limitations of factor analysis should also be acknowledged. Decisions based on modification indices can result in a model that is overfit-that is, includes terms that do not have substantive meaning or that represent sample-specific characteristics that do not cross-validate. Because additional terms should be included only when they can be theoretically explained, while we chose to include the covarying error terms that modification indices suggested, it is difficult to justify these from a theoretical perspective.

There was a decrease in the proportion of missing data between the implementation of the MHC-SF on the CCHS 2011-2012 and the CCHS-MH 2012. This could indicate that the inconsistent functioning of the scale, as compared to its functioning in other studies, may be due in part to interviewer training. These analyses should be repeated the next time the MHC-SF is implemented on a large-scale Canadian survey.

Conclusion

The findings of our study suggest that while as a whole, the construct validity of the MHC-SF is supported in two large, Canadian samples, caution is warranted when employing the social well-being subscale in Canadian population surveys. Further research is required to determine why this subscale did not function as well as in other samples, what might be the cause of this differential functioning and what alternative measures may need to be developed. Given this finding, we recommend the use of an alternative measure (such as sense of belonging to community) to report on the social well-being in the Canadian adult population until a more comprehensive measure is identified and tested. Further research should examine whether the approach presented here is similarly valid in other populations and study settings.

Acknowledgements

We would like to acknowledge the use-ful suggestions provided by the peer reviewers of this article. We would also like to thank Louise McRae for providing helpful comments about this project dur-ing its development.

Conflicts of interest

The authors have no conflicts of interest to disclose.

Authors' contributions

HO, JV, JD and GJ contributed to the study design. HO analysed the data. HO, JV, JD and GJ interpreted the data. HO drafted the paper. HO, JV, JD and GJ reviewed and revised the paper. All authors read and gave final approval of this version to be published and agreed to be guarantors of the work.