Agreement between survey data and Régie de l'assurance maladie du Québec (RAMQ) data with respect to the diagnosis of asthma and medical services use for asthma in children - CDIC: Vol 34, No 4, November 2014

Volume 34 · Number 4 · November 2014

Agreement between survey data and Régie de l'assurance maladie du Québec (RAMQ) data with respect to the diagnosis of asthma and medical services use for asthma in children

C. Plante, MSc (1); S. Goudreau, DESS (1); L. Jacques, MD (1, 2, 3, 4); F. Tessier, MSc (1)

https://doi.org/10.24095/hpcdp.34.4.09

This article has been peer reviewed.

Author references:

1. Direction de santé publique de l'Agence de la santé et des services sociaux de Montréal, Montréal, Quebec, Canada
2. Department of Environmental and Occupational Health, School of Public Health, Université de Montréal, Montréal, Quebec, Canada
3. Clinique de médecine du travail et de l'environnement, Montreal Chest Institute and Hôpital Notre-Dame, Montréal, Quebec, Canada
4. Department of Family Medicine, McGill University, Montréal, Quebec, Canada

Correspondence: Céline Plante, Direction de santé publique de l'Agence de la santé et des services sociaux de Montréal, 1301 Sherbrooke Street East, Montréal, QC H2L 1M2; Tel.: 514-528-2400 ext. 3285; Fax: 514-528-2459; Email: cplante@santepub-mtl.qc.ca

Abstract

Introduction: The goal of this study was to assess the agreement between the results of a respiratory health survey conducted in Montréal on children aged 6 months to 12 years and the Régie de l'assurance maladie du Québec (RAMQ, Quebec health insurance board) database in terms of the diagnosis of asthma and medical services use. A secondary aim was to evaluate the effect of the survey method used (Internet-based survey or telephone survey).

Methods: We assessed whether a diagnosis of asthma was made for 7922 children. In addition, we compared the use of medical services for asthma (emergency department visits and hospitalizations) in the 12 months preceding the survey for the 402 children considered to have asthma, using 2 groups of respiratory diagnoses and 2 data linkage periods. The agreement between the 2 data sources was evaluated using the kappa statistic (κ) and sensitivity and specificity, as well as percentages of agreement, overreporting and under-reporting with respect to health services use.

Results: Moderate agreement was found between the 2 data sources (survey and RAMQ data) in terms of the diagnosis of asthma (κ = 0.54 and κ = 0.60 depending on the definition used). Specificity was high (93% and 96%), but sensitivity varied (50% and 65%). Respondents over-reported health services use, resulting in moderate kappa values (0.49 for emergency department visits and 0.48 for hospitalizations). However, when more diagnoses were included in the definition and when the linkage period was extended (15 rather than 12 months), the kappa values increased (0.59 for emergency department visits and 0.64 for hospitalizations) and sensitivity and specificity were high. Slightly higher agreement was obtained for the Internet-based survey relative to the telephone survey.

Conclusion: The findings validate the use of survey data with respect to the diagnosis of pediatric asthma and major health services use for this disease.

Keywords: asthma, respiratory diseases, child, health survey, validation, administrative databases

Introduction

Questionnaire-based or telephone interview surveys are often used to obtain information on population health and health services use, particularly for chronic diseases, for example, asthma. The validity of the information gathered in surveys of this type may, however, be called into question since the data may be affected by both non-directional error (date errors, failure to understand information provided by physician, etc.) and directional error (social desirability, errors in recall, etc.). Nevertheless, the validity of survey data on health services use and on diagnoses can be assessed by comparing them to administrative health services databases.

The purpose of this study was to evaluate the concordance between a survey conducted in Montréal, Quebec, and the Régie de l'assurance maladie du Québec (RAMQ) database with respect to the diagnosis of asthma and use of medical services for asthma in children. The medical services we investigated consisted of emergency department visits and hospitalizations (for asthma) in the previous 12 months. We also assessed the effect of the survey method (Internet-based or telephone survey) on agreement. Through this research we set out to confirm the suitability of using survey data for studies on asthma, particularly pediatric asthma, as well as for studies on other chronic diseases.

The literature reports varying levels of concordance between self-reported data and administrative data with respect to chronic diseases and associated symptoms. For example, Lix et al.^{Endnote 1} found a low level of agreement for irritable bowel syndrome. Martin et al.^{Endnote 2} and Robinson et al.^{Endnote 3} reported high levels of agreement (along with high sensitivity and specificity) for hypertension and diabetes and lower levels of agreement for hypercholesterolemia. Kriegsman et al.^{Endnote 4} reported agreement levels ranging from low for various forms of arthritis and atherosclerosis to high for diabetes in a sample of community-based senior patients. Boyer et al.^{Endnote 5} found significant discrepancies between patient recall and medical records for symptoms of a form of arthritis. In effect, using survey data could lead to under- or overestimating the prevalence of chronic diseases.^{Endnote 4, Endnote 5, Endnote 6} In general, self-report data and administrative data showed better agreement for chronic diseases requiring repeated use of medical services (e.g. diabetes) and lower agreement for hard-to-diagnose diseases.^{Endnote 6, Endnote 7}

Lix et al.^{Endnote 1} found moderate agreement between self-reported data and administrative data, depending on which definition of asthma was used. Similarly, a Manitoba-based study conducted by Huzel et al.^{Endnote 8} found moderate agreement between the prevalence (in the previous 12 months) of self-reported symptoms of asthma in adults and the data in an administrative database. A Coloradobased study of several chronic diseases found good concordance between patient survey data and medical records for asthma.^{Endnote 9}

In a British Columbia study, Palin et al.^{Endnote 10} found that individuals tend to overestimate the number of their mental health visits to health professionals. In contrast, an earlier Quebec study indicated that psychiatric patients' self-reports about health services use were generally consistent with the administrative data.^{Endnote 11} Tisnado et al.^{Endnote 9} found varying levels of agreement for a range of chronic diseases, including asthma, with respect to health services use.

Previous research comparing survey and administrative data focused on the health status of the respondents themselves. To our knowledge, none of the studies used guardians' survey responses about their children in such comparisons.

Methods

Description of survey

Data were from a cross-sectional epidemiological survey of the respiratory health of nearly 8000 residents of the Island of Montréal, Quebec, aged 6 months to 12 years.^{Endnote 12} The goal of the epidemiological study was to identify the factors associated with the distribution of respiratory diseases (specifically asthma, allergic rhinitis and respiratory infections) in children in order to guide preventive actions and health care decisions.^{Endnote 12} Participants were selected from a random list of 17 661 families obtained from RAMQ. The questionnaire was designed by drawing on a literature review on the determinants of these diseases in children and questionnaires from other studies.^{Endnote 13, Endnote 14, Endnote 15} The data were collected in 2006 using a mixedmode survey methodology: telephone and Internet. The overall response rate was estimated to be 60%. The final sample consisted of 7964 subjects.

Definition of survey variables for comparison purposes

To confirm diagnosis of asthma, we used an affirmative answer to the survey question "Has a doctor ever said that your child has asthma?" The questions about the use of medical services for asthma concerned only a sub-sample of children in the survey since only those guardians who answered "yes" to the first question as well as to the question "Has your child had one or more asthma attacks in the last 12 months?" were required to fill out the corresponding part of the survey. We created 2 variables related to medical services use based on the answers to 2 subsequent survey questions–one concerning emergency department visits ("In the last 12 months, did your child have to go to emergency because of asthma?") and another concerning hospitalizations ("In the last 12 months, did your child have to be hospitalized for at least one night because of asthma?"). Another subquestion in the survey focused on the exact number of visits. When the respondent answered "yes" to the main question but "I don't know" to the question about the number of visits, we assigned a value of "1" (this occurred in 3 cases for emergency visits but none for hospitalizations).

RAMQ data

In Quebec, health care is covered by the provincial health insurance board, RAMQ. For every consultation with a patient, the physician submits a billing claim to RAMQ using a specific procedure code and a diagnostic code. As a result, RAMQ data include almost all health services provided, with the exception of unbilled private health care and health care provided by physicians who are not paid on a fee-for-service basis. RAMQ provided us with information on the diagnosis, the date and place of consultation, and the physician's identification number and area of specialization for each consultation with the children in the survey (for the period from the child's birth date to the date on which the questionnaire was filled out). This information consisted of all respiratory diseases including pneumonia and bronchitis (ICD–9Endnote * codes: 460–519, 786.0, 786.2; ICD–10Endnote † codes: J00–J99, R06.0–R06.8, R05.0). The billing data were linked to medical procedures performed in a physician's office or in the emergency department or to a hospitalization. Children not registered with RAMQ were considered to never have been diagnosed with asthma.

The confidentiality of the individual–level data obtained from the health insurance board was guaranteed. Pairing of RAMQ data with survey data was authorized by the Commission d'accès à l'information du Québec.

Definition of RAMQ variables for comparison purposes

Diagnosis of asthma

Each child was assigned either negative or positive asthma status according to two definitions. Under the first definition, any child with at least one diagnosis of asthma (ICD–9 code 493; ICD–10 code J45) in the RAMQ database, whether made during a visit to a physician, during a visit to an emergency department or during a hospitalization, was considered to have asthma. If no diagnosis of asthma was made or if no visit took place, we considered the child as free of asthma. According to the second, more restrictive definition, the child had to have been diagnosed at least twice with asthma during his/her life in order to be identified as having asthma since physicians do not always mention this diagnosis to parents on the first visit.

Medical services use

We also used 2 definitions for emergency department visits and hospitalizations: a diagnosis of asthma or bronchiolitis, which can easily be confused with asthma in young children (ICD–9: 493 and 466; ICD–10: J45 and J21), and any respiratory diagnosis (ICD–9: 460–519, 786.0, 786.2 and ICD–10: J00–J99, R060, R06.8, R05.0). As the questions about emergency department visits and hospitalizations concerned the 12 months before the survey, the health service and hospitalization data extracted from the RAMQ database were from the same 12–month period. We also tested a slightly longer period, 15 months, in view of the difficulty respondents have in recalling the exact dates of consultations. Visits at doctors' offices were not retained for comparison purposes.

Linkage of the two data sources

We used different samples to compare asthma diagnoses and medical services use. For the asthma diagnosis, the data from the 2 paired sources included 7922 children, as the data related to these questions were missing for 42 of the 7964 children. The sample used for the comparison of emergency department visits and hospitalizations consisted of 402 children, based on the answers to the sub–questions in the survey (see the "Definition of survey variables for comparison purposes" section above).

Statistical analyses

We used the kappa statistic^{Endnote 16} (κ) and computed the sensitivity and specificity to evaluate the agreement between the survey data and RAMQ data for the diagnosis of asthma and medical services use. We used the scale developed by Landis and Koch^{Endnote 17} to assess the degree of agreement obtained with the kappa statistic.

Unlike the kappa, which is a measure of agreement that does not take into account the validity of the 2 sources, calculating sensitivity and specificity requires that one of the 2 data sources be considered a gold standard. Sensitivity indicates the percentage of correctly identified positive elements (true positives), and specificity indicates the percentage of correctly identified negative elements (true negatives). The kappa value increases in conjunction with these 2 measures. We calculated specificity and sensitivity using the RAMQ database as the gold standard. Although this database may contain errors, we assumed it is more accurate than a survey. In a study comparing RAMQ administrative data with medical records, Wilchesky et al.^{Endnote 18} found high specificity but lower sensitivity for the administrative data. We take this limitation into account when applying the second definition of asthma status, that is, the one requiring 2 diagnoses in the RAMQ database.

For the number of emergency department visits and the number of hospitalizations, we also assessed concordance by comparing the proportion of under–reports, overreports and perfect agreement as a function of the diagnosis considered (asthma or any type of respiratory diagnosis) and as a function of the pairing period (12 or 15 months). Using the z–test, we also compared the agreement values (kappa coefficients, sensitivity and specificity) for the online surveys and the telephone surveys.

Although the survey data were adjusted for the response rate by sub–area as well as for the children's age and sex, the analyses were conducted on unweighted data, since the latter are used to assess the real agreement between the 2 data sources.

Results

Table 1 shows the personal and socioeconomic characteristics of the children in the 2 samples used in the comparison of asthma diagnoses and the comparison of health services use (emergency department visits and hospitalizations). The 2 samples are reasonably similar in terms of these characteristics.

Diagnosis of asthma

Table 2 shows the extent of agreement between survey data and RAMQ data for the 2 definitions used for the diagnosis of asthma. Moderate agreement was obtained (κ = 0.54) for the first definition (1 diagnosis of asthma recorded in the RAMQ database) and higher agreement (κ = 0.60) for the second definition (2 diagnoses of asthma in the database).

The diagnosis of asthma based on survey data is highly specific (> 93%) but moderately sensitive (50%–65%), which indicates, in general, that children identified in the survey as not having asthma were accordingly classified in the RAMQ database, but a certain proportion of children with asthma were not identified in the survey. Both sensitivity and kappa values increase when there are 2 diagnoses in RAMQ data, whereas specificity is slightly reduced. Sensitivity was increased (to 54.9 for one diagnosis and 70.7 for 2 diagnoses) with the same specificity (96.5 and 93.5 respectively) by the exclusion in the study of children who were taking asthma medication but had no report of an asthma diagnosis in the survey (data not shown). The prevalence rates^{Endnote ‡} for asthma are very similar for the second definition: 16.2% (survey) and 16.8% (RAMQ data).

The results are almost the same when children aged less than 6 years were compared with children aged 6 years and over. However, with 2 diagnoses, the kappa value is slightly higher for the older group–which is to be expected since the diagnosis is more definite in older children–but this difference is not statistically significant (z-test at the 95% level; data not shown).

Emergency department visits and hospitalizations

The results of the analyses show that the number of emergency department visits and hospitalizations of children with asthma to be generally over-reported by guardians (Table 3). The survey data show 295 emergency department visits, whereas the RAMQ database records between 122 and 240 visits (depending on the period and the diagnoses considered). Agreement improved when a 15-month period was used or when the list of diagnoses was broadened to include all respiratory diagnoses in RAMQ data (versus diagnoses of asthma and bronchiolitis only). However, the number of visits reported by guardians in the survey was 18% higher than that recorded in the RAMQ database. For hospitalizations, there is a slight trend of over-reporting by the guardians, but the proportion of over-reporting is similar to that of under-reporting when all respiratory diagnoses and the 15-month period are considered (Table 3).

The kappa value for the occurrence of one or more emergency department visits (no visits versus one or more visits) ranged from 0.49 to 0.59, which is indicative of moderate agreement between the 2 data sources (Table 4). Sensitivity and specificity were good. The kappa value for one ormore hospitalizations (versus no hospitalizations) suggests moderate agreement for diagnoses of asthma or bronchiolitis in the 12 months before the survey (κ = 0.48) and a substantial level of agreement for any respiratory diagnosis in the previous 15 months (κ = 0.64) (Table 4). Sensitivity and specificity were high when considering one or more hospitalization for asthma or bronchiolitis in the previous 12 months (94.1 and 92.5). For both emergency department visits and hospitalizations, sensitivity was reduced while specificity was slightly increased by broadening the inclusion criteria in the RAMQ data (Table 4).

Comparison according to survey method

The kappa values for the diagnosis of asthma were higher for the Internet-based survey than for the telephone survey: 0.56 and 0.51, respectively, for the first definition (only one diagnosis in the RAMQ database), and 0.71 and 0.65, respectively, for the second definition (2 or more diagnoses in the administrative data) (data not shown). The difference is significant at the 95% level for the first definition only. Sensitivity was also slightly higher for the Internet-based survey compared with the telephone survey (52.6% versus 48.4% for the first definition, and 67% versus 63% for the second definition); however, these differences were not statistically significant at the 95% level. Specificity values and disease prevalence were similar for the 2 survey methods.

With regard to medical services use (emergency department visits and hospitalizations), sensitivity and specificity were similar for the 2 survey methods (nonsignificant differences at the 95% level) (data not shown). The kappa values likewise did not differ significantly. It should be noted, however, that the small sample size for medical services use resulted in wide confidence intervals for the estimates.

Discussion

The goal of our study was to validate the results of a survey of pediatric asthma conducted in Montréal, Quebec, among the guardians of children aged 6 months to 12 years against data in the RAMQ database with respect to the diagnosis of pediatric asthma. The 2 data sources were also compared in relation to number of emergency department visits and hospitalizations in the 12 months prior to the survey using a subpopulation of children with asthma.

Diagnosis of asthma

When comparing survey data and RAMQ data, moderate concordance (k = 0.60) was found for the diagnosis of asthma when a definition requiring 2 diagnoses in the RAMQ database was used. If the RAMQ data are taken as the gold standard, specificity is high and sensitivity is moderate. Sensitivity was partly influenced by the exclusion in the study of children who were taking asthma medication but had no report of asthma diagnosis in the survey.

Some studies have found levels of agreement between self-report data and hospitalizations and billing claims in the administrative database for diagnoses of chronic diseases range from poor to good.^{Endnote 3, Endnote 7, Endnote 8} Other studies have found good agreement with medical record databases. ^{Endnote 2, Endnote 9} Kriegsman et al.^{Endnote 4} showed strong concordance between patient survey data and general practitioner questionnaires for the majority of chronic disease diagnoses. For asthma specifically, the findings of our study are similar to those of other published studies that relate to adults. As mentioned earlier, Lix et al.^{Endnote 1} found moderate agreement (estimated with the kappa statistic) between survey data and hospitalization and prescription billing data for chronic disease diagnoses. Huzel et al.^{Endnote 8} also found moderate agreement for the diagnosis of asthma between self-report data and physician claims in the Manitoba health insurance administrative database using the kappa statistic. The study by Tisnado et al.^{Endnote 9} showed good agreement (self-report data and medical records in Colorado) based on the kappa statistic as well as sensitivity and specificity.

In addition, our study showed that agreement varies with the disease definition used: the definition that required 2 diagnoses of asthma in the administrative database yielded better agreement with guardians' self-report survey data than the definition based on a single diagnosis. The reason for this can be that physicians do not always inform guardians about the first diagnosis, either because they want to wait until the next visit when the results of diagnostic tests are available or because they believe that the health problem consists of a transient bronchospasm. The prevalence of asthma is similar for the 2 data sources when the definition requiring 2 diagnoses in the RAMQ database is used. The study by Lix et al.Endnote 1 also reported variation in agreement depending on the definition used; however, the definition requiring 2 diagnoses did not result in stronger agreement between the 2 data sources in the case of asthma.

Some studies have suggested that agreement is likely to be lower for complex or difficult-to-diagnose diseases,^{Endnote 6, Endnote 7} or in cases where the physician and patient do not have the same understanding of the definition of the disease.^{Endnote 1, Endnote 9} The fact that asthma is a disease that is not always easy to diagnose, especially in young children, could partly explain the non-concordant cases mentioned here. Using survey data to derive estimates of diagnoses of complex diseases should therefore be more difficult.

Medical services use

The study showed that the number of emergency department visits and the number of hospitalizations based on the survey data were overestimated relative to RAMQ data. Moderate agreement was obtained with the kappa statistic when the occurrence of at least one visit was used as a criterion (rather than the exact number of visits). This also resulted in good sensitivity and specificity. Some guardians may have confused an emergency department visit with a hospitalization if their child was kept under observation for a short period of time. This would lead to overestimation of the frequency of hospitalization based on selfreport data. This type of error may be negligible in a case where the degree of control over or the severity of the child's asthma is of interest, as measured by the fact that the latter needed a follow-up emergency medical visit at least once during the year.

Palin et al.Endnote 10 found over-reporting of mental health visits to a physician based on the Canadian Community Health Survey (CCHS) compared with the number of such visits recorded in the British Columbia Medical Services Plan administrative data. In our study, we found better agreement for emergency department visits and hospitalizations for asthma and bronchiolitis in children with asthma when a longer time period was used (15 months instead of 12 months). This is consistent with the findings of other studies.Endnote 3 Agreement also improved when the list of diagnoses was expanded to include all respiratory diagnoses: when a child presents with a respiratory infection, the physician may not write down the diagnosis of asthma. Palin et al.Endnote 10 found a similar trend when they broadened the definition of mental health visits to include all visits to the main general practitioner. In contrast, Robinson et al.Endnote 3 did not find an improvement when more than one diagnosis was used to define a chronic disease; however, asthma was not included in their study.

Our study showed slightly better agreement for online surveys than for telephone interviews with regard to the diagnosis of asthma based on a single diagnosis in the RAMQ database. This finding could be explained in part by the fact that respondents had more time to answer questions in the online survey.

Limitations

The main limitation of this study consists of the small sample size used for the validation of health services use. In addition, we were unable to assess agreement based on the exact number of visits. Despite that a preliminary analysis showed poor agreement based on the exact number of visits, the occurrence (versus the absence) of a visit or a hospitalization provided valuable information in terms of validating the survey.

Some uncertainty persists in relation to the use of the kappa statistic for low prevalence rates, which is the case in our study with respect to hospitalizations and emergency department visits. The kappa value may remain low even in the presence of a large proportion of concordant pairs.^{Endnote 19} However, in such a case, our results would be conservative.

Conclusion

Our study shows good agreement between guardian self-report data and RAMQ data in relation to pediatric asthma diagnoses. Overall, we found that the survey data over-reported the number of emergency department visits and hospitalizations in the previous 12 months. However, moderate agreement was found for the occurrence/ non-occurrence of at least one visit and at least one hospitalization. A higher level of agreement, based on kappa values, occurred for emergency department visits and hospitalizations when the linkage period was increased from 12 months to 15 months and when diagnoses of respiratory infections were added to diagnoses of asthma and bronchiolitis, but is associated with a decreased sensitivity. These findings validate the use of survey data for the diagnosis of asthma in children and the use of major health services for this disease.

Acknowledgments

The authors extend their thanks to RAMQ. We also want to thank Michel Fournier of the Direction de la santé publique de l'Agence de la santé et des services sociaux de Montréal for providing guidance on the statistical methods.

References