Public Health Agency of Canada
Symbol of the Government of Canada
Help the Government of Canada organize its website! Complete an anonymous 5-minute questionnaire. Start now.

Share this page

Chronic Diseases and Injuries in Canada

Volume 32, no. 1, December 2011

2008 Niday Perinatal Database quality audit: report of a quality assurance project

S. Dunn, PhD (1,2); J. Bottomley, MHA (3); A. Ali, MSc (4); M. Walker, MD (1,5,6,7)

This article has been peer reviewed.

Author references:
  1. Better Outcomes Registry and Network (BORN Ontario), Ottawa, Ontario,Canada
  2. Champlain Maternal Newborn Regional Program (CMNRP), Ottawa, Ontario, Canada
  3. Children’s Hospital of Eastern Ontario, Ottawa, Ontario,Canada
  4. Ottawa Public Health, Ottawa, Ontario, Canada
  5. OMNI Research Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa,Ontario, Canada
  6. Division of Maternal-Fetal Medicine, Department of Obstetrics and Gynecology, Faculty of Medicine, Ottawa General Hospital, University of Ottawa, Ottawa, Ontario, Canada
  7. Tier 1 Canada Research Chair, Perinatal Epidemiology, University of Ottawa, Ottawa, Ontario, Canada

Correspondence: Sandra Dunn, BORN Ontario, The Ottawa Hospital, 501 Smyth Rd. Room 1818, Box 241, Ottawa ON  K1H 8L6; Tel.: (613) 737-8899 ext. 72070; Email: sadunn@ohri.ca

Abstract

Introduction: This quality assurance project was designed to determine the reliability, completeness and comprehensiveness of the data entered into Niday Perinatal Database.

Methods: Quality of the data was measured by comparing data re-abstracted from the patient record to the original data entered into the Niday Perinatal Database. A representative sample of hospitals in Ontario was selected and a random sample of 100 linked mother and newborn charts were audited for each site. A subset of 33 variables (representing 96 data fields) from the Niday dataset was chosen for re-abstraction.

Results: Of the data fields for which Cohen’s kappa statistic or intraclass correlation coefficient (ICCC) was calculated, 44% showed substantial or almost perfect agreement (beyond chance). However, about 17% showed less than 95% agreement and a kappa or ICC value of less than 60% indicating only slight, fair or moderate agreement (beyond chance).

Discussion: Recommendations to improve the quality of these data fields are presented.

Keywords: audit, data quality, quality assurance, reliability

Background

The Ministry of Health and Long-Term Care (MOHLTCC) in Ontario recognized that producing and sustaining quality surveillance data is the foundation of an effective and efficient health system.Footnote 1 Surveillance is defined as the ongoing systematic collection, analysis and interpretation of health data essential to the planning, implementation and evaluation of public health practices, integrated with the timely dissemination of these data to key stakeholders.Footnote 2  A surveillance system can function as both measurement tool and stimulus for actionFootnote 3  by providing early warning of health problems and evidence for policy and program development, risk assessment, trend analysis and the evaluation of prevention and control strategies.Footnote 4  However, the usefulness of a surveillance system is limited by the quality of the data it collects and analyzes.

In Ontario, the Niday Perinatal Database (the “Niday”) is the source of data to assess outcomes, risk factors and interventions related to perinatal care. It was created in 1997 under the direction of the Perinatal Partnership Program of Eastern and Southeastern Ontario (PPPESOO) to provide perinatal data to PPPESO partners. This Internet-based system has evolved significantly since its inception and has become a unique co-operative venture with over 100 health care organizations across the province contributing real-time perinatal data. It enhances the ability of health care providers in different parts of the province and within different service sectors to work together to improve perinatal health. At the time of the audit, 96% of Ontario births were captured in the Niday, and there were 90 defined patient elements covering the full spectrum of perinatal care (Table 1). In 2001, the province adopted the variables in the Niday as the minimum dataset.

Table 1. List of variables in the Niday Perinatal Database in 2008 (n = 90) including variables chosen for re-abstraction as part of the 2008 quality audit

Linked data

  • City/towna
  • Provincea
  • Mother’s agea
  • Record typea

Identifying variables

  • Siteb
  • Maternal chart numberb
  • Baby chart numberb
  • Baby birth dateb

Maternal history variables

  • Mother’s birth date
  • Postal code
  • Languagec
  • Aboriginalc
  • Previous Caesarian section
  • Number of previous Caesarian sectionsd
  • Maternal health problemsc
  • Obstetrical complicationsc
  • GBS screeningd
  • GBS (35–37 weeks) resultsd
  • Maternal transfer from
  • If transferred, reasonc
  • Antenatal care providerc
  • First trimester visitc
  • Prenatal classesc
  • Smoking
  • Intention to breastfeedd
  • Number of previous term babies
  • Number of previous preterm babies
  • Reproductive assistancec
  • Multiple gestation
  • Maternal history commentd

Labour and birth variables

  • Labour type
  • If induced, indication (17)
  • If induced, method (8)
  • Number of induction attemptsc
  • Augmentationd
  • Intrapartum complicationsc
  • Maternal pain management (11)
  • Fetal surveillance (6)
  • GBS antibioticsd
  • Antenatal steroids
  • Labour/birth commentd
  • Forceps/vacuum
  • Episiotomy
  • Laceration
  • Presentationd
  • Delivery type
  • If Caesarian section, indication (20)
  • If Caesarian section, typed
  • If Caesarian section, dilatationc
  • Time fully dilatedc
  • Time start pushingc
  • Time of birth
  • Delivered by

Newborn variables

  • Newborn resuscitation (7)
  • Baby's sex
  • Gestational age
  • Birth weight
  • Apgar score 1
  • Apgar score 5
  • Apgar score 10c
  • Infant feeding in hospitalc
  • Reason for breastmilk substitutec
  • Infant feeding on dischargec
  • Hearing screeningc
  • HBHC screenc
  • HBHC screen if not sent, why?c
  • Arterial cord pHc
  • Arterial base excessc
  • Venous cord pHc
  • Venous base excessc
  • Congenital anomaliesc
  • Phototherapyc
  • Newborn commentd
  • Neonatal death / stillbirth
  • Neonatal discharge / transfer datec
  • Neonatal discharge / transfer timed
  • Discharge weightc
  • Discharged / transferred toc
  • Reason for neonatal transferc
  • Neonatal transfer hospital

User-defined variables fieldse

  • Birth nurse ID
  • Birth physician ID
  • Discharge time
  • Mother’s date of admission
  • Mother’s time of admission
  • Mother’s height (centimetre)
  • Removal of placenta 
  • Mother’s weight (kilogram)
  • Newborn drug screening
  • Newborn drug screen results

Abbreviations: GBS, Group B Streptococcus; HBHC, Healthy Babies Healthy Children.

Notes:
Total variables in Niday Perinatal Database in 2008 (n = 90): Mandatory 24 + Non-mandatory 66.
Total number of variables included in re-abstraction (n = 33/90; 36.7% - resulting in 96 data fields for audit). 
Mandatory variables (n = 20/90) (4 providedb).
Non-mandatory variables (n = 13/90).

  1. Mandatory variables - linked data (n = 4/90; 4.4%).
  2. Provided identifying labels.
  3. Missing> 10% data (n = 31/90; 34.4%).
  4. Not identified as a priority at the time of the audit (n = 12/90; 13.3%).
  5. User defined variables (n = 10/90; 11.1%) – not available to all sites.

This is the only database in Ontario that provides immediate access to real-time population-based perinatal data for an entire region. The Better Outcomes Registry and Network (BORNN Ontario) Steering Committee now manages the project. The involvement of most hospitals in the province also permits inter-hospital/health unit comparisons necessary for benchmarking and performance improvement based on learning from others’ successes. As the system evolves, BORN is committed to ensuring high quality data, with powerful and efficient reporting tools.Footnote 5

In light of the fact that approximately 40% of all live births in Canada occur in Ontario (37.1% in 2008/2009),Footnote 6 this database provides rich perinatal information for a large proportion of the births in Canada. Although it is well recognized that the foundation of an effective and efficient health system requires the production of quality data,Footnote 1 it was unclear whether the Niday, as configured, was a reliable source of information. The goal of this quality assurance project was to assess objectively the reliability, completeness and comprehensiveness of the data in the Niday Perinatal Database.

Methods

The Data Quality Management Framework,Footnote 7 developed by the MOHLTC Health Results Team for Information Management, was used to guide this project. According to the Tri-council policy, and given the fact this was a quality assurance project, Research Ethics Board approval was not required.Footnote 8 Hospital participation in this project was voluntary, and every effort was made to ensure the confidentiality of patient information and privacy of participating hospitals.

Data re-abstraction

In order to determine the reliability and completeness of the data, re-abstraction of information from patient records was carried out to assess agreement between selected variables in the perinatal database and the mother and infant charts. Written consent was requested from and given by each site participating in the re-abstraction phase of the project. Information was handled confidentially, and each auditor signed a Pledge of Confidentiality Form. The auditors re-entered data from the patient records that had already been collected and entered by the hospital data entry person into the Niday. The laptops used for data entry were supplied to the auditors and returned following the re-abstraction process. The electronic data were then securely transferred to the statistician for analysis and deleted from the laptops. Data were aggregated for analysis, and findings were anonymized.

Setting and sample size (hospitals)

Purposive sampling was used to recruit 14 hospitals for the audit representing five regions of the province: East/Southeast, Greater Toronto Area (GTAA), Central West, South West, and North. The sample captured both obstetrical and newborn care practices and included all levels of care: level 1, or low-risk pregnancies (4 of 51 hospitals in Ontario); level 2, or women/babies with health problems (8 of 37 hospitals in Ontario), level 3, or specialized care (2 of 7 hospitals in Ontario). A combination of both paper and electronic documentation systems and a variety of data entry processes were used by the sample hospitals.

Sample size

A computer-generated random sample of 100 maternal chart numbers (and linked baby records) was identified for each participating site from existing records that had already been entered into the Niday in 2008 (total of 200 charts per site). The total sample size for this project was 1395 linked mother-baby dyads; in three cases the patient charts could not be located at the time of the re-abstraction, and in two cases the chart numbers were not for a perinatal client.

Variables for re-abstraction

A subset of variables (33/90; 36.7%) from the Niday perinatal dataset was chosen for re-abstraction. Selection was based on the following criteria: a) a mandatory variable; b) a non-mandatory variable with less than 10% missing data based on verification reports; and c) a variable that addressed a practice issue of interest (e.g. use of antenatal steroids, indication for Caesarean section, episiotomy, lacerations, fetal surveillance, forceps/vacuum, indications for induction, method of induction, maternal pain relief, smoking). This resulted in 96 data fields available for re-abstraction because some of the variables consisted of multiple data fields (e.g. indications for induction included 17 data fields; maternal pain management included 11 data fields). Table 1 lists the variables selected for re-abstraction and those excluded (with rationale).

Auditors

Due to the wide geographic distribution of the participating hospitals, and the travel and time involved to complete an audit in 14 sites across the province, six auditors with a health care background were hired and trained to expedite the process. Two auditors entered data at five sites each and each of the remaining four auditors re-abstracted data at one site each. Figure 1 shows a flow sheet of the data collection process.

Figure 1 - The data collection process

Figure 1 - The data collection process

Figure 1, Text Equivalent

Each of the auditors was told about the project and trained in the re-abstraction process, including where to find the information in the patient record and how to use the SPSS (version 15.0) spreadsheet for data collection to ensure consistent re-abstraction. Each received a handout containing the definition of terms for each of the variables in the Niday, contact information for the project coordinator, a list of their designated hospital(s) and an SPSS spreadsheet with pre-entered sample data (maternal chart number, baby chart number, baby date of birth) for each of their designated sites. For practice, the auditors entered data into the SPSS spreadsheet based on the same two charts; inter-rater reliability was evaluated based on these cases.

Data collection procedure

Following the random chart selection process, a list of the patient records from each of the participating hospitals was prepared. The identifying variables used were the mother’s chart number and the matching baby’s chart number. For added precision, the date of birth was printed out for each baby. This enabled auditors to verify that the record entered was the correct one. In each of the 14 participating hospitals, a key contact person was identified and informed about the project by the project manager. The key contact was asked to assist (or designate someone who could assist) the auditors to obtain entry to the site, access the patient charts from health records and problem-solve any site-related issues. Prior to data collection, the key contact person (or designate) met with the auditor to show the patient documentation systems and where to find the key information.

Primary data abstraction took place from April to July 2008. Data collection for one site had to be repeated in October 2008 as the original file for this site was overwritten and the data were lost.

The charts (paper or electronic records) were obtained from the Health Records Departments of each of the participating hospitals. The auditors reviewed and re-abstracted the data using the standardized data entry procedures. The data were collected using an SPSS version 15 data file template. A spreadsheet was created that included the data fields under review and pull-down menus matching those found on the current Niday entry screen. For ease of data entry, the variables were placed in the same order as they appeared in the majority of hospital records. Data were entered into two portable laptop computers. Re-abstraction took two to four days per site, due to standard delays when accessing patient records and the time it takes to work through the information in each patient record. The project manager was available by pager, phone or email during the re-abstraction process to address any questions that arose.

Patient records

Although hospital patient documentation systems are not standardized throughout the province, the chart reviews were conducted as consistently as possible. Auditors were trained to obtain information from the same sources used for the original data entry. The postal code, mother’s age and maternal transfer from another hospital were obtained from the admission record; the rest of the variables were obtained from the labour record, the delivery record, the antenatal record, the discharge summary, lab results, nurses’ notes, doctors’ orders, medication records and the postpartum screening record. Terminology and the organization of the patient chart varied somewhat from site to site, but the overall layout of the information was similar. In one region, a standardized documentation system was used by all of the participating hospitals except one. All of the records were in either English or English/French.

Analysis

Descriptive statistics (frequencies, means and percentages) were calculated using SPSS version 15 to describe the characteristics of the study sample groups. The reliability of the data was assessed by comparing the re-abstracted data from the patient record to the original data entered into the Niday Perinatal Database. Cross-tabulations were generated to explore non-agreements and missing data in an attempt to identify potential reasons for the variation between the auditor and the original data entered for each field.

Although sensitivity and specificity can be used to measure the accuracy of data gathered from an external source compared to a primary source of information, this approach requires that one of the data sources is identified as the gold standard.Footnote 9  Many factors can affect the transfer of information from a patient record, such as observer variation, poor documentation, illegible charts, data loss, unavailability and timeliness of chart completion.Footnote 10 This makes it impossible to identify a gold standard from either the original data entered into Niday or the re-abstracted data entered by the auditors. When neither data source can be designated as the gold standard, high agreement between the two suggests high reliability. In other words, when two similar datasets are compared and a high proportion of the data are the same, then it can most likely be interpreted that they are both correct. This is an indicator of having high quality data.

Therefore, for the purposes of this audit, we used percent agreement, Cohen’s kappa statistic (κ) and intraclass correlation coefficient (ICCC) between the variables11 to compare the data newly re-abstracted from patient records with data previously entered into the Niday by the participating hospitals. Percent agreement was calculated for all variables. For kappa and ICC, categorical/nominal variables (n = 87), and continuous variables (n = 3) were considered separately.

Categorical variables

The analysis for all the categorical/nominal variables (except for postal code) was by two-way cross tabulations of each variable and comparison of the entries, as explained above. Since postal codes are string variables, cross tabulation was not feasible so an equivalent equal/not equal statement on the SPSS program was used to calculate the percent agreement.

We used Cohen’s kappa statistic to examine the proportion of responses in agreement in relation to the proportion of responses that would be expected by chance, given symmetrical marginal distributions.Footnote 12-14 Cohen’s kappa statistic represents the proportion of agreements after chance agreement has been excluded. Kappa values range from 0 (no agreement) to 1 (total agreement). According to Landis and Koch, a kappa value of 0.90 (or 90%) indicates almost perfect agreement while a kappa value of 0.55 (or 55%) reflects only moderate agreement.Footnote 15

Continuous Variables

For continuous variables, agreement was assessed using an equal/not equal statement on the SPSS program and by calculating the ICC. ICC is a more appropriate measure of reliability for continuous data than Pearson’s product moment correlation coefficient or Spearman’s rank-order correlation coefficient since these measure association rather than agreement.12-14 ICC values range between 0 (no agreement) and 1 (total agreement), “with values approaching 1 representing good reliability.”Footnote 16, pg. 357  According to Portney and Watkins,Footnote 17 an ICC of over 0.9 (or 90%) indicates excellent agreement, while an ICC of 0.35 (or 35%) indicates poor agreement between variables. The notes to Table 2 shows more detailed interpretation of kappa and ICC values.

Results

This quality assurance project evaluated the reliability, completeness and comprehensiveness of the Niday Perinatal Database and found that the database met expectations either fully or partially.

Reliability

A total of 33 out of 90 variables (96 data fields) in the Niday were re-abstracted from patient records to determine the degree of agreement with data already entered in the database. Of the 89 data fields for which kappa or ICC was calculated, almost one-half (n = 39; 43.8%) showed substantial or almost perfect agreement (beyond chance), suggesting that these variables may be used with confidence. Just over one-third of the data fields (n = 34; 38.2%) were found to have kappa values below the moderate level (60% beyond chance) despite having excellent agreement rates. However, a prevalence effect due to asymmetrical imbalances of marginal totals was the likely cause of the low kappa value in this group.18  The remaining data fields (n = 15; 16.9%) showed both percent agreement of less than 95% and a kappa or ICC value less than 60% indicating only slight, fair, poor or moderate agreement (beyond chance). This suggests these data fields may be problematic and require further investigation. Table 2 summarizes the percent agreements, Cohen’s kappa or ICC for each data field.

Completeness

Approximately 34% of the variables in the Niday were missing more than 10% of data based on verification reports generated prior to the start of the audit. Only variables that were mandatory or had low rates of missing data (< 10%) just prior to the audit were selected for re-abstraction (Table 1).

Missing (not entered) data were also evaluated as part of the re-abstraction and were found to be associated with the following variables: antenatal steroids, forceps/vacuum, episiotomy, laceration and smoking. The missing data were limited to only three sites (F, J and K; see Figure 1). The primary reason for missing data at these sites was due the auditors or original hospital data entry personnel deciding to leave a cell empty rather than selecting “none” or “unknown.” At site F the auditor left the field empty while the hospital data entry person entered “none” or “unknown,” while the reverse took place at sites J and K. Missing data was not a significant issue and these data points were not excluded from the assessment of agreement. This was not a surprising finding, given the fact that these variables were selected for abstraction in the first place because of high completion rates.

Comprehensiveness

At the time of the audit over 96% of births in the province (involving 95 delivering hospitals and including midwifery hospital births and some home births) were captured in the Niday. There were 90 defined patient elements with 23 mandatory fields (at the start of the audit).

Discussion

Although neither of the datasets used during the audit can be declared as a gold standard, the moderate-to-high levels of agreement (beyond chance) between the two sources suggest that the variables are comparable across two methods of data collection.Footnote 19  The worst case scenario in interpreting these findings would be that all the differences are due to having wrong data in the Niday. When there is a level of disagreement between the two data sources for some data fields, part of this difference may be as a result of wrong data in the Niday, wrong data entered during the audit, or wrong data in both datasets.

Although the reasons for non-agreements could not always be discerned, a variety of potential factors were identified during detailed exploration of the data. Results from the audit indicated disagreement between the two data sources occurred across multiple sites, and included both hospital and auditor data entry issues. These issues have been clustered into four themes (data entry choice, clarity of information, inaccurate documentation and human error).
The first issue related to choices available for data entry has to do with the designation given to some variables. At the time of the audit, all data fields in the Niday were designated as either mandatory or non-mandatory. In reviewing non-agreements, it was evident that in some cases the auditor found information in the patient record that the original hospital data entry person did not record. Although, both groups were tasked with finding and entering as much information as possible, in reality it is possible that discretionary completion of some of the non-mandatory data fields at some sites contributed to the non-agreements. This example illustrates the importance of ensuring that all data fields are mandatory and that only essential, meaningful data are collected.

The second issue related to this theme was about pick-list choices and the availability of information in the patient health record. If the information is not documented in the patient record in such a way as to match the pick-list choices, data quality can be affected. For example, in the case of smoking during pregnancy, documentation may indicate that a women smoked, but not provide the detail required to determine the duration of smoking through pregnancy (e.g. above or below 20 weeks as required for Niday at the time of the audit). In some cases where non-agreement occurred, it was because some people entered “unknown” while others left the field empty when the required data was not available in the patient health record. This example illustrates the importance of aligning documentation tools with data entry processes to enhance data quality.

The second theme has to do with clarity of information available for each data field. Confusion over the wording, use of double negatives and different interpretations of the definitions for some variables may have contributed to non-agreements (e.g. interpreting what qualifies as an induction or augmentation of labour). This example illustrates the importance of ensuring the definitions for each variable are precise and applicable to practice.

The third theme was related to inadequate, illegible or inaccurate documentation. Data entry is dependent on the accuracy of the information recorded in the patient health record. Even though specific documents were identified to be the source of information for data entry for both the primary and audited datasets, some of the information entered was difficult to find, or inconsistent, contributing to non-agreement. For example, gestational age and birth weight both require double entry of the data. Double entry of these variables may provide verification that the original number entered is correct, which enhances reliability of the variable, but it does not ensure validity of the information. This is evidenced by the discrepancies between the original data entered and the auditors’ data for these variables.

Finally, even though every attempt was made to ensure a consistent process for data entry, it is always possible that human error contributed to non-agreements between the two datasets. Results of this audit have provided information about potential issues related to data entry for some variables in the database. A number of variables were more problematic. Further exploration of the issues is required in order to develop strategies to improve the data quality for these variables in the Niday.

Interestingly, eight of the data fields identified in this audit as less reliable were also found to be problematic during a previous audit of the Niday (Table 2).Footnote 20  This is significant in that some of these variables have been identified as priority items highly relevant for the perinatal reports being developed by BORN Ontario.

Table 2. Comparison of abstracted data from patient records (N = 1395) and data entered in Niday Perinatal Database using percent agreement, Cohen’s kappa and intraclass correlation coefficient (ICC)
No. Variable Name Data Field Label Coding Not matched
n/1395 (%)
Percent agreement
(%)
Cohen’s kappa [κ]
(%)
ICC
(%)

Abbreviations: FF02, free flow oxygen; FS, fetal surveillance; ICC, intraclass correlation coefficient; IUGR, intrauterine growth restriction; LGA, large for gestational age; NST, non-stress test; PPV, positive pressure ventilation; PROM, premature rupture of membranes; SGA, small for gestational age.

Notes:
Cohen’s kappa statistic (κ) degrees of agreement after chance agreement has been excludedFootnote 15: Poor < 0; Slight = 0–0.20; Fair = 0.21–0.40; Moderate = 0.41–0.60; Substantial = 0.61–0.80; Almost perfect = 0.81­–1.00.
Intraclass correlation coefficient (ICC) degrees of agreementFootnote 17: Poor < 0.50; Moderate = 0.50–0.75; Good ≥ 0.75-0.90; Excellent > 0.90.
a N/A - not applicable as equal/not equal was used, hence no cross tabulation to generate the kappa statistic.
b Data fields with < 95% agreement and kappa or ICC values < 60% indicating only slight, fair, poor or moderate agreement (beyond chance).
c Data fields also found to be problematic during a previous audit of the Niday Perinatal Database.Footnote 20

Mandatory data fields
1. SITE Site name     Pre-entered    
2. Maternal chart number Maternal chart no.     Pre-entered    
3. Baby chart number Baby chart no.     Pre-entered    
4. Baby birth date Baby birth date - DMY     Pre-entered    
5. Number of previous preterm babies No previous preterm babies Number (0–15)
Unknown
64 (4.6) 95.4 54.5  
6. Number of previous term babies No previous term babies Number (0–15)
Unknown
79 (5.7) 94.3 91.2  
7. Previous Caesarean section Previous C/S Yes
No
Unknown
50 (3.6) 96.4 81.8  
8. Maternal transfer from Maternal transfer from Pick from site list
Planned home birth
Out of region
No transfer
35 (2.5) 97.5 25.0  
9. Multiple gestation Multiple gestation Singleton
Twin
Triplet
Quadruplet
Quintuplet
Sextuplet
Septuplet
1 (0.1) 99.9 98.8  
10. Labour type Labour type Spontaneous
Induced
No labour
135 (9.7) 90.3 81.8  
11. Delivery type Delivery type Vaginal
Caesarean section
Unknown
4 (0.3) 99.7 97.3  
12. Mother’s birth date Mothers birth date - DMY Date of birth (D/M/Y) 128 (9.2) 90.8 N/Aa N/Aa
13. Birth weight Birth weight b,c Birth weight (grams) 114 (8.2) 91.8   35.1
14. Gestational age at birth Gestational age at birthb Gestational age (weeks)
Unknown
119 (8.5) 91.5   32.0
15. Baby’s sex Baby gender Male
Female
Ambiguous
Unknown
29 (2.1) 97.9 96.0  
16. APGAR - 1 APGAR1 Number (0–10)
Unknown
58 (4.2) 95.8 92.5  
17. APGAR - 5 APGAR5 Number (0–10)
Unknown
51 (3.7) 96.3 87.7  
18. Newborn resuscitation Noneb Not checked
Checked
352 (25.2) 74.8 46.7  
19. Drugs 12 (0.9) 99.1 64.3  
20. FF02 118 (8.5) 91.5 70.2  
21. Intubation 10 (0.7) 99.3 63.9  
22. PPV 54 (3.9) 96.1 63.4  
23. Chest Compression 5 (0.4) 99.6 28.4  
24. Unknown b,c 86 (6.2) 93.8 3.0  
25. Neonatal transfer to Neonatal transfer hospital Pick from site list
No transfer (if birth hospital)
Out of region
11 (0.8) 99.2 50.0  
26. Neonatal death / stillbirth Neonatal death / stillbirth Not applicable
Stillbirth ≥ 20 weeks
Neonatal death < 7 days
Neonatal death > 7–28 days
2 (0.1) 99.9 50.0  
Non-mandatory data fields
27. Maternal postal code Maternal postal code Full postal code 97 (7.0) 93.0 N/Aa N/Aa
28. Antenatal steroids Antenatal steroidsb,c None
1 dose < 24 hr
2 doses: last dose < 24 hours
2 doses: last dose ≥ 24 hours
Unknown
354 (25.4) 74.6 7.5  
29. Fetal surveillance FS - Admission stripb,c Not checked
Checked
424 (30.4) 69.6 39.2  
30. FS - Auscultationb,c 263 (18.9) 81.1 60.0  
31. FS – Intrapartum electronic fetal monitoring (external)b,c 265 (19.0) 81.0 53.2  
32. FS – Intrapartum electronic fetal monitoring (internal) b,c 125 (9.0) 91.0 45.0  
33. FS – No Monitoring 29 (2.1) 97.9 11.4  
34. FS - Unknown 36 (2.6) 97.4 13.5  
35. If induced - indication for induction None Not checked
Checked
10 (0.7) 99.3 12.5  
36. Diabetes 9 (0.6) 99.4 74.0  
37. Elective 31 (2.2) 97.8 26.8  
38. IUGR/SGA 14 (1.0) 99.0 64.5  
39. LGA 8 (0.6) 99.4 55.3  
40. Maternal obstetrical conditions 32 (2.3) 97.7 14.6  
41. Multiple gestation 4 (0.3) 99.7 66.5  
42. Non-reactive NST 5 (0.4) 99.6 28.4  
43. Oligohydramnios 7 (0.5) 99.5 79.8  
44. Poor biophysical score 5 (0.4) 99.6 28.4  
45. Post dates 64 (4.6) 95.4 73.8  
46. Pre-eclampsia 25 (1.8) 98.2 43.6  
47. Pre-existing maternal medical conditions 6 (0.4) 99.6 24.8  
48.   PROM   42 (3.0) 97.0 52.8  
49. Other maternal 51 (3.7) 96.3 32.1  
50. Other fetal 24 (1.7) 98.3 32.5  
51. Other 16 (1.1) 98.9 24.5  
52. If induced - method of induction None Not checked
Checked
2 (0.1) 99.9 85.0  
53. Amniotomyb 125 (9.0) 91.0 51.2  
54. Cervidil 53 (3.8) 96.2 70.0  
55. Cytotec/ Misoprostol 15 (1.1) 98.9 20.5  
56. Mechanical 10 (0.7) 99.3 63.9  
57. Oxytocin 129 (9.2) 90.8 66.1  
58. Other 26 (1.9) 98.1 18.0  
59. Other - Prostaglandin 31 (2.2) 97.8 38.3  
60. If Caesarian section - indication for Caesarian section None Not checked
Checked
2 (0.1) 99.9 85.0  
61. Breech 21 (1.5) 98.5 82.4  
62. Cord prolapse 1 (0.1) 99.9 80.0  
63. Diabetes 7 (0.5) 99.5 49.0  
64. Failed forceps/vacuum 3 (0.2) 99.8 72.6  
65. Fetal anomaly 0 100 100  
66. IUGR/SGA 5 (0.4) 99.6 54.4  
67. LGA 4 (0.3) 99.7 33.3  
68. Maternal request 26 (1.9) 98.1 17.9  
69. Multiple gestation 12 (0.9) 99.1 64.3  
70. Non-progressive labour/ descent/
dystocia
34 (2.4) 97.6 76.6  
71. Non-reassuring fetal status 31 (2.2) 97.8 72.3  
72. Placenta previa 1 (0.1) 99.9 90.9  
73. Placental abruption 4 (0.3) 99.7 60.0  
74. Preeclampsia 8 (0.6) 99.4 42.6  
75. Prematurity 8 (0.6) 99.4 19.8  
76. Previous Caesarean 22 (1.6) 98.4 89.7  
77. PROM 4 (0.3) 99.7 60.0  
78. Other fetal health problem 14 (1.0) 99.0 50.0  
79. Other maternal health problem 17 (1.2) 98.8 31.4  
80. Forceps
vacuum
Forceps/vacuumb None
Forceps
Vacuum
Forceps and vacuum
Unknown
189 (13.5) 86.5 55.5  
81. Episiotomy Episiotomyb None
Mediolateral
Midline
3rd degree extension
4th degree extension
Unknown
241 (17.3) 82.7 46.9  
82. Laceration Laceration None
1st degree
2nd degree
3rd degree
4th degree
Cervical tear
Other
Unknown
347 (24.9) 75.1 63.0  
83. Maternal pain relief None Not checked
Checked
69 (4.9) 95.1 52.4  
84. Epidural 101 (7.2) 92.8 85.5  
85. General 8 (0.6) 99.4 73.1  
86. Localb 111 (8.0) 92.0 45.8  
87. Narcotics 97 (7.0) 93.0 82.4  
88. Nitrous Oxide 94 (6.7) 93.3 71.9  
89. Non-pharmacologicalb 319 (22.9) 77.1 49.5  
90. Pudendal 1 (0.1) 99.1 92.3  
91. Spinal epidural combination 21 (1.5) 98.5 50.4  
92. Spinal 51 (3.7) 96.3 85.3  
93. Unknown 15 (1.1) 98.9 46.0  
94. Time of birth Time of birth Time of birth (24 hour format)
None
127 (9.1) 90.9 N/Aa N/Aa
95. Delivered by Delivered by Obstetrician
Family physician
Midwife at hospital
Midwife at home
Nurse practitioner
Specified midwife group
Other
Unknown
159 (11.4) 88.6 71.8  
96. Smoking status Smoking b,c No smoking
≤ 20 weeks
> 20 weeks
≤ 20 and > 20 weeks
Unknown
294 (21.1) 78.9 50.7  

 

A previous validation study that explored record linkage of births and infant deaths in Canada examined gestational age and birth weight and indicated good overall agreement.Footnote 21, Footnote 22  Gestational age was also found to have a relatively high degree of agreement between the Discharge Abstract Database (DADD) of the Canadian Institute for Health Information (CIHII) and the Nova Scotia Atlee Perinatal Database (NSAPDD).Footnote 23  This finding is in contrast to our study, where gestational age and birth weight achieved ICC values of between 30% and 40%, indicating poor agreement.

Caesarean delivery was found to be coded accurately in the DAD, and information on first to fourth degree perinatal lacerations and induction of labour was also reasonably accurate in this study.Footnote 23  Results of our audit were consistent with respect to delivery type and lacerations, with substantial or almost perfect agreement (beyond chance) achieved between the re-abstracted data and the information previously entered into the Niday. However, induction method (amniotomy) was less reliable with only 51.2% agreement (beyond chance) noted between the two datasets.

Ensuring completeness and reliability of the data entered into the Niday is a challenge. Data are entered manually via a secure Internet website or uploaded directly into the database from electronic documentation systems. Regional coordinators send reminders to hospital staff to facilitate the process of data entry and to troubleshoot problems when needed. Verification reports are generated quarterly by a data analyst to identify inconsistencies in numbers and types of births and find errors in the data. A training program has been developed so that all users have a thorough understanding of the system. Sustainability of this database depends on achieving broad support at all levels and valuing the system as a key attribute of the patient safety movement. Based on the results of this audit, and through consultation with experts in the field, a number of recommendations have been put forward to improve data quality (Table 3).

Table 3. Recommendations to improve quality of data

  1. Establish a system of ongoing surveillance of data quality in each organization;
  2. Encourage participating hospitals to promptly correct any data entry errors identified through the verification reports;
  3. Identify and communicate corrective action to reduce occurrence of recurring errors;
  4. Reinforce the need to ensure accurate documentation at point of care and to ensure access to information for data entry personnel;
  5. Re-evaluate and monitor use of terms (e.g. none and unknown);
  6. Establish automatic verification checks at the time of data entry (i.e. birth weight, gestational age, maternal data of birth, postal code);
  7. Build in logic checks (i.e. logic checks based on Neonatal Resuscitation Program standards);
  8. Set birth weight limits based on gestational age but allow override capability;
  9. Reassess variable options (i.e. antenatal steroids, episiotomy, lacerations, forceps/vacuum, maternal pain relief, newborn resuscitation, smoking status);
  10. Clarify definitions for the following variables: delivered by; fetal surveillance (intrapartum fetal monitoring internal or external, admission strip, auscultation); method of induction (amniotomy); labour type (induced); and augmentation
  11. Require mandatory completion of essential variables (i.e. those required for reporting), reinforce use of standard data entry worksheets;
  12. Provide ongoing training to ensure that all data entry personnel have had standardized training in data entry; and
  13. Use data dictionaries to ensure that everyone understands the options for each variable.

This audit is in line with the MOHLTC quality assurance initiatives, and it is a logical step to improving data quality and perinatal care practices. The Niday Perinatal Database is a comprehensive, multifaceted system providing data to perinatal care providers, decision makers, educators and researchers in Ontario. Since the audit, the Niday has expanded to capture data for 100% of births in the province. Many upgrades and improvements to the system have already been completed. Further exploration of quality issues is ongoing as part of the initiative to integrate the database with four other perinatal/newborn databases (Fetal Alert Network, Maternal Multiple Marker Screening, Newborn Screening, and the Ontario Midwifery Program (OMPP) Database. Recent Ministry funding and a newly established administrative body (BORN Ontario) have been established to carry these recommendations forward.

Limitations

There are two potential limitations to this audit: completeness and clarity of the patient health record and sampling method. Of the hospitals entering data into the Niday at the time of the audit, 14% were recruited to participate in the re-abstraction process. This sample pool was sufficient to identify a number of issues. Although, the patient charts were selected randomly, the hospitals were selected through purposive sampling; therefore, the results of these analyses may not be generalizable to all hospitals in the province. Data entry personnel for both the original data entry to the Niday database and the re-abstraction process were asked to collect as much information as possible from the patient chart and to be vigilant in entering the data. However, reliability of the data entered into the Niday database is dependent on completeness and clarity of the information documented. Deficits in either regard can influence the reliability of the data entered and influence the results of an audit.

Conclusions

There were 90 defined patient elements within the Niday Perinatal Database at the start of the audit. Approximately one-third of the variables were re-abstracted from the patient record to determine agreement with the data already entered in the Niday Database. Approximately 17% of the data fields audited showed both percent agreement of less than 95% and a kappa or ICC value of less than 60%, indicating only slight, fair, poor or moderate agreement (beyond chance) between the data originally entered into the Niday database and the data re-entered during the audit. This suggests these data fields may be less than reliable and require further investigation to ensure quality.

Acknowledgements

This project is the result of the effort of many individuals and organizations in Ontario. The Niday Quality Audit was conducted under the auspices of the Ontario Perinatal Surveillance System (OPSS). We thank Monica Prince (Prince Computing) who conducted the data analyses and provided input into the final report. We would also like to thank Dr. Ann Sprague for her assistance reviewing the final report, Deshayne Fell for her help in reviewing the manuscript, the auditors for their tireless efforts collecting data at the participating sites across the province and the countless practitioners, data entry personnel and decision makers who provided assistance to make the project possible.

References

  1. Brown A. Update: building on a foundation to sustain quality data. Toronto (ON): Ministry of Health and Long-Term Care; 2007 Jan. 
  2. Last JM. A dictionary of epidemiology. 4th ed. New York (NY): Oxford University Press; 2001.
  3. Health Canada. The Canadian Perinatal Health Report. Ottawa (ON): Minister of Public Works and Government Services Canada; 2000.
  4. Choi BC. Perspectives on epidemiologic surveillance in the 21st century. Chronic Dis Can. 1998;19(4):145-51.
  5. BORN Ontario [Internet]. Ottawa (ON): BORN [updated 2011] Mission and Vision [cited 2011 Apr 26]. Available from: http://www.bornontario.ca/about-born/vision-and-mission
  6. Statistics Canada. Births, estimates, by province and territory [Internet]. Ottawa (ON): Statistics Canada; [modified 2010 Oct 26; cited 2011 Apr 26]. Available from: http://www40.statcan.gc.ca/l01/cst01/demo04a-eng.htm
  7. Freedman G; Health Results Team for Information Management. Building a data quality management framework for Ontario. Toronto (ON): Ministry of Health and Long-term Care; 2006.
  8. Canadian Institutes of Health Research; Natural Sciences and Engineering Research Council of Canada; Social Sciences and Humanities Research Council of Canada. Tri-council policy statement: ethical conduct for research involving humans. Ottawa (ON): 2010.
  9. Iron K, Manuel DG. Quality assessment of administrative data (QuAAD): an opportunity for enhancing Ontario's health data. ICES Investigative Report. Toronto (ON): Institute for Clinical Evaluative Sciences; 2007 Jul.
  10. Hierholzer WJ. Health care data, the epidemiologist's sand: comments on the quantity and quality of data. Am J Med. 1991;91(3B):21S-26S.
  11. Juurlink D, Preyra C, Croxford R, Chong A, Austin P, Tu J, et al. Canadian Institute for Health Information Discharge Abstract Database: a validation study. Toronto (ON): Institute for Clinical Evaluative Sciences; 2006 Jun.
  12. Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. 4th ed. Oxford (UK): Oxford University Press; 2008.
  13. Norman GR, Streiner DL. Biostatistics: The bare essentials. 2nd ed. Hamilton (ON): BC Decker; 2000.
  14. Bartko JJ. Measurement and reliability: statistical thinking considerations. Schizophr Bull. 1991;17(3):483-9.
  15. Landis RJ, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74.
  16. Bedard M, Martin NJ, Krueger P, Brazil K. Assessing reproducibility of data obtained with instruments based on continuous measures. Exp Aging Res. 2000; 26(4):353-65.
  17. Portney LG, Watkins MP. Foundations of clinical research: applications to practice. 2nd ed. New Jersey:Prentice Hall; 2000.
  18. Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85(3):257-68.
  19. Bader MD, Ailshire JA, Morenoff JD, House JS. Measurement of the local food environment: a comparison of existing data sources. Am J Epidemiol. 2010;171(5):609-17.
  20. Ali AH. An evaluation of perinatal surveillance system in Eastern and Southeastern Ontario. Ottawa (ON): University of Ottawa; 2003.
  21. Fair M, Cyr M, Allen AC, Wen SW, Guyon G, MacDonald RC, et al. Validation study for a record linkage of births and infant deaths in Canada. Ottawa (ON): Statistics Canada; 1999 [Statistics Canada, Catalogue No.: 840013XIE].
  22. Fair M, Cyr M, Allen AC, Wen SW, Guyon G, MacDonald RC. An assessment of the validity of a computer system for probabilistic record linkage of birth and infant death records in Canada. The Fetal and Infant Health Study Group. Chronic Dis Can. 2000;21(1):8-13.
  23. Joseph KS, Fahey J. Validation of perinatal data in the Discharge Abstract Database of the Canadian Institute for Health Information. Chronic Dis Can. 2009;9(3):96-100.