ARCHIVED - Canadian Incidence Study of Reported Child Abuse and Neglect 2008

Chapter 2 - Methodology

Nico Trocmé, Barbara Fallon, Bruce MacLaurin, Vandna Sinha, Tara Black, Elizabeth Fast, Caroline Felstiner, Sonia Hélie, Daniel Turcotte, Pamela Weightman, Janet Douglas, and Jill Holroyd

The CIS-2008 is the third national study examining the incidence of reported child abuse and neglect in Canada. It captured information about children and their families as they came into contact with child welfare sites over a three-month sampling period. Children who were not reported to child welfare sites, screened-out reports (referrals that were not opened for an investigation), or new allegations on cases currently open at the time of case selection, were not included. A multi-stage sampling design was used, first to select a representative sample of 112 child welfare sites across Canada, and then to sample cases within these sites. Information was collected at the conclusion of the investigation directly from approximately 1,800 workers. The CIS-2008 sample of 15,980 investigations was used to derive estimates of the annual rates and characteristics of investigated children in Canada.

As with any survey, estimates must be understood within the constraints of the survey instruments, the sampling design, and the estimation procedures used. This chapter presents the CIS-2008 methodology and discusses its strengths and limitations, and their impact on interpreting the CIS-2008 estimates.

Sampling

The CIS-2008 sample was drawn in three stages (Figure 2-1): first, a representative sample of child welfare sites from across Canada was selected, then cases were sampled over a three-month period within the selected sites, and finally, child investigations that met the study criteria were identified from the sampled cases.

Site Selection

The primary sampling unit for the CIS was the local organization responsible for conducting child-maltreatmentrelated investigations. In some jurisdictions, these organizations were autonomous agencies; in others, they were local offices for the provincial or territorial child protection authority (Table 1-1). In the latter case, decisions needed to be made to determine the appropriate sampling unit. In most jurisdictions, organizations served the entire population in a specific geographic area; however, in some instances several organizations served different populations in the same area on the basis of religion, language, or Aboriginal background. While in most jurisdictions a provincial or territorial list of organizations was readily available, a more extensive review process was required to obtain a list of Aboriginal organizations with fullydelegated investigation authority. A final count of 412 organizations constituted the sampling frame for the 2008 study (Table 2-1).

[Click to enlarge]

Organizations were stratified by province or territory, and, in larger provinces, they were further stratified by size of the organization (defined by the number of case openings in a year) and by region. A final count of 412 organizations constituted the sampling frame for the 2008 study. In total, 39 strata provided the sampling structure from which 112 organizations were selected. Twenty-three sites were Aboriginal organizations.

The second sampling stage involved selecting cases opened in the study sites during the three-month period from October 1, 2008 to December 31, 2008.

The final sampling stage involved identifying children aged 15 years and under who had been investigated as a result of concerns related to possible maltreatment.

These procedures yielded a final sample of 15,980 children investigated because of maltreatment-related concerns. For additional information, refer to pages 12-15.

Organizations were stratified by province or territory, and, in larger provinces, they were further stratified by size of the organization (defined by the number of case openings in a year) and by region. In addition, separate strata were developed for Aboriginal organizations. Stratification ensured that all subpopulations were represented in the sample. The number and structure of the strata were set first to ensure representation of each province and territory and then to represent their relative population sizes. Alberta, British Columbia, Ontario, Québec, and Saskatchewan provided additional funds to oversample in their jurisdiction with the aim of producing province-specific estimates. Aboriginal sites were also oversampled in order to better understand investigations in Aboriginal organizations. In total, 39 strata provided the sampling structure from which 112 organizations were selected. Twenty-three sites were Aboriginal organizations.

Most sites were selected randomly within their regional strata using the SPSS Version 15.0 (SPSS Statistics, 2007) random selection application. Exceptions included sites sampled with certainty, sites that could not be feasibly included because of size (fewer than 50 investigations a year) or distance (geographical remoteness), and Aboriginal sites that were selected in consultation with the First Nations CIS-2008 Advisory Committee. Sites in the largest metropolitan areas were sampled with certainty. Sites from Nunavut, the Yukon, and the Northwest Territories were sampled by convenience, on the basis of accessibility, expected case volume and regional representation. In two of the oversampling provinces – Québec and Saskatchewan – all of the non- Aboriginal sites were included, with the exception of regions 17 and 18 in Northern Québec (Hudson Bay, James Bay and Nunavik). Seven organizations declined to be involved because of their particular circumstances; seven replacement sites were randomly selected from the remainder.

Case Selection

The second sampling stage involved selecting cases opened in the study sites during the three-month period from October 1, 2008 to December 31, 2008.⁸ Three months was considered to be the optimum period to ensure high participation rates and good compliance with study procedures. Consultation with service providers indicated that case activity from October to December was considered to be typical of the whole year. However, follow-up studies are needed to systematically explore the extent to which seasonal variation may affect estimates that are based on this three-month sampling period.

In small to mid-sized sites, every case opened during the three-month sampling period was selected. In larger sites that conducted over 1,000 investigations per year, a random sample of 250 cases was selected for inclusion in the study (Trocmé, Fallon et al., 2009). In sites from Québec, a random sample of approximately 50% of investigations was selected.⁹

In most jurisdictions outside of Québec and Alberta, families are the unit of service at the point of the initial decision to open a case. In Québec, the child is the unit of service and cases were selected on that basis. This meant that there were instances where several siblings were investigated, but only one was selected for inclusion in the CIS. Although the unit of service is also the child in Alberta, cases were selected for the CIS on a family basis.

Several caveats must be noted with respect to case selection. To ensure that systematic and comparable procedures were used, the formal process of opening a case for investigation was used as the method for identifying cases. The following procedures were used to ensure consistency in selecting cases for the study:

• situations that were reported but screened out before the case was opened were not included (Figure 1-1). There was too much variation in screening procedures to be able to track these cases feasibly within the budget of the CIS;
• reports on already-open cases were not included. This meant that in jurisdictions that count reports on already-open cases as new openings – as is done in Québec – careful attention had to be given to separating out new cases from already-open ones;
• only the first report was included for cases that were reported more than once during the three-month sampling period; and
• some jurisdictions have been developing differential or alternative response models that could have posed a challenge in capturing cases opened in alternative non-protection stream. However, because in most sites the decisions to stream occurred after the initial investigation, the CIS was generally able to capture both types of openings.

These procedures resulted in the selection of 9,933 cases (1,930 child-based cases from Québec and 8,003 family-based cases from the rest of Canada) (Figure 2-1).

Identifying Investigated Children

The final sampling stage involved identifying children who had been investigated as a result of concerns related to possible maltreatment. As noted above, since in most jurisdictions cases are opened at the level of the family, procedures had to be developed to determine which children in each family had been investigated for maltreatmentrelated reasons. Furthermore, cases can be opened for reasons that do not involve maltreatment concerns. For instance, in Québec, a case could have been opened because a family is requesting support when a child is displaying serious behavioural problems. Similarly, some jurisdictions classify home studies for prospective adoptive or foster homes as case openings.

In jurisdictions outside of Québec, children eligible for inclusion in the final study sample were identified by having child welfare workers complete the Intake Face Sheet from the CIS-2008 Maltreatment Assessment Form. The Intake Face Sheet allows the worker to identify any children who were being investigated because of maltreatmentrelated concerns (i.e., investigation of possible past incidents of maltreatment or assessment of risk of future maltreatment). In Québec, the identification of maltreatment-related investigations was done by including all “retained”¹⁰ cases with maltreatment-related case classification codes.

The age range covered by provincial and territorial child welfare statutes varies from 0–15 to 0–19 years. To ensure consistency in developing national estimates, only children 15 and under are included in the final sample used in this report.

These procedures yielded a final sample of 15,980 children investigated because of maltreatment-related concerns.

Investigating Maltreatment versus Assessing Future Risk of Maltreatment

The primary objective of the CIS is to document investigations of situations where there are concerns that a child may have already been abused or neglected. While investigating maltreatment is central to the mandate of child protection authorities, their mandates can also apply to situations where there is no specific concern about past maltreatment but where the risk of future maltreatment exists. Cases that were being assessed for risk of future maltreatment were not explicitly included in previous cycles of the CIS. To better capture both types of cases, the CIS-2008 was redesigned to track investigations versus cases opened only to assess the risk of future maltreatment. Investigating workers were asked to complete a data collection instrument for both types of cases. For cases involving maltreatment investigations, workers described the specific forms of maltreatment that were investigated and whether the investigation was substantiated. In cases that were opened only to assess future risk of maltreatment, the investigating workers were asked to indicate whether the risk was confirmed, but they were not asked to specify the specific forms of future maltreatment that they may have had concerns about. Identifying the specific form of future maltreatment being assessed was not feasible, given that risk assessments are based on a range of factors including the child’s and the caregivers’ strengths and vulnerabilities, and sources of familial support and familial stress.

While this change provides important additional information about risk of future maltreatment cases, it has complicated comparisons with past cycles of the study. Thus, comparisons with previous cycles in Chapter 3 are limited to comparisons of rates of all maltreatment-related investigations including risk assessments. In contrast, risk of future maltreatment cases are excluded from the 2008 estimates of rates and characteristics of substantiated maltreatment in Chapter 4 and Chapter 5.

Forms of Maltreatment Included in the CIS-2008

The CIS-2008 definition of child maltreatment includes 32 forms of maltreatment grouped into five categories of maltreatment: physical abuse, sexual abuse, neglect, emotional maltreatment, and exposure to intimate partner violence (Appendix E). This classification reflects a fairly broad definition of child maltreatment and includes forms of maltreatment that are not specifically indicated in some provincial and territorial child welfare statutes (e.g., exposure to intimate partner violence).¹¹ The CIS-2008 tracked up to three forms of maltreatment for each investigation.

A source of potential confusion in interpreting child maltreatment statistics is an inconsistency in the categories of maltreatment included. Most child maltreatment statistics refer to physical and sexual abuse, but other categories of maltreatment, such as neglect and emotional maltreatment, are not systematically included. There is even less consensus with respect to subtypes or forms of maltreatment (Portwood, 1999). For instance, some child welfare authorities include only intra-familial sexual abuse, while the justice system deals with cases of extra-familial sexual abuse (see Chapter 4: Primary Categories of Maltreatment for list of specific forms).

Investigated Maltreatment versus Substantiated Maltreatment

Child welfare statutes in most jurisdictions require that professionals working with children and the general public report all situations where they have concerns that a child may have been maltreated or where there is a risk of maltreatment. The investigation phase is designed to determine whether the child was in fact maltreated. Some jurisdictions use a two-tiered substantiation classification system that distinguishes between substantiated and unfounded cases, or verified and not verified cases. The CIS uses a three-tiered classification system for investigated incidents of maltreatment, in which a “suspected” level provides an important clinical distinction in cases where there is not sufficient evidence to substantiate maltreatment, but where maltreatment cannot be ruled out (Trocmé, Knoke, Fallon, & MacLaurin, 2009).

In reporting and interpreting maltreatment statistics, it is important to clearly distinguish between risk of future maltreatment investigations, maltreatment investigations, and substantiated cases of maltreatment. Estimates presented in Chapter 3 of this report include maltreatment-related investigations (specific allegations and risk of future maltreatment) and the estimates in Chapter 4 and Chapter 5 focus on cases of substantiated maltreatment.

Risk of Harm versus Harm

Cases of maltreatment that draw public attention usually involve children who have been severely injured or, in the most tragic cases, have died as a result of maltreatment. In practice, child welfare workers investigate and intervene in many situations in which children have not yet been harmed, but are at risk of harm. For instance, a toddler who has been repeatedly left unsupervised in a potentially dangerous setting may be considered to have been neglected, even if the child has not yet been harmed.

Provincial and territorial statutes cover children who have suffered demonstrable harm due to abuse or neglect and children at risk of harm. Substantiation standards in all jurisdictions across Canada include situations where children have been harmed as a result of maltreatment as well as situations where there is no evidence of harm but where children are at substantial risk of harm as a result of maltreatment. The CIS-2008 included both types of situations in its definition of substantiated maltreatment. The study also gathered information about physical and emotional harm attributed to substantiated maltreatment (Chapter 4).

The CIS-2008 documented both physical and emotional harm; however, definitions of maltreatment used for the study did not require harm. This is similar to the fourth United States (U.S.) National Incidence Study of Child Abuse and Neglect (NIS-4), which included two standards in calculating estimates of maltreatment: a narrow standard based on evidence of harm to the child, and a broader endangerment standard that includes cases of children at risk of harm (Sedlak et al., 2010).

There can be confusion around the difference between risk of harm and risk of maltreatment. A child who has been placed at risk of harm has experienced an event that endangered her/his physical or emotional health. Placing a child at risk of harm is considered maltreatment. For example, neglect can be substantiated for an unsupervised toddler, regardless of whether or not harm occurs, because the parent is placing the child at substantial risk of harm. In contrast, risk of future maltreatment refers to situations where a specific incident of maltreatment has not yet occurred, but circumstances, for instance parental substance abuse, indicate that there is a significant risk that maltreatment could occur.

Instrument

The CIS-2008 survey instruments were designed to capture standardized information from child welfare workers conducting maltreatment investigations or investigations of risk of future maltreatment. Because investigation procedures vary considerably across Canada (Chapter 1), a key challenge in designing the CIS-2008 survey instruments was to identify elements common across jurisdictions that could provide data in a standardized manner. Given the time constraints faced by child welfare workers, the instrument also had to be kept as short and simple as possible.

CIS-2008 Maltreatment Assessment Form

The main data collection instrument used for the study was the CIS-2008 Maltreatment Assessment Form which was completed by the primary investigating child welfare worker upon completion of each child welfare investigation (Appendix F). The data collection form consisted of an Intake Face Sheet, a Household Information Sheet, and two identical Child Information Sheets.

Intake Face Sheet

Workers completed the Intake Face Sheet for all cases opened during the study period where a specific allegation of maltreatment had been made or where there was a concern about future risk of maltreatment. This initial review of all child welfare case openings provided a consistent mechanism for differentiating between cases investigated for suspected maltreatment or risk of maltreatment and those referred for other types of child welfare services (e.g., preventive services).

Information about the report or referral as well as identifying information about the child(ren) involved was collected on the Intake Face Sheet. The form requested information on: the date of referral; referral source; number of children in the home; age and sex of children; the reason for the referral; whether the case was screened out; the relationship between the caregiver and each child; and the type of investigation (maltreatment or risk of future maltreatment).12 The section of the form containing any identifying information was kept at the site. The remainder of the form was completed if abuse or neglect was suspected at any point during the investigation, or if the worker completed a risk investigation only.13

Household Information Sheet

The Household Information Sheet was completed when at least one child in the family was investigated for alleged maltreatment or risk of maltreatment. The household was defined as all adults and children living at the address of the investigation. The Household Information Sheet collected detailed information on up to two caregivers living in the home at the time of referral. Descriptive information was requested about workers’ assessment of the level of cooperation by the caregiver with the investigation, other adults in the home, type of housing, housing safety, caregiver functioning, case status (i.e., whether the case was closed), and referral(s) to other services (Appendix F).

Child Information Sheet

The third page of the instrument, the Child Information Sheet, was completed for each child who was investigated for maltreatment or for whom there was a risk assessment completed.¹⁴ The Child Information Sheet documented up to three different forms of maltreatment, and included levels of substantiation, alleged perpetrator(s), and duration of maltreatment. In addition, it collected information on child functioning, physical and emotional harm to the child attributable to the alleged maltreatment, child welfare court activity, out-of-home placement, and transfers to ongoing services. Workers who conducted investigations of risk of future maltreatment did not answer questions pertaining to investigated maltreatment but did complete items about child functioning, placement, court involvement, previous reports, and spanking. In those investigations involving risk assessments, workers were asked whether they were concerned about future maltreatment.

Québec Child Assessment Form

The CIS-2008 Maltreatment Assessment Form was adapted to appear as an electronic pop-up form integrated into the client information system in Québec (Projet Intégration Jeunesse [PIJ]). The form appeared as a series of nine tabbed windows following the basic structure of the form. However, unlike the CIS-2008 Maltreatment Assessment Form which was designed around a family-based case opening system (one form per investigated family), the Québec electronic form was designed to reflect their system: one form per investigated child. Nearly one quarter of the data fields in the Québec form were automatically completed by the client information system. Due to differences in the structure of child welfare services in Québec and to constraints inherent in the use of an electronic client information system, it was not possible to match all the items on the Québec data form with the items on the CIS-2008 Maltreatment Assessment Form. As a result, data from Québec was excluded from Table 3-4a and Table 3-7 in this report.

CIS-2008 Guidebook

A significant challenge for the study was to overcome variations in the definitions of maltreatment used in different jurisdictions. Rather than using specific legal or administrative definitions, a single set corresponding to standard research classification schemes was used (Appendix E). All items on the case selection forms were defined in an accompanying CIS-2008 Guidebook (Appendix G).

Revising & Validating the CIS-2008 Maltreatment Assessment Form

The CIS-2008 data collection instrument was based on the CIS-2003 (Trocmé, Fallon et al., 2005), CIS-1998 (Trocmé et al., 2001), and Ontario Incidence Study of Reported Child Abuse and Neglect 1993 (OIS-1993) (Trocmé et al., 1994) data collection instruments in order to maximize comparisons across cycles of the study. A key challenge in updating instruments across cycles was to find the right balance between maintaining comparability while making improvements based on the findings from previous cycles. For instance, very low response rates on income questions in previous studies lead to the development of a simpler question about families running out of money. In addition, changes over time in child welfare practices required changes to data collection forms. At the time of the OIS-1993 study, for example, exposure to intimate partner violence was generally not considered to be a type of maltreatment and was not a specific maltreatment category on the CIS-1998 Maltreatment Assessment Form.

Changes to the CIS-2008 version of the forms were made in close consultation with the Research Working Group, a subcommittee of the National CIS-2008 Steering Committee. Changes were based on data collection problems noted during the CIS-2003, an analysis of response rates (Tonmyr, 2004), a validation study, focus groups with child welfare workers in several jurisdictions, and a reliability study which compared different versions of the form.

Changes to the data collection instrument included: the addition of a series of questions designed to distinguish maltreatment investigations from risk of future maltreatment cases, a more detailed procedure to identify the relationship between each child and the caregivers in the home, a more elaborate housing safety question, a new measure of poverty, more specific intimate partner violence maltreatment codes, and revised emotional maltreatment categories. The final version of the data collection instrument is in Appendix F.

Case File Validation Study

Review of the data collection instrument for the 2008 cycle of the study started with a case file validation study (Trocmé, Fallon et al., 2009). Data collected in 2003 using the CIS-2003 version of the form was compared with information in the case files from one of the larger CIS-2003 sites. While there was good correspondence on many items, it became apparent that despite specific instruction in 2003 to include only investigations of child maltreatment, a number of cases that appeared to involve only concerns about future risk had been coded as maltreatment investigations.

Validation Focus Groups

The CIS-2008 Research Team conducted six focus groups with front-line child protection workers and supervisors across Canada from late July to late October 2007 (Trocmé, Fallon et al., 2009). The purpose of the groups was to give feedback on the proposed changes to the CIS-2008 data collection instrument. The process was iterative. Feedback from each focus group was used to make changes to the instrument prior to the next focus group. Groups were held in Montréal, Toronto, St. John’s, Halifax, Regina, and Calgary. One of the participating groups was an Aboriginal site.

Reliability Study

A reliability study (Trocmé, Fallon et al., 2009) examined the test re-test reliability of the data collection instrument. The consistency of worker judgments was evaluated by comparing case ratings on the instrument at two points in time. Test re-test reliability was examined for a wide range of variables, such as characteristics of the alleged maltreatment, the household, caregivers, children, maltreatment history, and service-related variables. A convenience sample of eight child welfare sites was selected based on availability and proximity to study team research personnel. Workers participated in the study on a voluntary basis.

The test re-test procedure was arranged as follows: workers completed the instrument for new investigations that had an allegation or suspicion of child maltreatment (Time 1), then an average of 3.8 weeks later, the same worker completed the instrument a second time for the same investigation (Time 2). At Time 1 the sample size was 130 investigations. Time 2 for some sites could not be scheduled prior to the finalization of the instrument and therefore their data were not included in the analysis.

All sites were collapsed, yielding a sample of 100 children from 68 households. Two measures of agreement were calculated for categorical variables: percent agreement and the Kappa statistic. The Kappa statistic adjusts for agreement that occurs by chance alone; values between 0.4 and 0.6 are usually interpreted as moderate agreement; between 0.6 and 0.8 good agreement; and values that exceed 0.8 reflect excellent agreement (Landis & Koch, 1977). Similar testing was conducted in CIS-2003 (Knoke, Trocmé, MacLaurin, & Fallon, 2009).

The vast majority of items on the CIS-2008 form showed good to excellent test re-test reliability. Among the most reliable groups of variables were primary forms of maltreatment, family’s maltreatment history, child age and gender, case disposition items, and indices related to emotional harm. ‘Any service referral’ and ‘any family-focused referral’, and the majority of items related to household and caregiver characteristics also showed good to excellent agreement.

A number of items fell slightly below the criterion for acceptable reliability. In order to address the low reliability of two questions (i.e., accessible drugs/drug paraphernalia and police involvement in the child maltreatment investigation), questions were re-ordered and/or clarified on the final CIS-2008 data collection instrument. The low reliability for secondary and tertiary maltreatment codes was similar to that found for the CIS-2003 data collection instrument. Analysis of secondary and tertiary maltreatment should be interpreted with caution. However, co-occurring maltreatment has been a significant predictor of service provision in multiple secondary analyses of the CIS data (e.g., Black, Trocmé, Fallon, & MacLaurin, 2008).

The study team’s review of the brief written description of the investigation provided by the worker in the reliability study revealed that the newly developed procedures to categorize risk cases were creating confusion and inconsistent results. This led to an unplanned set of revisions to the way that risk was operationalized on the data collection instrument. Time constraints prevented final reliability testing of the CIS-2008 Maltreatment Assessment Form. Although the final data collection instrument differed from the versions that had been tested, the final changes were limited to only a few items.

Data Collection and Verification Procedures

Training

Site Researchers coordinated training and case selection at each CIS-2008 site (Appendix A). The case selection phase began with a training session, conducted by a Site Researcher, to introduce participating child welfare workers to the CIS-2008 instruments and case selection procedures. After a review of the forms and procedures, workers completed the form for a selected case vignette (Appendix J). The completed forms were then discussed and discrepancies in responses reviewed to ensure that items were being properly interpreted. Each worker was given a CIS-2008 Guidebook, which included definitions for all items and study procedures (Appendices G and I).

Timing of Form Completion

The data collection instrument was completed at the point when workers finished their written report of the investigation. The length of time between the receipt of the referral and the completion of the written assessment differed according to provincial, regional, and site practices, although in most instances some type of report was required within six weeks of the beginning of an investigation. In instances where a complex investigation took more time, workers were asked to complete the data collection instrument with their preliminary assessment report.

Site Visits

Site Researchers visited the CIS-2008 sites on a regular basis to collect forms, respond to questions, and monitor study progress. In most instances, six visits to each location were required. Additional support was provided depending on the individual needs of workers at each site. Site Researchers collected the completed forms during each site visit and reviewed them for completeness and consistency. Every effort was made to contact workers if there was incomplete information on key variables (e.g., child age or category of maltreatment) or inconsistencies. Identifying information was stored on-site, and non-identifying information was sent to the central data verification locations.

Data collection was organized in Québec to accommodate established approaches to conducting site-based research, as well as to take into account the particularities of using an electronic data collection form. Instead of using Site Researchers, each participating youth centre identified a liaison person who facilitated and monitored data collection within their own jurisdiction. Three CIS-2008 Research Coordinators worked with the liaisons to provide support and to maintain consistent data collection and verification procedures.

Data Verification and Data Entry

Data collection forms were verified twice for completeness and inconsistent responses: first on-site by the Site Researchers or liaison personnel, and a second time at the University of Toronto, McGill University or University of Calgary locations. Consistency in form completion was examined by comparing the data collection instrument to the brief case narratives provided by the workers.

Data collection forms sent to the CIS-2008 offices in Toronto and Montréal were entered by scanner using TELEform Elite scanning software, V.8.1. Intake Face Sheet information was entered manually using Microsoft Access 2000. The data were then combined into an SPSS Version 17.0 (SPSS Statistics, 2008) data file. Inconsistent responses, missing responses, and miscodes were systematically identified. Checks for duplicate cases were made at the child welfare site and duplicates deleted on the basis of site identification numbers, family initials, and date of referral.

The Québec data were gathered in electronic format from each site. Microsoft Excel 2003 based data collection forms were programmed to extract data from the client information system for a quarter of the items; the remaining three quarters were completed by the worker. Item completion was tracked to ensure that forms could not be finalized until all items had been adequately addressed. The liaison workers verified each form for completeness and checked for inconsistent responses. Excel files were then downloaded to a flash drive and sent to the Research Coordinators who completed a second verification. The files were then uploaded to an SPSS data file.

Item Completion Rates and Participation

The case selection form was kept as short and simple as possible to minimize response burden and ensure a high completion rate. Completion rates were over 98% on most items.¹⁵

The participation rate was estimated by comparing the number of cases for which data collection instruments were completed to the actual number of cases opened during the case selection period.¹⁶ The overall participation rate was 96%, ranging from a low of 30%¹⁷ to a high of 100%. Participation rates below 95% were discussed with the CIS-2008 liaisons for each site to examine the possibility of skewed sampling. In all cases, low participation could be attributed to events such as staff holidays and staff turnover and no evidence of systematic bias was found.

Estimation Procedures

Weighting

The data collected for the CIS-2008 were weighted in order to derive national annual incidence estimates. Two sets of weights were applied. First, results were annualized to estimate the volume of cases investigated by each site in 2008. The annualization weights were derived by dividing the total number of cases opened by site in 2008 by the number of cases sampled from that site. For example, if 225 cases were sampled over 3 months in a site that opened 1,000 cases over the year, a weight of 4.44 (1,000/225) was applied to all cases in the site. The average annualization weight was 7.41.¹⁸ While this annualization method provides an accurate estimate of overall volume, it cannot account for qualitative differences in the types of cases referred at different times of the year.

To account for the non-proportional sampling design, regional weights were applied to reflect the relative sizes of the selected sites. Each study site was assigned a weight reflecting the proportion of the child population of the site relative to the child population in the stratum or region that the site represented. For instance, if a site with a child population of 25,000 were randomly sampled to represent a region or province/territory with a child population of 500,000, a regionalization weight of 20 (500,000/25,000) would be applied to cases sampled from that site. Child population counts for sites and strata were obtained using Statistics Canada Census 2006 subdivision-level¹⁹ data (Statistics Canada, 2007). Regionalization and annualization weights were combined so that each case was multiplied by an annualization weight and by a regionalization weight. National incidence estimates were calculated by dividing the weighted estimates by the child population (aged newborn to 15 years).

In some instances, Aboriginal communities had declined participation in the 2006 Census. Therefore, child population estimates had to be determined through other means. Whenever possible, child population counts for these communities were obtained directly from Aboriginal sites. In one site, child population estimates were based on 2001 Census data. For some Aboriginal sites, like those in Manitoba, which serve the entire child population of a geographic area – including First Nations, other Aboriginal and non- Aboriginal – the Aboriginal weight created reflected the proportion of the Aboriginal child population served by the site relative to the child population in the stratum or region that the site represents.

Case Duplication

Cases reported more than once during the three-month case sampling period were unduplicated (see Case Selection section in this Chapter), however, the weights used for CIS-2008 annual estimates include an unknown number of “duplicate” cases, i.e. children or families reported and opened for investigation two or more times during the year. Although each investigation represents a new incident of maltreatment, confusion arises if these investigations are taken to represent an unduplicated count of children. To avoid such confusion, the CIS-2008 uses the term “child investigations” rather than “investigated children”.

An estimate of how often maltreated children will be counted more than once can be derived from those jurisdictions that maintain separate investigation-based and child-based counts. The U.S. National Child Abuse and Neglect Data System (NCANDS) (U.S. Department of Health and Human Services, 2005), reports that for substantiated cases of child maltreatment, the six-month recurrence rate during 2003 was 8.4%. Further estimates of recurrence have been made in the U.S.: in a 24-month follow up of all investigations from eight states, 16% of children were re-reported within 12 months, and another 6% were re-reported in the subsequent 12 months (Fluke, Shusterman, Hollinshead, & Yuan, 2008). In Québec, the recurrence rate was 8.8% of screened-in investigations over a 12-month period (Hélie, 2005).

Sampling Error Estimation

Although the CIS-2008 estimates are based on a relatively large sample of 15,980 child maltreatment investigations, sampling error is primarily driven by the variability among the 112 sites. Sampling error estimates were calculated to reflect the fact that the survey population had been stratified and that primary sampling units (sites) had been selected randomly from each stratum. As the stratified random sample’s variability among strata is zero, the variance at the national level is then calculated as the sum of the variance in each stratum. In most instances, two sites, the primary sampling units, were chosen from each stratum. Variance estimates were calculated using WesVar 5.1 (Westat, 2008), which computes estimates and their variance estimates from survey data using replication methods.

Standard error estimates were calculated for select variables at the p < 0.05 level.20 Most coefficients of variation were in a reliable range:21 between 5.06% (social assistance estimate for household source of income) and 16.32% (shelter/hotel estimate for housing type). Estimates that should be interpreted with caution ranged from 16.86% (child as the subject of referral) to 33.18% (substantiated sexual abuse and neglect as multiple categories of maltreatment). Estimates that were based on fewer than 100 investigations were not reported. There were a few estimates based on over 100 investigations with coefficients of variation greater than 33.30%: substantiated physical abuse, emotional maltreatment and exposure to intimate partner violence as multiple categories of maltreatment; substantiated sexual abuse, neglect and emotional maltreatment as multiple categories of maltreatment; and chemicals or solvents used in production of drugs in the home.

The error estimates do not account for any errors in determining the annual and regional weights, nor do they account for any other non-sampling errors that may occur, such as inconsistencies or inadequacies in administrative procedures from site to site. The error estimates also cannot account for any variations due to seasonal effects. The accuracy of these annual estimates depends, in part, on the extent to which the sampling period was representative of the whole year (Appendix K).

Ethics Procedures

The CIS-2008 data collection and data-handling protocols and procedures were reviewed and approved by McGill University, the University of Toronto, and the University of Calgary Ethics Committees. Written permission for participating in the data collection process was obtained from the Provincial/ Territorial Directors of Child Welfare as well as from each site administrator or director. Where a participating site had an ethics review process, the study was also evaluated by that site.

The study utilized a case file review methodology. The case files are the property of the delegated site or regional authority. Therefore, the permission of the site was required in order to access case files. Confidentiality of case information and participants, including workers and sites, was maintained throughout the process. No directlyidentifying information was collected on the data collection instrument. The Intake Face Sheet collected nearidentifying information about the children including their first names and ages. The tear-off portion of the Intake Face Sheet had a space for the file/case number the site assigns and the study number the CIS-2008 Site Researchers assigned and also provided space for the first two letters of the family surname. This information was used for only verification purposes. Any names on the forms were deleted prior to leaving the site.

The data collection instruments (that contained no directly-identifying information) were either scanned into an electronic database at the Universities of Toronto or McGill, or uploaded from encrypted CDs or flash drives. At both locations this electronic data was stored on a locked, password-protected hard drive in a locked office and on a CD stored in a locked cabinet off-site. Only those University of Toronto and McGill University research personnel with security clearance from the Government of Canada had access to this information through password-protected files. All paper data collection instruments were archived in secure filing cabinets.

This report contains only national estimates of child abuse and neglect and does not identify any participating site.

Aboriginal Ethics

The First Nations component of the CIS-2008 adhered to the principles of ownership, control, access and possession (OCAP), which must be negotiated within the context of individual research projects. Adherence to OCAP principles was one of three shared concerns which shaped the collaborative relationship between the advisory committee and the research team, and which guided the approach to research design and implementation. The First Nations CIS-2008 Advisory Committee, which mediates Aboriginal ownership of and control over the project, had a mandate of ensuring that the CIS-2008 respects OCAP principles to the greatest degree possible, given that the CIS is a cyclical study which collects data on First Nations, other Aboriginal, and non-Aboriginal investigations. The First Nations CIS-2008 Advisory Committee is responsible for guiding and approving analyses of First Nations data, including potential comparisons with non-Aboriginal sites.

Study Limitations

Although every effort was made to make the CIS-2008 estimates precise and reliable, several limits inherent in the nature of the data collected must be taken into consideration:

• as a result of changes in the way risk of future maltreatment cases are identified in the CIS-2008, comparisons between study cycles must be made with caution. Tables in the CIS-2008 report cannot be directly compared to tables in the two previous reports. Chapter 3 presents selected comparisons across study cycles; please interpret this chapter with caution;
• the weights used to derive annual estimates include counts of children investigated more than once during the year, therefore the unit of analysis for the weighted estimates is a child investigation;
• the CIS tracks information during the first 6 weeks of case activity, however there were slight provincial and territorial differences in this length of time; service outcomes such as out-of-home placements and applications to court included only events that occurred during those first approximately 4-6 weeks; Table 3-5 and Table 3-6 are affected by this limitation;
• as a result of differences in data collection procedures, data from Québec could not be included in Table 3-4a, which displays sources of referral in investigations across CIS cycles, and in Table 3-7, which displays the previous case openings for children in the three study cycles. Québec estimates for Tables 3-5 and 3-6 are derived from an updated version of the Étude d’incidence québécoise (EIQ) [Québec Incidence Study] 2003 database. Subsequent to the publication of the CIS-2003 report, the EIQ research team was able to retrieve previously unavailable information from the PIJ information system, including information on ongoing services and placement. Because estimates were derived from aggregate figures from the EIQ-2003 technical report (Tables 12b and 14b), tests of significance could not be completed;
• the annual national counts presented in this report are weighted estimates. In some instances, sample sizes were too small to derive publishable estimates. For example, Table 4-4 presents the nature of physical harm by primary maltreatment category; the number of substantiated physical abuse investigations involving broken bones or fatality could not be reported due to small sample sizes;
• the CIS tracks only reports investigated by child welfare sites and does not include reports that were screened out, cases that were investigated only by the police and cases that were never reported. For instance, Table 4-1 presents the estimated number of substantiated incidents of exposure to intimate partner violence in Canada. This number does not include incidents of intimate partner violence that were investigated only by the police, and it does not include incidents of intimate partner violence that were never reported to child welfare authorities; and
• the study is based on assessments provided by the investigating child welfare workers and could not be independently verified. For example, Table 5-2 presents the child functioning concerns reported in cases of substantiated maltreatment. The investigating workers determined if the child subject of the investigation demonstrated functioning concerns, for instance depression or anxiety. However, these child functioning concerns were not verified by an independent source.