Pan-canadian public health network council report and policy recommendations on the use of antivirals for prophylaxis during an influenza pandemic

[Previous] [Table of Contents] [Next]

Annex 3.7–Report on Citizen and Stakeholder Deliberative Dialogues on the Use of Antivirals for Prophylaxis

Submitted by Communications:

Intended for:
Task Group on Antivirals for Prophylaxis

---

Contents

Executive Summary

Context

Part I: Process

• The Deliberative Dialogue Methodology
• The Elements of the Dialogue Process
• Dialogue Objectives
• TGAP Dialogue Sessions
• Regional Citizen Dialogues & Quebec Dialogue
• First Nations on Reserve Session
• National Stakeholder Sessions & Target Group Recipient Sessions
• Pedagogical Tools and Materials
• Evaluation Framework

Part II: Dialogue Results – What We Heard

Part III: Process Observations, Evaluation and Next Steps

• Were the overall objectives established for the Dialogue sessions met?
• Did Deliberative Dialogue prove to be an appropriate methodology for assessing the views of participants in relation to the issues in question?
• Was the process developed under TGAP’s direction an effective one?
• Is additional evaluation required and, if so, what should be the priorities?

Part IV: Annexes (Forthcoming)

• Annex A: Schedule of Dialogues
• Annex B: Participant Workbook
• Annex C: Core Presentation
• Annex D: TGAP Issues Framing Report
• Annex E: Participant Surveys/Questionnaires
• Annex F: Evaluation Framework
• Annex G: EKOS Final Report

Executive Summary

The Task Group on Antivirals for Prophylaxis (TGAP) opted to use a “deliberative dialogue” technique to assess the views of citizens (including First Nations on Reserve) and stakeholders on whether governments should stockpile antivirals for prophylaxis and, if so, which groups should have priority access to the stockpile.

Deliberative Dialogue offers policy-makers a deeper insight into the values and “common ground” of participants in relation to a particular set of issues and in relation to alternative courses of action through dialogue and interaction. Deliberative Dialogue outputs can provide a sound reference framework for assessing complex, ethically fraught policy choices.

Eleven separate dialogues were convened between November 2006 and February 2007, including: 6 regional citizen sessions (including 1 in Quebec); 1 First-Nations on Reserve Session; 2 National Stakeholder Sessions; and, 2 Target Group Sessions. The Citizen Sessions were co-hosted by the provinces/territories.

Following a detailed education session, participants were asked to reflect on the pros and cons of three different approaches:

Approach 1: Minimize Serious Illness and Death (“Minimize serious illness and death by providing antivirals for prevention to take care of the most vulnerable”)

Approach 2: Keep Society Functioning (“Maintain basic health, social and economic functions by providing antivirals for prevention to essential workers”)

Approach 3: Minimize Governments’ Role (“Trust institutions and individuals to make their own decisions about whether or not to stockpile antivirals for prevention”)

Participants’ views on these approaches and on priority group rankings were assessed before and after the Dialogues to enable assessment of whether participants’ views changed over the course of the process. Participants were also specifically asked whether governments should in fact proceed with stockpiling antivirals for prophylaxis. A comprehensive evaluation framework was developed and applied to the process.

Salient findings include:

• While there are some discernable differences of perspective across the various Citizen, First Nations, National Stakeholder and Target Group sessions, from a public policy perspective, these differences are not significant enough to require any variation in either the decision confronting policy makers or in future information or outreach.

• Of the three approaches presented, participants rated Keep Society Functioning highest. Minimize Serious Illness and Death also enjoyed considerable (though less) support while Minimize Governments’ Role rated lowest. In the post-dialogue questionnaire, overall support for the first two rated approaches rose, but decreased for the third.

• There is an overall and unambiguous consensus that governments should create an additional national stockpile of antivirals to be used for prophylaxis for specific and limited use.
• Notwithstanding this general consensus, support for creating this additional stockpile of antivirals for prophylaxis appears nuanced. In the Dialogues, participants expressed significant concerns about the “unknowns” including: safety, effectiveness and equity considerations; and opportunity costs. They also expressed a range of views in regard to what role and responsibility individuals and institutions should have in deciding whether to use antivirals for protection. As such, there was also a general consensus that their use be limited to as few people as necessary to obtain results.
• In terms of the “core values” participants appear to want expressed in the decisions taken in relation to this issue, pragmatism rated highest. Consistent with this, participants felt it important to focus scarce resources on health and emergency workers- not because they were at greater risk (although for some participants this was a factor) but because they played an important role in keeping society functioning and containing the spread.
• Participants also stressed the need for strong government leadership and for knowledge to be widely shared and transparent in order to avoid public outrage and potential anarchy. (i.e., Governments have a “duty” to be proactive in keeping the public informed about an outbreak, what the plans are for dealing with it, and for reassuring them that these plans have been objectively developed based on the best available information and will be applied fairly.)
• While compassion was a strong value for participants and there was significant discussion over the course of the Dialogues on issues related to equity and fairness, for most participants, “pragmatism” nonetheless took precedence.
• Many participants concluded society had an “obligation” to take every reasonable step to protect the health and safety of those individuals (and to a lesser extent their immediate families) likely to find themselves on the front-lines of a pandemic response. Hence, they viewed the provision of antivirals for prevention for these health care and emergency workers – even if they were not entirely convinced they would ultimately prove effective – as reasonable, fair and necessary. (There was also support for the notion of reciprocity: the responsibility of those receiving the medications to actually show up for work and to complete the course of treatment.)
• In terms of priority rankings, citizens rated front-line health care workers with close patient contact the highest (and those without close patient contact second highest). Support for emergency workers rated lower and was essentially in the same general range as that for vulnerable groups. (Within the “vulnerable” category, children rated the highest.)
• For target group members and stakeholders, the results were less ambiguous, with health care workers with close patient contact rated the highest, followed closely by other health care workers (regardless of patient contact). Emergency and essential workers ranked second highest, while vulnerable groups ranked last.
• There was also support for the notion that, once the needs of health care and emergency workers had been addressed, society had a duty to “do right” for the most vulnerable segments of society – with children being given precedence over other potential “vulnerable” categories.
• Participants strongly emphasized the need for early and sustained public education in relation to pandemic preparedness generally, but also in the event of a positive decision on stockpiling so that citizens understand there is an evidence-informed policy in place with clear rules that will be consistently applied.
• Participants also strongly recommended additional research to address current knowledge gaps which they tended to characterize as “significant”. Accordingly, it would not be unreasonable to conclude that availability of new information in regard to risks/benefits might yield different results.
• At the First Nations on Reserve session in Edmonton, some participants spoke of the need for decision-makers to recognize the potential contribution of traditional medicine to pandemic planning and response efforts.

Finally, a number of suggestions are proposed for further evaluation of the process by participants, by TGAP and other decision-makers and by those responsible for the delivery of the Dialogues themselves. Certain of these evaluation proposals recognize the need for participants whose views are being assessed to be apprised of how the Dialogue results were taken into account in formulating recommendations or policy decisions on the broader issues associated with stockpiling publicly-funded antivirals for prophylaxis.

Context

Through the Public Health Network Council (PHN Council), the Government of Canada and provincial and territorial jurisdictions have continued to refine and update the landmark February 2004 Canadian Pandemic Influenza Plan for the Health Sector (the Plan).

The use of antivirals for treatment is an integral part of the Plan and accordingly, some 55 millions doses of antivirals are being stockpiled across the country. More recently, the PHN Council has focused its attention on formulating a national policy recommendation to Canada’s Health Ministers on the prophylactic use of antivirals in a pandemic. To that end, the PHN Council established a Task Group on Antivirals for Prophylaxis (TGAP) to provide advice on whether antivirals drugs should be used to prevent illness during an influenza outbreak and if so, who should have priority for receiving them (priority recipients).

TGAP subsequently developed a comprehensive work plan for ensuring the advice it provides the PHN Council is informed by seven inputs: scientific, legal, and ethical reviews, a federal/provincial/territorial logistics review, First Nations and Inuit considerations, and a review of international best-practices and the policies of international bodies. TGAP’s work plan also recognizes the critical importance to policy-makers of hearing from Canadian citizens, First Nations, national stakeholders, and potential priority recipients on these issues as one of the seven inputs.

For the sake of brevity, this report is divided into four parts as follows:

Part I describes the process used by TGAP to engage citizens, First Nations, national stakeholders and potential priority recipients on the broad issues associated with the potential use of antivirals for prevention in a pandemic.

Part II summarizes the results of the process, highlighting areas of common ground and divergence within and across each of the 11 separate sessions the process entailed. This section also incorporates the results of the February 16-17 2007, citizen dialogue session in Montreal convened by the Government of Quebec using an identical methodology and similar pedagogical materials as were used in citizen sessions elsewhere in Canada. This section summarizes some of the highlights of the larger report on the Dialogue Process prepared by EKOS Research and expands on the significance of certain findings.

Part III provides observations on the process, evaluation and next steps.

Part IV comprises seven annexes.

Part I: Process

1.The Deliberative Dialogue Methodology

After considering the merits of various alternative engagement methodologies including focus groups, town hall meetings, quantitative public opinion research and online surveys, TGAP opted to use a “deliberative dialogue” technique as its instrument of choice.

The deliberative dialogue process differs from other research designs in that it offers an opportunity to gain a deeper insight into the values and “common ground” participants involved in the process share in relation to a particular issue or set of issues. While traditional public opinion research and focus group methodologies provide for “top-of-mind” responses, deliberative dialogue has a different emphasis. It is premised on first providing a broad and ideally representative cross-section of participants with the opportunity to gain a deeper understanding of a particular set of issues, alternate courses of action to address them, and the consequences flowing from these actions. It then brings individual participants together to engage in a broader group discussion of the issue and of the trade-offs associated with these courses of action.

There are no right or wrong answers; the process does not require participants to choose from among competing options or approaches. Rather, it is designed to allow participants to identify what elements they like or dislike about them, and if necessary, to propose alternative ones more compatible with their values and priorities.

In the context of the dialogues on antivirals for prevention, the common views, values and priorities of participants that have emerged from the process and that are reviewed in Part II provide a reference framework governments can use to develop a national policy recommendation on the issue.

2.The Elements of the Dialogue Process

The essential components of each of the 11 separate dialogue sessions (Annex A) that were convened were virtually identical- save for where noted below. For example, all sessions were 1.5 days in duration and had the same format and time allocation; participants received the same information package in advance of the session, and a common workbook at registration; surveys and questionnaires were also largely identical. In terms of the mechanics of the process:

• Participants for each citizen dialogue session were randomly recruited using standard recruitment methodologies. Prior to being selected, participants were asked to respond to a series of demographic and attitudinal questions (these are detailed in the EKOS Final Report on the process). Another 100 or so randomly selected residents from the region where participants were drawn were also surveyed on the same issues. This was done to establish that the top-of-mind views of those who participated in the dialogues were essentially similar to those of the broader community from which they were drawn.

• Participants in the Citizen, First Nations on Reserve, and Target Group Dialogues each received a $175 incentive for participating in the session and had eligible transportation and accommodation costs paid. National Stakeholder representatives did not receive this incentive but did have their travel expenses covered.

• Every participant in the Deliberative Dialogue process received a short information package in advance of the meeting comprising:

• a letter co-signed by the Chief Public Health Officer for Canada and the Chief Medical Officer of Health for the jurisdiction in which the session was taking place, thanking them for agreeing to participate and outlining the process and expectations (the letter was reviewed and approved by TGAP);

• the first half of a Workbook describing the essential elements of the Canadian Pandemic Influenza Plan for the Health Sector and providing general information on the influenza virus, the history of influenza pandemics, the expected impact in Canada of a pandemic outbreak and the measures governments have taken and are considering in anticipation of a pandemic, including the use of vaccines and antivirals for treatment. It concludes with a brief discussion of the potential uses of antiviral medication for prevention, emphasizing that the benefits and risks of using them remain largely unproven. The Workbook (Annex B) contents were developed in consultation with and reviewed and approved by, TGAP.

• Each Dialogue Session lasted 1.5 days, beginning on Friday evening and ending Saturday afternoon. The Friday session consisted of a 2.5 hour information/education session, and the all-day Saturday session consisted almost exclusively of facilitated dialogue.

• At registration, participants were asked to complete an initial questionnaire assessing, among other things, their top-of-mind responses to three hypothetical approaches for using publicly funded antivirals for prevention. They then received a complete Workbook that included the initial information package they had received earlier by mail, as well as an agenda for the Dialogue session, rules of engagement and detailed information about three hypothetical approaches regarding the use of publicly funded antivirals for prophylaxis in a pandemic and arguments “in support” and “against” each approach. These approaches are discussed on the next page.
• Once participants were assembled around a horse-shoe shaped table, a facilitator at the front of the room welcomed them, introduced the Chief Medical Officer of Health for the province or territory (or his or her designate) and invited participants to briefly introduce themselves and speak to their interest in the session. The Chief Medical Officer of Health then went through a carefully scripted PowerPoint presentation (Annex C) that reviewed the history and science of pandemics; information on what is known about the use of vaccines and antivirals for both treatment and prevention; as well as other materials in the Workbook and the arguments for and against the three hypothetical approaches.

• The local/provincial pandemic preparedness plan was also described in the context of this presentation. Participants were actively encouraged to ask questions to clarify information and to discuss issues. Following the presentation and question and answer session, participants were asked to review the information in their Workbooks and arrive the next morning with any questions they wanted clarified.

• The next morning began with the facilitator asking participants if they had any additional questions for the Chief Medical Officer of Health vis-à-vis the materials or scenarios, and then providing an overview of the process. Participants were then asked to share their own personal experiences with a public health emergency in order to encourage dialogue and provide a sense of context for the discussion. Following this short discussion, the facilitator asked participants to focus on the following question:
  
  “On what basis should publicly funded antivirals for prevention be provided during an influenza pandemic?

• Three approaches were described for the purposes of discussion:

Approach 1: Minimize Serious Illness and Death

(Minimize serious illness and death by providing antivirals for prevention to take care of the most vulnerable)

Approach 2: Keep Society Functioning

(Maintain basic health, social and economic functions by providing antivirals for prevention to essential workers)

Approach 3: Minimize Governments’ Role

(Trust institutions and individuals to make their own decisions about whether or not to stockpile antivirals for prevention).

• These three approaches were developed by TGAP pursuant to a rigorous “issues-framing” session. During this session, TGAP assessed the issues and implications associated with various potential courses of action and their practical and political viability as “values-based” policy options. These approaches were then focus-tested with citizens to ensure they constituted “realistic” policy choices for which a reasonable degree of public support could be expected and then further refined (Annex D).

• The facilitator emphasized that the approaches were not current policy options being considered, but “scenarios” designed to generate dialogue, and that none of the three approaches was mutually exclusive. They were then told the approaches were based on the following assumptions:

• publicly funded antivirals for treatment will be available to everyone who requires them;
• it will take at least 6 months to produce sufficient vaccine to immunize all Canadians because production of vaccine can only begin after the particular influenza virus is known;
• it would not be feasible to give antiviral drugs for preventive use to all 33 million people in Canada; and
• the pandemic will be of moderate severity.
• Participants were then asked to discuss the merit of the three approaches. Following a lunch break, participants then formed smaller breakout groups to discuss what aspects of the three approaches they liked, did not like, and why. The results of these individual breakout sessions were then presented back to the plenary, and areas of consensus and disagreement across the groups in relation to each approach noted on a flip chart.

• After a short break, participants were again asked to form breakout groups, this time to reflect on what they had learned thus far and based on this, to discuss what types of categories of individuals they believed should be considered priority recipients for publicly funded antivirals in the event of a pandemic. Again, the results of these individual breakouts were reported back to the plenary and areas of consensus and divergence noted on a flipchart

• Participants were then asked to complete a post-Dialogue survey (Annex E) that sought to assess whether their top of mind views on the issue had evolved since the start of the dialogue process and if so, how and why.

• Next, participants were asked to consider whether there were any additional issues or elements governments needed to consider when coming to a decision on whether or not to make publicly funded antivirals available for prevention in a pandemic. Following this discussion, participants were asked to complete a final questionnaire which asked them whether, based on what they had learned over the course of the past day-and-a-half, they felt governments should provide publicly funded antivirals for protection in a pandemic, and why. The questionnaire also asked them to describe how they typically obtained information about public health issues and what types of public education efforts or materials they would counsel governments to use in relation to pandemic influenza.

• After closing comments by participants, facilitators and hosts, participants were asked to complete an evaluation of the process itself.

3. Dialogue Objectives

The dialogues were developed to meet three over-arching objectives:

• To better understand the expectations and perspectives of citizens, potential target group recipients and national stakeholders (as expressed in separate deliberative dialogue sessions) in relation to the formulation of a national policy recommendation on the use of publicly funded antivirals for prophylaxis in a pandemic influenza outbreak;
• To enable decision-makers to attenuate potential criticisms of the difficult policy choices that may be required vis-à-vis the use of antivirals for prophylaxis by demonstrating that a broad and inclusive process of dialogue and reflection preceded the decision;
• To demonstrate the value and benefits of involving citizens and other key constituencies and stakeholders in structured deliberative dialogue processes on difficult, ethically fraught public policy choices that may be associated with pandemic influenza outbreaks.

4. TGAP Dialogue Sessions

The process undertaken by TGAP and reported on in Part III comprised 11 separate sessions as follows:

• 5 Regional Citizen Dialogue Sessions
• 1 Citizen Dialogue Session convened by the Government of Quebec using an identical methodology and tools as TGAP
• 1 First Nations on Reserve Session
• 2 National Stakeholder Sessions
• 2 Target Group Sessions

Regional Citizen Dialogues & Quebec Dialogue

Citizen Dialogue sessions were convened in Halifax (November 2006), Toronto (November 2006), Winnipeg (November 2006), Vancouver (December 2006) and Iqaluit (January 2007). An additional session using an identical methodology and tools was convened in Montreal (February 2007) by the Government of Quebec. Each session brought together between 20-25 persons, randomly recruited by a public opinion research company to be representative of the region according to gender, age, education and income. Most participants came from within commuting distance of the city in which the consultations were held, although some were flown in from distant regions of the province, adjacent provinces or other territories.

Given the small sample size, the extent to which the results of these processes can be extrapolated is obviously limited. However, steps were taken to try and ensure that the views of Dialogue participants on key issues related to pandemic influenza approximated those of the wider population from which they were drawn. Accordingly, during the recruitment phase, all individuals contacted by the recruitment firm – whether or not they were eventually selected to participate – were surveyed to assess their awareness of, and views on, pandemic influenza. As Dialogue participants arrived at the sessions, they were asked to complete a pre-registration survey that asked questions similar to those posed during the recruitment process. The results of these separate assessments (i.e., the recruitment survey and the pre-registration questionnaire) suggest that on the whole, the views of Dialogue participants and the general population they were drawn from were essentially similar.

In addition to the overarching process objectives, the specific objectives for the Citizen Dialogues were:

• To identify the core values/principles citizens would want/expect to see reflected in decisions about whether and how publicly-funded antivirals for prophylaxis in a pandemic would be used;
• To provide a citizen perspective on the preferred priority recipients (different categories of stakeholders or individuals) in the event a decision is made to distribute limited supplies of publicly funded antivirals for prophylaxis in a pandemic;
• To provide a citizen perspective on the issue of whether the use of publicly funded antivirals for prophylaxis in a pandemic is warranted;
• To obtain an insight on where the public is lacking information on pandemic (in order to inform public education or information efforts);
• To assess what types of information would serve to reinforce or change the “top-of-mind” perspectives of citizens in relation to the use of preventative antivirals in a pandemic;
• To assess whether there are differences between the “group judgement” (expressed by participants through their common ground at the dialogues) and their individual judgements (as expressed on questionnaires);
• To ascertain any notable regional differences in perspective on the issues under review; and
• To learn more about the value and benefits of involving citizens in structured deliberative dialogue processes on difficult, ethically fraught public policy choices that may be associated with pandemic influenza outbreaks.

First Nations on Reserve Session

A separate session was convened in Edmonton in January 2007 that brought together some 28 First Nations peoples living on reserve from Alberta, Saskatchewan, British Columbia and the Yukon. The objectives for the First Nations on Reserve Dialogue were similar to those for the Citizen sessions, with the exception that an additional objective was to try and ascertain any notable differences in perspective between the results of the citizen dialogues and the On-Reserve First Nations dialogue.

National Stakeholder Sessions & Target Group Recipient Sessions

To complement the Citizen Dialogues, separate sessions were also convened in Ottawa for National Stakeholder Groups (28 participants) and for potential Target Group Recipients (46 participants). TGAP reviewed and approved both a Target Group invitation matrix and a proposed Stakeholder invitation list. Both sessions included a common Day 1 briefing/orientation session for all participants, but separately facilitated smaller dialogue sessions on Day 2.

Two distinct categories of National Stakeholders were invited to participate in the Dialogues: those with specific health sector expertise or members working in the health sector (e.g., nurses, physicians, allied health workers, epidemiologists); and non-health related groups whose interests or members would be directly affected by a pandemic influenza outbreak (e.g., police, firefighters, insurance companies, funeral homes, social services agencies, utility companies, business interests). An effort was made to ensure a balance of health and non-health participants in each of the Day 2 dialogue sessions.

Target Group participants were non-affiliated individuals selected based on the likelihood of their being identified as priority recipient of antivirals for prevention by citizens based on the nature of their work or professional qualifications (e.g., nurses, doctors, teachers, day-care workers, morticians, religious orders, “essential services” personnel working in hydro, water quality, transportation, etc). It should be noted that target group representatives were asked to participate as individuals rather than as advocates for their organizations’ position or interests.

The objectives set for the Stakeholder and Target Group Dialogues were:

• To identify the core values/principles national stakeholder organizations and potential target group recipients would want/expect to see reflected in decisions about whether and how publicly-funded antivirals for prophylaxis in a pandemic would be used;
• To provide perspectives of national stakeholder organizations and potential target group recipients as to the preferred priority recipients (different categories of stakeholders or individuals) if a decision is made to distribute limited supplies of publicly funded antivirals for prophylaxis in a pandemic;
• To provide perspectives of national stakeholder organizations and potential target group recipients on the issue of whether the use of publicly funded antivirals for prophylaxis in a pandemic is warranted;
• To obtain an insight as to where national stakeholder organizations and potential target group recipients are lacking information about pandemics (in order to inform education or information efforts);
• To assess what types of information would serve to reinforce or change the “top-of-mind” perspectives of national stakeholder organizations and potential target group recipients in relation to the use of preventative antivirals in a pandemic;
• To assess whether there are differences between the “group judgement” (expressed by participants through their common ground) and their individual judgements (expressed on questionnaires);
• To ascertain any notable differences in perspective between health and non-health stakeholders on the issues under review;
• To ascertain any notable differences between stakeholders and the target group participants they represent; and
• To learn more about the value and benefits of involving national stakeholder organizations and potential target group recipients in structured deliberative dialogue processes on difficult, ethically fraught public policy choices that may be associated with pandemic influenza outbreaks.

5. Pedagogical Tools and Materials

To ensure the “information-in” was consistent across the country and across sessions, every participant received the same Workbook and invitation letter. A common education session in the form of a PowerPoint presentation providing background information on pandemic influenza was also developed by TGAP for delivery by the Chief Medical Officer of Health for the designated jurisdiction where a Dialogue occurred. The common PowerPoint presentation was supplemented by information locally relevant to participants at the discretion of the Chief Medical Officer of Health.

As noted, participants were also asked to complete:

• a Pre-Registration Questionnaire to assess the top-of-mind responses of participants to key issues (i.e., the appropriate focus of pandemic preparedness efforts; the 3 approaches; priority group rankings)
• a Post-Dialogue Questionnaire to assess whether the top-of-mind views of participants shifted, how and why?
• an End-of-Day Questionnaire where participants were asked to advise government on whether or not to stockpile antivirals for prophylaxis as well as how best to communicate information on pandemic influenza planning and preparedness to Canadians
• an Evaluation Form on the value/quality of the Dialogue itself

6. Evaluation Framework

TGAP asked that the architects of the deliberative dialogue process develop a robust evaluation framework for assessing its value, both in terms of the dialogue outcomes themselves and in relation to the use of the “deliberative dialogue” approach as a decision support tool for policymakers.

Accordingly, an evaluation framework was developed that included explicit evaluation criteria in relation to the process itself and identified potential post-process areas of evaluation (Annex F). More to the point, the process architects viewed evaluation in two major ways: “…as a structured process of evaluating success against pre-set criteria; and, with equal importance, as an opportunity to learn, and to share learnings among participants, agency staff/organizers, and process designers.

In relation to the process itself (i.e., each of the 11 separate dialogue sessions), four key questions served to frame the evaluation approach and methodologies:

1. Did the process provide a perspective of the relevant participant group (e.g., citizens, stakeholders, etc) on these issues?
2. Was the process effective in providing insight on where the participant groups (e.g., citizens, stakeholders, etc) are lacking information and what types of information would affect or reinforce “top-of-mind” perspectives?
3. Were the value and benefits of involving participant groups in structured deliberative dialogue demonstrated?
4. What lessons were learned over the course of the entire process, from pre-planning to post-consultation?

For each area of inquiry, indicators were developed and potential data sources identified. The decision at the recruitment stage to assess the relative top-of-mind views of citizen dialogue participants and of the larger regional population from which they were drawn; the pre-and post-Dialogue questionnaires that asked participants to state their views on the salient issues both before and after discussion; the third questionnaire that asked participants whether governments should purchase antivirals for prevention and if so, under what conditions; and, the process evaluation questionnaire that asked participants to rate various technical aspects of the consultation were all elements of the comprehensive approach to evaluation.

A similar approach was taken to the drafting of the final report. In addition to the questionnaire responses and flip chart notes taken at the meetings themselves, each session was audio recorded and attended by a note-taker/observer from EKOS, the research firm engaged to prepare the final report. Facilitators were also asked to provide their top-line views of the process and these initial views were shared with observers to ensure consistency. Before summarizing each individual session, EKOS reviewed the observers’ notes, the facilitators’ notes and listened to the audio-recording of the session to confirm where there was consensus or disagreement on issues.

Finally, it should be noted that while the original proposed process design included additional components, timing and resource constraints precluded their implementation. These additional components, which would have added additional depth and breadth to the process, were to have included:

• A national synthesis session wherein a cross-section of participants from each of the 11 sessions would have been invited to participate in a national meeting to present and discuss the results of their individual sessions with a view to assessing the degree of consensus and common ground across the various sessions; and
• A parallel on-line dialogue process wherein an e-version of a variant of the Workbook along with a closed survey would have been posted to assess whether the results of the Dialogue process and the views of the broader population would dovetail.

The fact that TGAP opted not to proceed with these additional process elements in no way diminishes the overall value of the exercise. Had there been significant differences of view expressed through the various dialogue sessions, the importance of these additional elements would have increased significantly. This being said, from an academic and research perspective, there is considerable merit in further evaluation of the process, the methodology deployed and the views and perspectives of both participants and decision-makers regarding the value of the process over both short-term (i.e., was it helpful in formulating a decision?) and medium-term (i.e., did the final policy decision take account of the dialogue inputs and if so, how?).

A broader discussion of evaluation is included in Part III.

Part II: Dialogue Results – What We Heard

As noted in Part I of this report, a rigorous evaluation framework was developed for assessing the results of the Dialogue process. While a brief summary of the highlights of individual Dialogue sessions and of the overall process results relative to the pre-set evaluation criteria is presented later in this section, the following narrative captures the essential elements of the process for policy-makers:

1. While there are some discernable differences of perspective across the various Citizen, National Stakeholder and Target Group sessions, from a public policy perspective, these differences are not significant enough to require any variation in either the decision confronting policy makers or in future information or outreach.
2. When asked to select from among the three potential approaches for framing policy decisions, a majority of participants (Citizens, Stakeholders and Target Groups) rated Keep Society Functioning the highest. The second rated approach, Minimize Serious Illness and Death also enjoyed considerable support while Minimize Government’s Role rated lowest. In the post-dialogue questionnaire, overall support for the first two rated approaches rose, but decreased further for the third.
3. There is an overall and unambiguous consensus that governments should create an additional national stockpile of antivirals to be used for prophylaxis.
4. Notwithstanding this overall consensus, the level of support for creating this additional stockpile of antivirals for prophylaxis can best be characterized as guarded and cautious.
5. This nuanced support for stockpiling antivirals for prophylaxis reflects a variety of concerns about the “unknowns” and includes:

Safety:

• the risks of side-effects from taking antivirals for extended periods or from not maintaining the course of prophylaxis (i.e., 1 pill daily for 8 weeks)
• the potential for antiviral resistance developing

Effectiveness:

• the daunting logistical challenges associated with the timely distribution of antivirals (especially in rural/remote area and outside of institutional settings)
• concern about compliance (i.e., is it realistic to think that people will follow the course of prophylaxis of 1 pill daily for 8 weeks?)
• uncertainty as to whether antivirals will prove effective for prevention

Equity

• whether defensible and fair criteria will be used for selecting priority recipients
• whether this criteria will be respected
• concern about the potential impact on the principle of universal access to health care

Opportunity Cost

• whether investments in areas like “public education” might be more effective
• the opportunity cost of purchasing antivirals versus other public health investments

Individual/Institutional Accountability

• what role and responsibility should individuals and institutions have vis-à-vis a decision to use antivirals for prevention
• the circumstances under which using public funds for the purchase of antivirals for prevention is warranted
• the value of reciprocity or the responsibility of those receiving the medications to actually show up for work and to complete the course of treatment.

6. These caveats are clearly evident in the general consensus that in the event a decision is taken to stockpile publicly funded antivirals for prevention, they should be made available to the fewest possible number of recipients necessary to “do the job”- not because of cost, but because of the “unknowns”. Accordingly, it would not be unreasonable to conclude that availability of new information in regard to risks/benefits might yield different results.
7. In terms of priority rankings, when asked to select the top 3 from a list of potential recipients, (for access to publicly funded antivirals for prevention), citizens rated front-line health care workers with close patient contact rated highest (and those without close patient contact rated second highest). Support for the vulnerable groups and for emergency workers showed considerable variance but was mostly equal. (Among the vulnerable, children were cited most frequently, followed by the chronically ill, (the immune-deficient, and the elderly.)
8. Among target group members and stakeholders, the results were less ambiguous and more pragmatic. Health care workers with close patient contact rated the highest. There was also considerable support for other health care workers regardless of patient contact. Emergency and essential workers rated second highest, while vulnerable groups rated last. It should be noted that the high ranking health and emergency workers received was more a reflection of pragmatism (i.e., “they are needed to keep society functioning and to contain the spread”) than of concern over the fact that they are at greater risk of exposure by virtue of their occupation.

In terms of the “core values” participants “brought to the table” and that they appear to want to see expressed in the policies and decisions taken with respect to the potential stockpiling of antivirals for prevention and the designation of priority recipients, a number are self-evident:

1. Pragmatism: ensuring society continues to function was seen as the over-riding concern for most dialogue participants. Consistent with this, for some participants, containing or limiting the spread of the virus appeared to be the central rationale for their support of government involvement in antivirals for prophylaxis. In this context, the almost unqualified support for providing antivirals for prophylaxis to front-line health care workers (and to a lesser extent emergency workers) is scarcely surprising (i.e., not only are these groups essential to keep society functioning, it is easier to distribute the medication to them in a timely way, they are more likely to complete the course of prophylaxis, and it will enable them to care for the ill and thereby reduce the stress on an already over-stressed health care system). Accordingly, despite concerns over the “unknowns”, investing in an antiviral stockpile for prevention was seen as prudent, responsible and reasonable.
   
   In a similar vein, the reasons cited for rejecting Approach 3 (Minimize Governments’ Role) are equally instructive (i.e., this approach precludes governments using their purchasing power to keep costs low; a “laissez faire” attitude, or the perception that government is not providing leadership, could result in public outrage and anarchy; and, someone needs to be accountable for those who are unable to get a prescription because they have no doctor, or who may be essential workers – like hospital cleaning staff – but cannot afford to purchase antivirals out-of-pocket).

2. Compassion, Fairness and Responsibility: Although for most dialogue participants “pragmatism” appears to have trumped “compassion”, this by no means suggests a “carte blanche” for policy makers. Indeed, compassion was a strong value for participants and there was significant discussion over the course of the Dialogues on issues related to equity and fairness.
   
   For example, many participants argued that society had an “obligation” to take every reasonable step to protect the health and safety of those individuals (and to a lesser extent their immediate families) likely to find themselves on the front-lines of a pandemic response. Hence, they viewed the provision of antivirals for prevention for these groups – regardless of whether the efficacy of these medications could be clearly demonstrated – as reasonable, fair and necessary. (However, there was also considerable discussion of reciprocity: the responsibility of those who receive antivirals for prophylaxis to actually show up for work and to complete the course of treatment.)
   
   Other participants emphasized the need for an early and sustained public education effort to ensure people understood that there was an objective and defensible rationale for designating priority groups, and that there would be clear rules, consistently applied, in regard to who would receive antivirals for protection. The concerns expressed in the Iqaluit citizen dialogue and at the Edmonton First Nations on Reserve session also underscore equity concerns: that the needs of rural and remote residents not be given short shrift by policy makers.
   
   There was also an acknowledgement that, all things being equal and once the needs of health care and emergency workers had been addressed, society had a duty to “do right” for the most vulnerable segments of society – with children being given precedence over other potential “vulnerable” categories.

9. As noted earlier, in its detailed analysis of the Dialogue process, EKOS Research notes that across the regional Citizen Dialogue sessions, there are discernable differences of perspective on certain issues. However, from a public policy perspective, these differences are not so significant as to require any variation in either the decision confronting policy makers or in future information or outreach in relation to the decision FPT Health Ministers take. These regional variations include:

In terms of Approach I: Minimize Serious Illness and Death

• Participants in Halifax and Iqaluit, and in the First Nations on Reserve sessions found this approach particularly compelling, while those in Vancouver were wary of the costs associated with this approach.

In terms of Approach III: Minimize Governments’ Role

• While this approach was generally rejected at all of the Dialogues, First Nations on Reserve participants saw greater community involvement (and less government involvement) as a positive thing. However, they also were emphatic that governments have an obligation to make publicly funded antivirals for prevention available to any community that wants them.

In terms of Common Ground

• The Toronto session placed a greater emphasis on issues relating to universal access; Vancouver participants tended to focus more on opportunity costs and the need for research; Halifax participants expressed stronger concerns than other sessions on the needs of the economically vulnerable; First Nations were dubious of the health system’s capacity and inclination to meet First Nations’ needs; and, Iqaluit participants emphasized the importance of community-based strategies.

In terms of Priority Recipients

• Halifax, Toronto and First Nation participants tended to have greater empathy for the vulnerable and in fact, included low-income individuals and families in their definition of “vulnerable”.

10. Similarly, the Final Report by EKOS Research (Annex G) also notes some differences of perspective between citizens, stakeholders and target groups on certain issues. Again, these differences are not so significant as to require any variation in either the decision confronting policy makers or in future information or outreach in relation to the decision FPT Health Ministers take. These variations include:

In terms of Approach I: Minimize Serious Illness and Death

• Stakeholders and Target Groups saw some merit in this but were generally less enthusiastic about this approach than were citizens.

In terms of Common Ground

• Stakeholders and Target Groups placed greater emphasis on practical considerations (e.g., impact; logistics; opportunity costs) than on values-based ones.

11. Among the other salient findings and observations from the Dialogues, three in particular stand-out:
    
    

The Imperative for Public Education as a Form of Prevention

• At every Dialogue session, participants emphasized time and again the need for governments to invest in public education on pandemic preparedness as a form of prevention. Indeed, at several sessions, there was considerable discussion as to whether in terms of value-for-money, investing in public education would yield better results than investing in antivirals for prophylaxis.

The Need for Additional Research on the Use of Antivirals for Prevention

• Another recurring theme across all of the Dialogue sessions was the need for additional research on the long-term benefits and risks of using antivirals for prevention. In this regard, it is again worth noting that the level of support for stockpiling antivirals for prophylaxis was nuanced and based more on prudence/insurance principles than on a deep conviction that antivirals for prophylaxis made sense. The more significant and perhaps obvious point from an objective reading of the Dialogue results is that the level of support for stockpiling antivirals for prophylaxis could increase or decrease significantly based on clear research evidence as to the safety and effectiveness of such medications when used by healthy individuals for an extended period of time. To that end, a case could be made that rather than closing the door on the possibility of antivirals being stockpiled for prophylaxis, some participants expressed support for stockpiling pending relevant research results becoming available. More colloquially, this position could be expressed as: “I am not convinced antivirals for prophylaxis will work, but I am not prepared to rule out stockpiling them for prevention either until there is clear evidence that the potential health and safety risks outweigh the potential benefits”.

Need for Government Leadership

• A third recurring theme or undercurrent in the Dialogue sessions was the desire and expectation for clear and visible government leadership on the issue. This was expressed in a number of ways. For example, in the closing remarks at each of the Dialogue sessions, many participants indicated that although they expected governments to take account of the various Dialogue results in their deliberations, that this did not necessarily mean governments should adopt their proposed prescription. Participants also made it clear that they did not want the decision on whether or not to stockpile antivirals for prevention to be based solely on science or economics, but also on fairness, equity and compassion.

• In addition, several participants noted that a potential by-product of governments not providing leadership was panic and anarchy. They saw this leadership as including the imperative for governments to share knowledge in a transparent and proactive manner. This was expressed as governments having a “duty” to keep citizens informed in the event of an outbreak, to explain clearly to them what the plans are for dealing with it, and to reassure them that these plans have been objectively developed based on the best available information and will be applied fairly.

• Taken together, these themes highlight the imperative for governments to reconcile the range of issues associated with this decision, make its choice and communicate the rationale clearly and aggressively in advance of a potential outbreak.

12. At the First Nations on Reserve session in Edmonton, some participants spoke of the need for decision-makers to recognize the potential contribution of traditional medicine to pandemic planning and response efforts.

Part III: Process Observations, Evaluation and Next Steps

Were the overall objectives established for the Dialogue sessions met?

As noted in Part II, four key questions served to frame the evaluation approach and methodologies:

1. Did the process provide a perspective of the relevant participant group (e.g., citizens, stakeholders, etc) on these issues?
2. Was the process effective in providing insight on where the participant groups (e.g., citizens, stakeholders, etc) are lacking information and what types of information would affect or reinforce “top-of-mind” perspectives?
3. Were the value and benefits of involving participant groups in structured deliberative dialogue demonstrated?
4. What lessons were learned over the course of the entire process, from pre-planning to post-consultation?

Based on the findings and conclusions contained in the EKOS Final Report (Annex G), a review of the pre- and post-dialogue questionnaires and of the participant evaluation survey for each session, the short answer to these first three questions is a resounding, YES. Moreover, an informal canvassing of the views of TGAP members who observed various sessions indicates that most were surprised by the ease with which participants proved capable of assimilating the core issues under discussion and the breadth and sophistication of the dialogue that took place during the sessions. Indeed, more than one TGAP member observed that over the course of the 11 sessions, every issue or consideration that TGAP had been grappling with – and not just those included in the Workbook – had been raised and debated by the participants.

In regard to the fourth point, a more comprehensive assessment of the process should be considered once a decision has been made on if and how to proceed on the two issues under discussion: whether to stockpile antivirals for prophylaxis and if so, who should be given priority access to them. Ideally, this review would assess whether or not, how and why the results of the Dialogues affected the choices or directions taken. In addition to the intrinsic value of such an undertaking from a policy development perspective, sharing with Dialogue participants how the results of the process were ultimately taken into account would also enable them to make a better informed judgment on the value of the initiative.

Did Deliberative Dialogue prove to be an appropriate methodology for assessing the views of participants in relation to the issues in question?

There are two separate dimensions to this question. The first is whether or not assessing the views of citizens, stakeholders, target groups and representatives from First Nations on Reserve makes any sense for an entity like the Task Group on Antivirals for Prophylaxis (TGAP). The second is whether the Deliberative Dialogue was the appropriate instrument for gauging the views of the various participants.

In regard to the first dimension, the answer is wholly context-specific. There is no question that it is always useful and valuable for policy and decision-makers to ascertain the views of citizens and of those potentially affected by a proposed public policy decision. Doing so strengthens democracy, confers a degree of legitimacy on the decision taken and makes good political sense (depending on how credible the engagement process is seen to be). But in the context of antivirals for prophylaxis, the more relevant questions are whether TGAP could arrive at a “politically palatable” decision without assessing the views of citizens or stakeholders, or if the recommendation TGAP ultimately makes is affected by the information acquired through the Dialogue process.

Given the broad scope of the research TGAP has marshalled or commissioned (besides the Deliberative Dialogues), it is likely that a compelling and evidence-informed case could be made to the PHN Council on how to proceed. Indeed, TGAP was established as an expert advisory panel to assess the available scientific evidence, best practices and legal and ethical considerations that should inform such a policy decision. The fact that TGAP chose to seek the perspective of a broad spectrum of citizens and stakeholders suggests that the scientific evidence was sufficiently unclear that a decision based on the “usual parameters” was not easily forthcoming and that assessing common “values” mattered. But the decision also offers some potentially significant and constructive “value-added” to the advice TGAP offers (i.e., “Ministers, the good news is that the approach we are proposing resonates strongly with a whole lot of people and groups” or, “Ministers, we believe that on balance, this is the right way to proceed, but we can tell you now that you’ll have a significant public education challenge ahead of you”). More to the point, TGAP could also easily have said: “Ministers, here is our best advice; we think you should focus-test/consult on it before proceeding with implementation to assess how the public and key stakeholders will react.”

This being said, over the past several years, regulatory authorities across North America have been experimenting with new methods for involving citizens in resolving difficult policy choices where the science is unclear or where science and ethics “bump up” against each other. For example, both in Canada and the United States, expert scientific panels established to determine whether or not to resume sales of Cox-2 inhibitor drugs (VIOXX) specifically solicited “non-scientific” and “experiential” information both from the families of those harmed by the drug and from individuals whose quality of life was significantly improved by them. There is also a growing body of literature on how expert scientific panels can factor public values and non-scientific evidence into what are ostensibly meant to be evidence-based technical decisions. In this context, the use of deliberative dialogue processes can be useful and appropriate mechanisms for developing and then supporting decisions on ethically fraught science-based decisions.

In terms of whether Deliberative Dialogue was the appropriate engagement methodology, the answer again is a matter of perspective subjective. On one hand, few other research methodologies offer participants the same opportunity to immerse themselves in an issue and arrive at an informed decision. As well, few methodologies oblige participants to work through complex, multi-faceted issues and make the types of difficult choices and trade-offs policy-makers are regularly confronted with (i.e., “…does your belief in individual accountability trump your belief that society has a duty to protect the vulnerable?”). Indeed, a strong case can be made that the perspectives or opinions of participants in a well-managed deliberative dialogue process are likelier to be far more robust than those flowing from other methodologies that simply assess “top-of-mind” perspectives. The question is, does this really matter and is it worth the cost?

In regard to “does this really matter”, the answer would appear to be, YES. Understanding not just the initial perspectives of participants, but the core values that frame their initial views, how competing values are reconciled through the dialogue process, and what types of points of considerations are found to be the most compelling can provide invaluable cues for how to communicate the rationale for, and elements of, a difficult or ethically fraught policy decision. The process also can help to assess where “public judgment” would be situated when citizens have the opportunity to think through the issues.

Finally, regarding the question of “is it worth the cost”, the answer is partly subjective and partly contextual. Certainly, other research methodologies can be less expensive to implement, less demanding on planners, participants and facilitators, and faster to complete. However, these do not necessarily produce results that are as robust as those flowing from properly conceived and well implemented dialogues. Similarly, attempting to quantify the value of deliberative processes (or, indeed, any other form of citizen or stakeholder consultation or engagement) is always difficult. This being said, from a pure “risk communications” perspective, it is hard to deny the value to policy-makers of being able to say that prior to taking their decision, they engaged ordinary citizens across the country in a process of dialogue to better understand the values citizens wanted expressed in the policies or programs being announced.

Was the process developed under TGAP’s direction an effective one?

This question cannot fully be answered until TGAP has taken its final decision and communicated its rationale for the decision to the PHN Council, and until the PHN Council has decided whether to heed the TGAP advice and why. Indeed, there is an obvious need to follow up with TGAP members on how the results of the deliberative dialogue processes influenced their individual and collective decisions.

However, the overwhelmingly positive assessment by participants in the various dialogue sessions as to the value of the exercise is striking. So too was the extent of consensus across the sessions on most of the salient issues and decisions. It seems clear that the elements of the three proposed approaches “captured” the issue well. This being said, for many – but not all participants – the dialogues served to reinforce their initial top-of-mind views (although over the course of the Dialogues, support for children being included as priority recipients increased somewhat at the expense of other groups in the “vulnerable group” category, and support for Approach 3, Minimize Governments’ Role fell).

From a purely methodological perspective, there are a number of potential variables associated with the process that could have been better controlled. It should also be noted that there are caveats in relation to the process that suggest caution be used when interpreting or extrapolating the results of these sessions. Some of these process “challenges” reflect the realities and dynamics of inter-governmental processes in sensitive areas of shared jurisdiction. Others result from the short time frame those responsible for implementing the process had to work with. The key point is that none of these “challenges” – either individually or collectively – was fatal to, or significantly compromised, the process results. Their enumeration below is for future research and evaluation purposes:

Recruitment for the Citizen Sessions and First Nations on Reserve Session

Process planners made it clear at the outset that in light of the relatively small number of participants at each of these sessions, and given that most sessions included “regional” participants, the results should not be considered as “representative” of anything other than the views of the 30 or so citizens (or the First Nations residents) who were drawn largely, but not exclusively, from the host city and environs. However, care was taken in selecting citizen participants to verify that their “top-of-mind” views on key issues were largely identical to those of the regional population from which they were drawn.

It should also be noted that despite efforts to ensure balanced representation at each session, the limited number of participants and the requirement in some sessions to include “regional participants” resulted in some obvious demographic anomalies in terms of the number of visible minorities, aboriginals and youth participants.

Recruitment of Target Group Members

While an effort was made through the development of a selection matrix to recruit a wide cross-session of potential priority group recipients, not all who were invited attended. As well, those participating were specifically requested to participate in the sessions as “individuals” rather than as advocates for their company, organization or association position. This underscores the need for caution when making reference to the “position” of target group participants.

Day 1 Presentations

Although a “core presentation” was developed for the Day 1 sessions and approved by TGAP, provincial Chief Medical Officers of Health (or their designates) had considerable latitude to incorporate locally relevant information. Indeed, some presenters chose to tailor their presentations to what they believed to be relevant to their “audience”. Similarly, external factors may also have affected the results of some of the sessions (i.e., the Vancouver sessions were held shortly after severe storms and extended boil water advisory; the release of a Commission of Inquiry into SARS was released prior to the January sessions). The extent to which these differences and factors may have affected the dialogue outcomes is not clear.

In addition, in their Day 1 remarks, most presenters outlined in some detail the specific elements of their home province’s pandemic preparedness plan. Given that most citizen sessions included participants from adjoining provinces (e.g., the Halifax session included participants from PEI, New Brunswick, and Newfoundland), it is also unclear whether the views of “regional” participants were different than those of the host province who had the benefit of an overview of what steps were being taken locally to prepare for a pandemic.

Questionnaires

Several participants indicated that they found some parts of the questionnaires difficult to understand and requested help in completing them. This is especially true in regard to their being asked to rank the three approaches and priority recipients. It is not clear whether or how this affected the results. However, the broad consensus across the dialogues on most issues suggests that this was not necessarily a major problem.

Clarification of Key Concepts

Over the course of the dialogues, participants saw little differentiation between “essential workers”, “emergency workers”, and “first responders”, and where different categories of “health care workers” fit in relation to these groupings (or whether they formed a separate category).

There also seemed to be wide variation in terms of how they chose to define the concept of “vulnerable groups” in that they viewed them as including the economically vulnerable, children, the elderly, the sick, the disabled, those with immune deficiencies, the homeless, and so on. While this may have caused some confusion, these different perspectives on how to characterize the “vulnerable” proved both insightful and useful.

Is additional evaluation required and, if so, what should be the priorities?

There are a number of areas where a further evaluation of the Deliberative Dialogues and of the TGAP process could be undertaken or mined for useful public policy information. These include, among others:

• A thank-you letter to all session participants inviting general feedback on the process; asking whether they have shared any of their “learnings” with friends, families or colleagues; and, having had a chance to reflect on the session, whether their views have continued to evolve and if so, how?
• A questionnaire for TGAP/PHN Council members on the perceived value and utility of the process, how the Dialogue results influenced their thinking or advice, and if they believe value-for-money was achieved.
• One-on-one interviews with key TGAP members/process observers to obtain a deeper insight into the value and utility of the Dialogue process that was utilized.
• A roundtable discussion with TGAP members on the process design, methodology and results at a future TGAP meeting.
• A facilitated roundtable with Dialogue staff, facilitators and conveners to identify lessons learned and to discuss the process design and value.
  
• A letter to participants once the EKOS Final Report becomes public seeking their views on the perceived value of the process based on how the results of their session compared to others (and, ideally, supplemented by information on how the results of the sessions affected the policy choices or directions taken in regard to the issues in question).
• A final “roll-up” evaluation report that incorporates each of the elements above as well as the various participant process evaluation survey results included in the EKOS report.

Part IV: Annexes (FORTHCOMING)

Annex A: Schedule of Dialogues

Annex B: Participant Workbook

Annex C: Core Presentation

Annex D: TGAP Issues Framing Report

Annex E : Participant Surveys/Questionnaires

Annex F: Evaluation Framework

Annex G: EKOS Final Report

[Previous] [Table of Contents] [Next]