ARCHIVED - Formative Evaluation of the National Collaborating Centres for Public Health Program (NCCPH)

Appendix E: Detailed Results by Evaluation Issue

Issue 3:  Resource Adequacy

Issue #3 relates to the evaluation question – to what extent is the right level of resources available to effectively deliver the NCCPH Program?  The title for this issue in the February 2008 Progress Evaluation Framework was cost-effectiveness.  It was beyond the scope of this formative evaluation to conduct a formal cost-effectiveness analysis; thus, the title was changed to Resource Adequacy.  Information to address the question is largely based on perceptions of resource sufficiency as reported by NCC informant groups.

Findings are presented under the following section headings:

• Financial resources; and
• Human resources.  

Financial Resources

Description of Financial Resources

The original intended funding allocation for the NCCPH Program as described in the February 2008 Progress Evaluation Framework is provided in Table 6.  These projected budget figures include ongoing performance measurement and NCCPH evaluation.

Available funding and eligible expenses under the Population Health Fund were outlined in the RFP instructions.  For start up, each NCC could apply for up to $450,000 for a duration of up to 12 months.  It was determined NCCs could request funds for:

• Key staff positions including a Scientific Director and/or other project staff;
• Development of program materials and activities;
• Overhead expenses (e.g., telephone, travel, conference attendance, networking activities); and
• Evaluation activities. 

Ineligible activities and expenditures included:

• Pure research;
• Profit-making activities;
• Provision of direct services that are the responsibility of other governments;
• Audiovisual productions without approval;
• Capital costs;
• Ongoing organizational support/overhead; and/or
• Administrative fees, contingency allowances or other miscellaneous fees.

Based on Government of Canada spending reductions, an amendment was made to each NCC’s Contribution Agreement to decrease funding by $41,000 in the 2007/08 and a further $69,000 in 2008/09, or $110,000 in total.  Thus, actual budgets for the NCCs through contribution payments were available for amounts up to $1.459 million and $1.39 million in 2007/08 and 2008/09 respectively.  The actual budgets for each NCC by fiscal year are presented in Table 7.

According to the financial review conducted by BMCI Consulting Inc., in which cumulative budgets (projected expenses) were compared with actual expenses from 2005/06 to 2008/09, five out of six NCCs were noted to have under spent their available dollars, and one NCC was reported to have exceeded their budget.^{Footnote 39}

Managing Directors were asked to identify additional resources leveraged.  Three out of six NCCs have received additional dollars.  Examples of the types of organizations from which these NCCs have leveraged resources included provincial governments; federal government (e.g., FNIH, PHAC); and/or provincial, national, or international organizations (e.g., British Columbia Occupational and Environmental Health Research Network, Canadian International Development Agency, United Nations Children's Fund).  Two NCCs identified in-kind contributions, including AB members and partners/collaborators time and host contributions (e.g., financial systems, infrastructure, website development).  A detailed description of reported resources leveraged by NCC is included in the summary of activities and achievements in Appendix F.  

When asked to specify the infrastructure developed for their NCC, five out of six NCCs identified the existing facilities and infrastructure (e.g., IT, financial and human resource systems) of the host organization.  Two NCCs listed the physical space of the office including furniture, computers, and printers as infrastructure developed as a result of the program.  A detailed description of each NCC’s reported infrastructure is included in the summary of activities and achievements in Appendix F.  

Feedback

Informant groups were asked if, in their opinion, the allocated NCCPH Program budget was sufficient to achieve the Program’s intended outcomes.

Four NCC informant groups stated that they are at a point where they require more funding to reach the potential they envision.  Representatives from two NCCs said they tailor their activities to the available dollars.  Only one NCC strongly advocated for additional funding based on the breadth of their mandate.  Some NCC representatives highlighted how acutely aware they were of the funding reduction.  One other noted theme was the challenge associated with spending start up dollars during the first year.

When asked for recommendations, half of NCC informant groups advocated for more money or progressive increases in order to plan and develop strategies.  Two NCCs recommended additional funding to expand their scope to knowledge generation. 

Informant groups were asked to provide suggestions to improve cost efficiency and effectiveness in delivering the NCCPH Program.  Suggestions included:

• Collaboration with other agencies (e.g., build on their processes, use existing networks);
• Using the website as a vehicle for information sharing;
• Use internal resources rather than issuing a request for proposals;
• Contract services out;
• Leverage NCC funds as an inducement to raise additional funds; and
• Reduce travel costs by using alternative communication (e.g., videoconferencing).

Human Resources

Description

Human resources available to the NCCPH Program are described by: NCCPH Program Secretariat, NCCs, Advisory Council and Advisory Boards.  Based primarily on the revised Contact List - National Collaborating Centres for Public Health (June 2008) the NCCPH Program Secretariat consists of five members including Senior Program Advisor, Acting Manager, Senior Projects Officer, Senior Policy Analyst, and Project Officer. 

As of the June 2008 revised contact list, there is 46 staff across all six NCCs, ranging from five to ten members per NCC.  In general, each NCC employs a Scientific Director and a Managing Director.  NCC staff possess a mixture of backgrounds in research, relevant public health content areas, and KSTE.  Examples of staff titles (other than Scientific Director and Managing Director) across the NCCs include: administrative assistant; agente de recherché; associate director; chargee de projects; communications coordinator/officer; executive assistant; knowledge broker; knowledge translation manager/officer/ scientist; network coordinator; post doctorate fellow; program administrator; program assistant; project coordinator/manager; redacteur anglophone et soutien aux produits de connaissances; research assistant; research coordinator; and technicienne en administration.

According to the most recent contact list of Advisory Council members (July 2008), there is a total of 11 members with a mixture of public health expertise.  Four members are affiliated with policy or practitioner organizations (e.g., Medical Officers of Health, government), five members are associated with research or academic centres, and two members represent international expertise (e.g., U.S. Centers for Disease Control, National Institute for Health and Clinical Excellence).

Based on available data, NCC Advisory Boards range in size from nine to 18 members per board.  Two had a mix of half policy makers/practitioners and half researchers/academics.  The other four Advisory Boards were more heavily weighted with policy makers/practitioners (70-80%) than with researchers/ academics (20-30%).  Three NCCs identified staff from other NCCs as AB members, ranging from one to three NCC members per AB.  

Feedback and Suggestions

Human resource mix

Informants were asked to comment on the mix of personnel in the NCCs or in the NCCPH Program generally.  Responses were mixed.  Respondents in five informant groups said the mix of personnel is not adequate (e.g., not enough people, lack KT expertise). 

Respondents in four informant groups identified recruitment issues at the NCC level as a result of the location of the NCC, uncertainty of funding, inability to attract front line workers because of term positions, or difficulty finding qualified staff (e.g., skills in content area and KT).  Three informant groups indicated the mix of personnel is improving. 

When feedback from the NCC informant groups only are considered, the perception is more positive, with NCC representatives suggesting an appropriate mix of people available at the NCC level.  Some highlighted specific achievements in this regard (e.g., conducted a human resources review, hired a knowledge dissemination person). 

The requisite NCCPH skill set, as reported by informants, are:

• Content knowledge;
• Highly educated with credentials;
• Relevant background;
• Public health experience;
• Ability to translate content;
• Access to networks; and
• Versatility and flexibility.

Respondents in four informant groups discussed the position of Scientific Director when asked about the mix of personnel, particularly in relation to finding the right person with the appropriate qualifications.  Their comments included: 

• The Scientific Director position is crucial and you need the right person who understands the issues, purpose and vision of NCCs;
• Scientific Directors need to be experienced in field but not struggling for tenure;
• Degree of qualification (e.g. PhD) may limit ability to find the appropriate person; and
• There has been definite improvement with a strong Scientific Director at a particular NCC; Scientific Director has public health experience and people are drawn to him/her.

Four Advisory Board members commented on the mix of personnel at this level.  Of these, the majority said there was the right mix of people at the board level. 

Representatives from half of the NCCs and one other informant group discussed the NCCPH Program Secretariat.  Staff turnover within the Program Secretariat was raised as a concern.  People within two informant groups said the commitment of the Secretariat staff goes beyond the call of duty

Several informant groups identified their host organizations as facilitating the availability of the right mix of personnel, including:

• Availability of leaders within the NCC content area; and
• Access to people with the necessary backgrounds, enthusiasm and/or talent to support the work of the NCC. 

Human resource challenges and barriers

Informants were asked to describe any human resource challenges they encountered.  Their comments focused around:

• Recruitment/retention;
• Financial resources;
• Need for additional human resources; and
• Barriers associated with the host organization.

Respondents in the majority of NCC informant groups and one other informant group identified recruitment and retention issues related to:

• Short-term contracts;
• Difficulty finding the right people with content knowledge;
• Job insecurity and uncertainty of funding;
• No opportunities for career laddering; and
• Changing job descriptions.

Several informant groups cited financial resources as an issue.  Examples included insufficient budgets to be able to do a good job, and inadequate NCC staff salaries.

Half of the NCC informants groups identified barriers associated with host organizations as challenges for human resources, including:

• Time required to go through fulltime and union positions process;
• Requirement by host’s human resource department to complete a job classification questionnaire for new positions;
• Difficulty undertaking contract arrangements – a feature of the provincial body under whose rules the host organization operates; and
• Different host human resource policies resulting in NCCs having different contracts (e.g., some NCC contracts with host include benefits for employees).

A few NCC informant groups identified the need for additional human resources, including specific positions at the NCC level (e.g. KT Program manager, senior scientist, and researcher).

Suggestions for enhancement

Informants were asked to provide recommendations regarding human resources.  Half of the NCC groups and one other informant group identified the need for additional staff at the NCC level or the NCCPH Program Secretariat.  Half of the NCC informant groups described the need for the appropriate skills among personnel, including the right mix of skills for KSTE, variety of backgrounds, people with public health experience, and more generalists versus specialists.   One suggestion was to second senior people on a short-term basis to infuse public health expertise into the program.

In relation to recruitment and retention of personnel, three informant groups recommended the following solutions:

• Longer funding period;
• Offer longer contracts;
• Offer job security;
• Commit to program continuation;
• Offer part-time employment for KT people; and
• Develop career laddering opportunities.

A few informant groups recommended additional resources with two specifying resources for the NCCPH Program Secretariat.

A few informant groups commented on the need for flexibility in relation to human resources (e.g., flexible staffing model and flexibility to contract people).

Others highlighted facilitating factors to successful recruitment, including the Scientific Director’s previous relationships, the appeal of working for the NCCPH, and having a strong host institution.