The Alliance for HPV Free Colorado: A Multidisciplinary Approach to HPV Vaccination

State: CO Type: Promising Practice Year: 2020

LHD Description: Denver Public Health (DPH) is a nationally recognized public health department located in Denver, Colorado. Denver's estimated population is 706,914 and is rapidly growing and diverse; 54% identify as White, 31% Hispanic, 9% Black, and 3% Asian. DPH is organizationally housed under Denver Health and Hospital Authority (DH), a large safety-net institution providing integrated services ranging from primary prevention to acute care. DPH collaborates with many partners to inform, educate, offer services, and promote policy change that aim to make Denver a healthy community for all people. More information on DPH is available at

Public Health Issue: Each year approximately 34,800 cases of HPV-attributed cancer are diagnosed in the United States (CDC). Despite the availability of a safe and effective vaccine to prevent HPV-attributed cancers since 2007, many parents choose not to vaccinate. All counties in Colorado are below the Healthy People 2020 (HP2020) goal of 80% vaccination coverage.

Goals/Objectives: The goal of the Alliance for HPV Free Colorado (Alliance) is to increase HPV vaccination rates among 11-17 year olds in a 15 county region in northeastern Colorado. The objectives are: 1) Utilize community and partner stakeholders, including an HPV Advisory Council, to promote and champion HPV vaccine education and awareness, 2) Plan and implement a health systems approach to increase adherence to national HPV vaccination guidelines among practices and providers, 3) Adapt and implement a targeted communications plan to increase demand for and knowledge about the HPV vaccine among parents/guardians of adolescents aged 11-17 years, and 4) Utilize sustainable analytic infrastructure to measure HPV vaccination rates.

Practice Implemented: DPH partners with eight local public health agencies (LPHA) representing urban, rural, and frontier populations in Adams, Arapahoe, Boulder, Broomfield, Denver, Douglas, Jefferson, Larimer, Logan, Morgan, Philips, Sedgwick, Washington, Weld, and Yuma counties. The Colorado Children's Immunization Coalition (CCIC), American Cancer Society (ACS), and the Adult and Child Consortium for Health Outcomes Research and Delivery Science (ACCORDS) at the University of Colorado are additional key Alliance partners.

The Alliance is funded under the Clinical Systems Quality Improvement strategy of the Colorado Cancer, Cardiovascular, and Chronic Pulmonary Disease Grants Program (CCPD). The current project is an expansion of a previously funded CCPD program, the Denver Metro Alliance for HPV Prevention (Metro Alliance), to increase vaccination rates in five metro Denver counties. The Alliance uses a results-based accountability model to link population-level goals to programmatic-level goals by working collaboratively to increase HPV vaccination rates and eliminate HPV-associated cancers.

Results: Dissemination and Implementation Science (D&I) frameworks help the Alliance to plan, implement, and evaluate project outcomes. Expected outcomes for the program include:

  • High collaboration and alignment of HPV vaccination activities across Colorado
  • Standard work for improving HPV vaccination processes across health systems
  • Increased understanding of the importance of the HPV vaccine and readiness to promote it among providers, stakeholders, and parents
  • Coordinated messaging to parents/guardians  of 11-17 year old boys and girls that the HPV vaccine is cancer prevention
  • Expansion of technical infrastructure to receive, process, and store the Colorado Immunization Information System (CIIS) data of all 15 counties

Were all objectives met? The Alliance is currently in year two of a three year grant period, and year five of six of the program overall. Process measures from all components of the program are continuously measured. Final outcome measures will not be known until the end of the project period, June 30, 2021.

Success Factors: Stakeholder collaboration is the most important factor leading to the success of this program. The Alliance is not able to conduct all the work needed to eliminate HPV-associated cancers or diseases; this work is a shared responsibility, and as such, we have developed a robust collaborative of stakeholders across Colorado to help plan, implement, and evaluate our work. The Alliance effectively leverages relationships and engages the community to ensure that materials developed and practices implemented meet the needs of our diverse audiences.

Public Health Impact: The current Alliance project region includes 70% of Colorado teens ages 11-17. Alliance interventions have the potential to impact over 350,000 individuals. HPV vaccination is a simple, effective intervention that has great public health impact; recent evidence indicates that one case of HPV-associated cancer is prevented for every 202 people vaccinated.

If there were a vaccine against cancer, wouldn't you get it for your kids? An effective and safe vaccine against HPV has been available since 2007 in the United States, yet in Colorado only 53.8% of teens 13-17 are fully protected against the virus (NIS-Teen 2017). Due to issues related to access, vaccine hesitancy, and safety concerns, uptake of the HPV vaccine has been less than optimal.

HPV infection causes cervical, vaginal, and vulvar cancers in women; penile cancers in men; oropharyngeal cancers, anal cancers, and genital warts in both men and women (CDC 2017). The full benefits of vaccination cannot be realized until higher coverage levels among adolescents are achieved. With more than 80% of sexually active adults infected with HPV by age 50, HPV vaccination is a cost effective public health intervention (Chesson et al 2012). Widespread use of the vaccine would save billions of healthcare utilization dollars related to the management of HPV-related diseases (Chesson 2012 & Durham et al 2016). Recent evidence presented to the Advisory Committee on Immunization Practice (ACIP) indicates that for every 202 vaccines given one case of HPV-attributable cancer is prevented (Chesson 2019).

In Colorado, various immunization and cancer inequities exist. HPV-associated cancers disproportionately affect certain Colorado communities at rates higher than the national averages (SEER 2016), and the state has immense differences in HPV vaccine coverage, with rural and frontier counties generally having lower HPV vaccine series completion rates than urban counties (CIIS 2016). Only 55% of Colorado children 13-17 who live in a non-metropolitan statistical area are up-to-date with their HPV vaccine, compared to 62% of children in a metropolitan central city (NIS-Teen 2017). In time, these geographical gaps will translate into disparate cancer rates.

The Alliance for HPV Free Colorado (Alliance) began in 2015 as the Denver Metro Alliance for HPV Prevention (Metro Alliance), a five county Denver metro initiative to increase HPV vaccination rates. The Metro Alliance worked to increase demand for and knowledge of the HPV vaccine among parents/guardians of 11-17 year olds in Adams, Arapahoe, Denver, Douglas, and Jefferson counties. Similar to the Alliance, the Metro Alliance worked to increase vaccination rates through four key initiatives.

The Metro Alliance worked with individual clinical practices on a six month intervention to improve processes and increase HPV vaccination rates among their adolescent populations. During the practice intervention, the Metro Alliance worked with 19 different practices that cared for a total of 24,257 adolescents 11-17 years old. At the end of the intervention period, 98% of providers surveyed said they were likely to continue the HPV vaccine prioritization efforts in their clinics and 84% reported that giving a strong provider recommendation for the HPV vaccine was useful in increasing their vaccination rates.

The Metro Alliance developed and implemented a targeted communications campaign aimed at parents/guardians of 11-17 year olds to increase their knowledge of the HPV vaccine and drive them to talk to their child's provider about the vaccine. The campaign served 119,561,586 digital impressions when it ran in the spring of 2018. A telephone survey showed that 60% of parents surveyed recalled the Metro Alliance advertisements and those who did were more likely to express intention to have their child vaccinated against HPV. 

An Advisory Council of stakeholders was developed to champion HPV vaccine education, promotion, and awareness by aligning goals and strategies related to HPV vaccination across the Denver metro region. The Denver Metro Advisory Council has continued and expanded throughout the past four years.

An analytic infrastructure, the Immunization Business Intelligence System (IBIS) was developed by DPH to measure HPV vaccination rates. IBIS uses a regional data-sharing approach whereby patient data are uploaded, aggregated, and analyzed in a way that aligns program metrics and population outcomes. Agreeing to uniform approaches surrounding data collection and measurement allowed the Metro Alliance to track improvements, identify areas for interventions, and adapt the tool as needed.

The Alliance is currently a collaborative of multidisciplinary stakeholders that has expanded to a 15 county region across northeastern Colorado. These counties are: Adams, Arapahoe, Boulder, Broomfield, Denver, Douglas, Jefferson, Larimer, Logan, Morgan, Philips, Sedgwick, Washington, Weld, and Yuma. The Alliance work is facilitated by Denver Public Health (DPH) and includes partners from eight local public health agencies (LPHA), the Colorado Children's Immunization Coalition (CCIC), the Adult and Child Consortium for Health Outcomes Research and Delivery Science (ACCORDS) at the University of Colorado, and various stakeholders in the community. The Alliance aims to implement a multidisciplinary, socio-ecological approach that results in a sustainable increase in completion of the HPV vaccine series for 11-17 year olds. The socio-ecological model helps to describe the interconnected relationships of personal and environmental factors that impact health (CDC) and addresses several individual, community, health system, and environmental barriers to increase vaccination rates. All counties in the Alliance region are well below the Healthy People 2020 (HP2020) goal of 80% vaccination coverage.

The Centers for Disease Control and Prevention (CDC) recommend routine vaccination of all adolescents against HPV at 11-12 years old, but the series can begin as early as age nine and as late as age 26 for routine vaccination. The Alliance focuses on vaccination for 11-17 year old boys and girls in the 15 county region. The region is home to approximately 344,910 individuals who make up 69% of all Colorado teens ages 11-17 and 6% of the total Colorado population (Colorado Department of Local Affairs 2016). More than one third of these teens are Hispanic, accounting for 119,994 of the adolescents in the target age group.

Over the past few years, Colorado has made progress in coordinating efforts to achieve the HP2020 goal of 80% vaccination coverage for teens. However, there is still significant room for improvement to align LPHA efforts to improve HPV vaccination rates through education and advocacy with healthcare systems, Regional Accountable Entities (RAEs), dental systems, youth and parent serving organizations, and other stakeholders that address immunizations and cancer prevention.  Around the time the Metro Alliance began strategic planning for future grant cycles, organizations both in Colorado and nationally started moving toward health systems approaches to improve population health. For example, the Denver Metro Partnership for Health (Partnership) formed as a collaborative of LPHAs serving seven counties. The Partnership teams up with regional leaders in health care, human services, behavioral health, environment, philanthropy, education, business, local government and others to achieve its goals of promoting health and well-being across the region.

Previously, the Metro Alliance strategy to work with individual clinics and providers proved to be time consuming and difficult to change overall vaccination rates. Therefore, starting in 2018 the Alliance adopted a health systems approach to impact more adolescents with vaccination while requiring less direct support from Alliance staff.

The current practice of the Alliance is a combination of evidence-based interventions (EBI) described in The Community Guide and by the HPV Vaccination Roundtable. The practice has the potential to impact countless individuals because of the focus on all levels of the socio-ecological model and a top-down approach to increasing vaccination rates within health systems. The HPV Vaccination Roundtable, developed by the American Cancer Society and funded by the CDC, created an action guide with top evidence based practices for large health systems to improve HPV vaccination rates. The five action steps for health systems described in the action guide are incorporated into Alliance interventions with health systems.

The Community Guide provides evidence that interventions, such as education campaigns coupled with health systems and provider-focused work, are warranted to increase overall rates of vaccination for all adolescents, regardless of race, gender, or geography, across the state. Physicians', nurses', and healthcare partners' strong recommendation for the vaccine using a presumptive, announcement approach is the most effective strategy to increase HPV vaccination rates (Gilkey, Calo, Moss et al 2013). Improving clinical process and increasing awareness of the consequences of not being vaccinated are evidence-based approaches proven to decrease vaccine disparities (Farmar, Love-Osborne & Chichester 2016). The Alliance uses these innovative strategies in combination to address barriers to vaccination to ensure that all Coloradans are protected against HPV and related diseases.

Community and partner collaboration are integral to the success of the Alliance for HPV Free Colorado (Alliance) program. The Alliance aims to collaborate with stakeholders at every level of the socio-ecological model in order to affect change and increase HPV vaccination rates. In order to achieve the goals and objectives, a project structure was created with committees and workgroups dedicated to working toward each of the objectives.

Through a strategic planning process in the fall of 2018, Alliance members created a mission and vision for the project. The mission is that the Alliance will foster alignment with the community, providers, and partners to ensure that all Coloradans are vaccinated against HPV and has a long-term vision that all Coloradans are protected against HPV-related cancers.

The Alliance is funded through the Cancer, Cardiovascular, and Pulmonary Disease (CCPD) grants program at the Colorado Department of Public Health and Environment (CDPHE). Funding comes from Colorado Amendment 35 tobacco tax revenue. The Denver Metro Alliance for HPV Prevention (Metro Alliance) was also funded by the CCPD grants program from July 2015-June 2018. The money spent in the first year of the Metro Alliance went toward start-up costs and planning for the practice in the five county region. The overall budget for Year 1 of the Metro Alliance was $875,895. Although a significant amount of the work from the Metro Alliance continued into the 15 county Alliance program, much of the first fiscal year of the Alliance budget was also used to plan, expand, and adapt the program for the 15 counties. We estimate that approximately 40% of this first year of the 15 county Alliance grant cycle funding ($1.2 million) was used for planning and start-up costs. Within the objectives below are the activities that DPH and the Alliance completed from June 2018 to December 2019.

Objective 1: Utilize community and partner stakeholders, including an HPV Advisory Council, to promote and champion HPV vaccine education and awareness.


  1. Plan and convene quarterly Advisory Council meetings.
    1. An Advisory Council was developed by the Metro Alliance in 2015 and has expanded in the current grant cycle. The main goals of the Advisory Council are to develop and foster connections between community organizations and LPHAs and for members to serve as HPV vaccination champions in their communities.
    2. DPH partners with CCIC to organize quarterly meetings, recruit speakers, and plan activities.
    3. Speakers are invited to present on relevant HPV topics to support Advisory Council members in serving as HPV vaccine champions.
    4. Members are asked to provide input on products and tools created by the Alliance committees.
    5. A meeting satisfaction survey is administered at the end of each meeting that helps inform the planning of upcoming meetings. It includes suggestions for meeting topics, speakers, and individuals/organizations to partner with to expand the reach of the Alliance Advisory Council.
    6. An engagement survey is conducted at the end of each fiscal year to collect demographics of all Advisory Council members (e.g., region and populations served). The engagement survey aims to gather feedback from members who are both able and unable to attend individual quarterly meetings to help improve the experience for everyone involved in the project.
  2. Disseminate information from quarterly Advisory Council meetings.
    1. Advisory Council members are extensions of the community who provide a diverse and pragmatic perspective on current HPV prevention issues.
    2. Advisory Council planners ensure that members are up to date on the newest information, recommendations, and resources.
    3. Members are expected to:
      1. Serve as HPV vaccine champions for friends, family, colleagues, peers, and other community members.
      2. Disseminate project materials as appropriate.
      3. Share lessons learned related to HPV vaccination and cancer prevention to their networks.
      4. Ensure that messaging is conveyed equitably across the region.
      5. Assist in recruitment of new Advisory Council members.

Objective 2: Plan and implement a health systems approach to increase adherence to national HPV vaccination guidelines among practices and providers.


  1. Research best practices for improving HPV vaccination rates in health system and dental settings.
    1. Alliance committees conducted literature reviews to create a list of evidence-based interventions (EBIs) and best practices for health systems and dental settings.
    2. The Metro Alliance previously focused on increasing HPV vaccination rates from the clinic and provider level through education trainings and a six month practice intervention. This strategy was time consuming and could only reach a small number of providers. The Alliance has elevated the interventions to a health system level to use a top-down approach to increasing vaccination rates.
    3. DPH partner, Larimer County Department of Health and Environment (LCDHE), previously conducted formative research to understand the role of dental professionals in oral cancer screening, cancer prevention education, and approaches to recommending the HPV vaccine to patients. Their research identified a need for more education for dental providers on recommending the vaccine and varying oral cancer screening protocols. The Alliance has continued this work to create protocols, toolkits, and educational materials specific to dental providers and has partnered with dental providers in the community to assist in the development of these materials.
  2. Create an outreach roadmap for identifying health systems to engage and gain support for promoting HPV vaccination.
    1. Using a Dissemination and Implementation Science (D&I) framework, the Alliance created an outreach roadmap to identify potential health systems with which to collaborate. Guided by the  RE-AIM evaluation framework (see Evaluation section for description of the tool), the Alliance developed a scoring rubric with guiding questions to gauge whether a health system was ready to work on an HPV vaccine quality improvement initiative
    2. Feedback was gathered from stakeholders, LPHAs, and American Cancer Society partners on how best to engage and work with health systems. Leveraging existing relationships with health system personnel was identified by committee members as the first step in engagement efforts.
    3. During discovery meetings, the Alliance evaluated a health system based on current relationships, priority alignment, and level of readiness to participate in a project.
  3. Develop an implementation plan for working with health systems.
    1. The Alliance created an implementation plan using the EBIs identified through research. The implementation plan is used as a guide for choosing an EBI, creating action and sustainability plans, completing regular check-in meetings with the Alliance facilitator, and measuring progress.
    2. Health systems are chosen to participate in the 12 month intervention based on their readiness from the outreach roadmap questions. Once a health system and the Alliance agree to collaborate on a project, a Memorandum of Collaboration is signed to begin the project.
  4. Create a sustainability plan with health system to continue work after project completion.
    1. A main goal of the Alliance intervention projects is to increase HPV vaccination rates after the Alliance intervention is completed. As part of the implementation plan, systems are established to create a sustainability plan that continues the work after the 12 month intervention period with the Alliance. The sustainability plan outlines activities that will be continued and the responsible party.
  5. Develop educational materials and update existing materials.
    1. The Metro Alliance created educational trainings that were used to train providers on HPV 101 and Motivational Interviewing for vaccine conversations. The Motivational Interviewing training slides were adapted from ACCORDS research on motivational interviewing for HPV vaccine conversations.
    2. The DPH HPV nurse created the HPV 101 slides to give trainees an overview of the virus, disease and related cancer epidemiology, and the HPV vaccine with evidence provided by the CDC. Over time the training has been adapted to incorporate a more global perspective on HPV vaccination and tips for providers to recommend the vaccine as well as strategies to discuss the HPV vaccine with hesitant parents. The slides are updated regularly as new data becomes available.
    3. All workgroup members have been trained on HPV 101 and Motivational Interviewing to train providers across the regions.
  6. Implement a model for HPV-related oropharyngeal cancer prevention in dental settings.
    1. LCDHE and dental partners developed and adapted educational training materials to be specific to the dental setting. A protocol and toolkit were developed for a dental practice intervention and training slides were developed that are specific to dental health professionals. Trainees learn about HPV, risk factors for oral cancer, the HPV vaccine, and a model for dental providers to give a strong provider recommendation.
    2. Private, public, and Federally Qualified Health Center (FQHC) practices in the region were recruited to participate in a six week intervention using the tools and resources developed by the Alliance.
  7. Disseminate and implement HPV-related oropharyngeal cancer prevention model to dental providers and other appropriate professionals.
    1. The dental pilot program will conclude in January 2020. With evaluation plan results and lessons learned, the Alliance will develop a wider dissemination plan to increase the impact that the dental intervention can have on HPV vaccination rates in the region.

Objective 3: Adapt and implement a targeted communications plan to increase demand for and knowledge about the HPV vaccine among parents/guardians of adolescents aged 11-17 years.


  1. Collaborate with Public Information Officers (PIO) and vaccine expert partners across the region.
    1. DPH partners with PIO representatives from each LPHA, representatives from the Colorado Children's Immunization Coalition (CCIC), and researchers from the Adult and Child Consortium for Health Outcomes Research and Delivery Science (ACCORDS) to serve as the Communications Committee and execute communications plan activities.
  2. Review existing creative materials and assess applicability to the 15 county region.
    1. In 2015, the Metro Alliance adapted the CDC's HPV vaccination messaging and materials for HPV vaccination as the basis for the Alliance's campaign and creative materials. DPH and partners conducted a literature review and focus groups to inform any messaging changes, preferred dissemination outlets and visual changes that would make the CDC work more applicable to metro Denver audiences. Through a request for proposal process, Peri Marketing & Public Relations (Peri) was contracted to create campaign materials from the findings of the focus groups and literature review. Peri also used market research to determine the most impactful dissemination outlets to reach the target audience of mothers with adolescents 11-17 years.
    2. The first Denver Metro paid media buy ran from February-April and July-September 2017. The advertisements were placed on broadcast radio, Pandora, digital sites, social media, and out of home locations. The campaign ran again between February-May 2018.
    3. The Alliance continued to use the same advertisements in 2019 in the expanded 15 county region. Peri developed a communications plan to reach the populations in all 15 counties that included varying dissemination outlets depending on location. A paid media buy ran from February-May 2019.
  3. Review HPV research articles and identify best-practices for effective communications strategies to reach rural and Hispanic/Latino audiences.
    1. The Alliance conducted two additional literature reviews in the fall of 2018 to gather information about the campaign's applicability to the expanded 15 county region with rural/frontier and Spanish-speaking populations.
    2. DPH and partners used results from the literature reviews to develop the questions used in the focus groups intended to gather more in-depth information.
  4. Manage execution of key informant interviews and/or focus groups with targeted audiences.
    1. A rural/frontier focus group was conducted in June 2019 in Fort Morgan, Colorado using the interview guide developed from the results of the literature review. Participants were recruited through an existing Community Advisory Board (CAB) in Northeastern Colorado and flyers placed in LPHA office locations.
    2. Spanish focus groups were held in two Denver suburbs with a Spanish-language facilitator using the interview guide developed from the results of the literature review. Participants were recruited from existing community groups and through flyers placed in LPHA office locations.
    3. Focus group participants were incentivized with $30 Amazon gift cards and the Alliance provided child care during the focus group sessions.
    4. The focus groups were recorded, transcribed and translated, where applicable.
  5. Develop creative materials needed to execute an integrated communications plan and utilize new outlets to reach rural/frontier and/or Hispanic/Latino audiences.
    1. Recommendations for adaptation based on the literature review and focus groups were discussed and prioritized by Alliance PIOs and communications partners in August 2019.
    2. LCDHE is adapting existing campaign creative for digital and print outlets based on the recommendations from the committee. This includes adding pictures of slightly older, more diverse children and families, making background scenes more applicable to different populations across the state, and translating the call to action into Spanish.
  6. Oversee implementation of communications buy with Peri.
    1. Since they were first contracted in 2016, Peri has worked to create and execute a paid media plan that addresses the overall strategy, placements, and cost for each year of the media buy. Alliance PIOs and partners review the plan before it is executed by Peri.
    2. The targeted advertisements appear on users' pages that are likely to have children in the target audience age range. Audio advertisements are placed on top radio stations and Pandora. Social media paid advertisements appear through the DPH Facebook Exchange account and are seen on Facebook, Instagram, and Twitter. Additionally, Peri worked with Comcast to place short 15-second advertisements shown on cable and in-home programming. Media outlets are chosen to ensure that advertisements are seen or heard in all of the 15 counties.
    3. Alliance PIOs, partners, and Peri monitor the media buy budget throughout the campaign. Peri works to increase value added with additional posts and spots at no additional cost. Peri provides the Alliance with monthly reports on the media buy that include impressions and value added, which allows the media buy to be adjusted if needed.
  7. Coordinate and track earned media and social media from LPHAs.
    1. In addition to paid media, the Alliance also uses social and earned media to increase awareness for the HPV vaccine without spending additional money on advertising. Social media posts and messaging were developed by Alliance PIOs and partners; LPHAs use their own social media accounts to post content consistent with campaign messaging and creative. The Alliance gains earned media exposure through outlets such as radio interviews, professional newsletters, and op-ed articles in newspapers.
  8. Build a group of HPV vaccine community influencers and advocates.
    1. To support the overall goal of the communications committee via grassroots advocacy, the committee was tasked with recruiting a group of community and parent vaccine spokespeople/influencers and training them in HPV 101 (including the basics of HPV, transmission and the importance of vaccination to prevent cancer) and vaccine communications best practices. The committee works to integrate members of the group, HPV Free COmmunity Advocates, into the Alliance's communications campaign by promoting peer-to peer learning and providing opportunities to promote the HPV vaccine in their communities via social media, media interviews, letters to the editor, blog posts and attendance at local community and health events.
    2. The Alliance recruited existing HPV vaccine advocates to join the HPV Free COmmunity Advocate group based on partners' existing relationships with these advocates. Additional members were recruited through social media, flyers, word of mouth advertising, and through marketing to existing organizations such as Cervivor, a non-profit for cervical cancer awareness and support.
  9. Support Alliance efforts in creating and executing communications materials for health systems.
    1. Health systems working with the Alliance have the option to conduct evidence-based interventions (EBI) related to communications campaigns in their system. These campaigns may target providers or the patient population. As experts in communication, the Alliance PIOs and partners work to create these materials specific to health system use.
  10. Maintain an up-to-date webpage,, to serve as a landing page for digital advertisements and a resource hub for parents, providers and stakeholders.
    1. The Alliance webpage is hosted on the CCIC website. It includes information about the HPV vaccine and its safety and efficacy, as well as links to sites with more information. The webpage serves as a landing site for all Alliance advertisements.
    2. In 2020, the webpage will be translated into Spanish and links to Spanish resources will be included.
  11. Develop evaluation plan for the communications campaign.
    1. A post-campaign evaluation was conducted in 2018 to evaluate the effectiveness and impact of the campaign in the five-county region. See Evaluation section for results. 
    2. With DPH epidemiologists and ACCORDS researchers, the Alliance has developed an evaluation plan for the campaign in the 15 county region. A contractor will be hired to conduct approximately 750 telephone surveys. A pre-evaluation will be completed before the campaign launch in April 2020 and a post-evaluation will be conducted in September 2020.
    3. Earned and social media are evaluated by each LPHA and partner to assess for reach, clicks, and reactions to posts.

Objective 4: Utilize sustainable analytic infrastructure to measure HPV vaccination rates.


  1. Sign data use agreement (DUA) with all 15 counties to allow DPH to review all immunization data.
    1. As of February 21, 2019, DPH has a fully executed DUA with all LPHAs, partners, and the Colorado Department of Public Health and Environment (CDPHE) and partners to pull immunization data from CIIS.
  2. Update the Immunization Business Intelligence System (IBIS) and ensure vaccination rates are correct.
    1. IBIS was created by DPH epidemiologists to assess vaccination rates for the Metro Alliance.
    2. Vaccination rates are pulled from the Colorado Immunization Information System (CIIS) via IBIS. IBIS is and now able to consume IIS data from 15 counties, assess the validity of each vaccine, assign each patient an up-to-date status for each vaccine, and visualize population and practice level up-to-date rates.
    3. Data is received monthly from CIIS, and is processed, geocoded, and cleaned. A user then creates a ‘batch' of data to be processed based on a selected evaluation date. Vaccines are processed by the Immunization Calculation Engine (ICE) ™, an open-source web application that evaluates vaccine validity. Finally, ICE-ed data is loaded into a reporting database and up-to-date rates are visualized using a series of Tableau dashboards.
    4. The Alliance Steering Committee decided to use vaccination rate definitions consistent with the CDC and CDPHE. HPV vaccine series initiation is defined as having one dose in the series. HPV vaccine series up-to-date (UTD) is defined as having two doses of the vaccine if initiation began before the patient's 15th birthday and three doses of the vaccine if initiation began after the patient's 15th birthday.
  3. Assess CIIS penetration rates.
    1. The Metro Alliance conducted a penetration analysis to assess how many providers in our region reported vaccinations to CIIS. The analysis showed that approximately 60% of pediatricians and 40% of family medicine providers reported to CIIS in the five county Denver Metro region.
    2. Another penetration analysis has not been conducted in the 15 county region because of the difficulty of getting an accurate denominator.
  4. Pull and use data from IBIS to inform project activities.
    1. Immunization data is downloaded from the CDPHE database. DPH submits the batch of data through an ICE web portal that determines if vaccine doses are valid. The validated data is then uploaded onto Tableau servers.
    2. DPH epidemiologists created a Tableau dashboard to visualize vaccination rates for all 15 counties.

Using a socio-ecological model, the Alliance for HPV Free Colorado (Alliance) aims to address several individual, community, health system, and environmental barriers to increase vaccination rates among 11-17 year olds. As such, we have coordinated a multidisciplinary and multifactorial effort to promote evidence-based strategies, promising practices, and program specific roles to ensure a logical connection between population goals and program-level goals. The Alliance objectives are: 1. Utilize community and partner stakeholders, including an HPV Advisory Council, to promote and champion HPV vaccine education and awareness, 2. Plan and implement a health systems approach to increase adherence to national HPV vaccination guidelines among practices and providers, 3. Adapt and implement a targeted communications plan to increase demand for and knowledge about the HPV vaccine among parents/guardians of adolescents aged 11-17 years, and 4. Utilize sustainable analytic infrastructure to measure HPV vaccination rates. An evaluation plan was developed in the first year of the grant cycle that assesses process and outcome measures from all aspects of the project. The Alliance is on track to meet all of its objectives; however, outcome metrics are not yet available.

With the socio-ecological model as our base, the Alliance uses dissemination and implementation (D&I) science grounded frameworks to plan, implement, and evaluate project outcomes. D&I helps bridge research and practice and is a study of methods to promote the adoption and integration of evidence-based intervention into routine health care to improve the impact on our population health outcomes. RE-AIM is one D&I tool the Alliance uses to ensure our evaluation plan measures appropriate outcomes for various stakeholders and audiences. RE-AIM stands for Reach, Effectiveness, Adoption, Implementation, and Maintenance, and aims to increase external validity and sustainability of programs. The following questions helped guide us through strategic planning, implementation, and evaluation phases of the project:

Reach: Who is intended to benefit and who actually participates or is exposed to the intervention?

Effectiveness: What are the most important benefits you are trying to achieve and what is the likelihood of negative outcomes?

Adoption: Where is the program or policy applied?

Implementation: How consistently is the program or policy delivered, what adaptations to the original plans were made, and how much did it cost?

Maintenance: When did the initiative become fully operational, how long do the results last and how long will the initiative be sustained?

Objective 1: Utilize community and partner stakeholders, including an HPV Advisory Council, to promote and champion HPV vaccine education and awareness.

The Alliance hosted three Advisory Council meetings in the first fiscal year of the grant. Between 30 and 40 participants attended each of the three meetings online and in-person. In November 2018, the group brainstormed barriers to HPV vaccination that helped inform the strategic plan for the Alliance. During the January and May meetings, guest speakers provided educational sessions, participants engaged in networking activities, and updates were provided from each Committee and Workgroup.

At the end of each Advisory Council meeting, participants complete a survey to assess goals and productivity of the meeting, as well as to provide suggestions for future meeting topics and additional partners to recruit into the Advisory Council. The Alliance tracked three questions for qualitative trends during fiscal year 19: goals of the meeting, level of participation, and overall satisfaction with the meeting. Overall, an average of 98% of participants reported being satisfied or very satisfied with meetings, 90% reported excellent or good participation at meetings, and 97% reported that the goals of the meeting were excellent or good.

In June 2019, the Alliance created and distributed an engagement survey to send to the listserv. The survey was designed to understand the demographics, barriers, and motivators of both those who attended and those who were unable to attend quarterly meetings. Email addresses were removed from the listserv of those that could not be delivered and for those who were no longer involved in the project. In total, 120 individuals received the survey and 46 responded, giving a response rate of 38.3%. The most common professions among Advisory Council members are public health professionals, nurses, and researchers and 45% serve in the Denver Metro region. Survey respondents identified educational sessions and project updates as the preferred meeting content. They believe they can best support the Alliance by continuing to attend quarterly meetings to stay informed on the project.

Objective 2: Plan and implement a health systems approach to increase adherence to national HPV vaccination guidelines among practices and providers.

The first fiscal year of the grant was spent planning and developing the health systems approach to increasing vaccination rates. The first health system to agree to work with the Alliance signed a Memorandum of Collaboration (MOC) in September 2019. To date, six health systems have been engaged and are beginning the process of implementing a year-long quality improvement project, far exceeding our original goal to engage two health systems in the first year of implementation. The Alliance does not have process or outcome measures for work with health systems yet. A variety of data collection tools and methods to evaluate the intervention process and outcome metrics will be used. These include:

  • A health system survey will be completed quarterly during the intervention period by leadership, HPV vaccine champions, implementers, and the Alliance points of contact. The survey assesses the usefulness of the evidence-based interventions (EBI), time spent implementing EBI, and attitudes among health system providers on value of HPV vaccine.
  • Qualitative interviews will be conducted twice during the intervention period: once six months after baseline and again at project completion. The qualitative interviews collect feedback from at least one representative in leadership, HPV vaccine champions, implementers, and the Alliance points of contact to assess the use of the implementation plan, institutionalization of HPV vaccination best practices, and use of communications materials.
  • Initiation and up-to-date (UTD) HPV vaccination rates for 9-17 year old boys and girls in the health system will be reported at baseline and quarterly throughout the intervention year.

Between July 2018 and December 2019, two train-the-trainer sessions, five HPV 101 training sessions, and seven Motivational Interviewing training sessions were completed. In the train-the-trainer sessions, participants were trained to lead HPV 101 and Motivational Interviewing trainings to others. In addition to training Alliance workgroup members, CDPHE and American Cancer Society representatives were trained so that we are able to conduct trainings outside of the 15 county Alliance region. Participants complete pre- and post-training surveys to assess knowledge change from the training sessions.  Pre- and post-training survey results indicate that knowledge, skills, and attitudes toward HPV and HPV vaccination improved among trainees and that they felt more confident in understanding the importance of the vaccine. Among providers trained in the Metro Alliance period, there was a 20% improvement in confidence of using motivational interviewing skills when discussing HPV vaccination with hesitant parents. The Alliance will continue to analyze survey results from the HPV 101 and Motivational Interviewing training sessions to assess the knowledge, skill, and attitude change among trainees and improve the content of the training sessions.

The Alliance has engaged three dental practices for pilot projects in 2019. The dental pilot process is ongoing and the evaluations have not been completed. The pilot process facilitators will assess the following indicators through a combination of dental provider surveys and qualitative interviews:

  • Percent of eligible patients in the clinic population prompted to get HPV vaccination as a result of the practices implemented.
  • Value dental providers place on the HPV vaccine compared to other adolescent preventive dental health needs.
  • Confidence among dental providers to recommend the HPV vaccine.
  • Confidence among dental providers to discuss HPV infection as a risk factor for oropharyngeal cancer.
  • Time spent implementing the protocol.
  • Usefulness of the protocol.
  • Dental professionals reporting using communication materials for HPV education and awareness.
  • If and how HPV vaccine best practices are sustained.

Objective 3: Adapt and implement a targeted communications plan to increase demand for and knowledge about the HPV vaccine among parents/guardians of adolescents aged 11-17 years.

The Denver Metro Alliance for HPV Prevention (Metro Alliance) evaluated the communications campaign via 454 telephone surveys of parents with children living in their home ages 9-17 between September 28 and October 17, 2017, after that media buy had concluded. The survey asked parents to recall their knowledge, attitudes, and beliefs about the HPV vaccine from the fall prior to the campaign and compare them to the point in time after the campaign was complete. Among participants, 19% reported more positive attitudes toward HPV vaccination than the year prior. Of those who recalled the HPVFreeCO advertisements, 24% reported having more positive attitudes toward HPV vaccination and were 50% more likely to report that they planned to have their child vaccinated over the previous year.

A paid media campaign, conducted by Peri Marketing and Public Relations (Peri), totaling $295,000 was implemented in the 15 county region from January 29-May 31, 2019. The advertisements ran on Comcast Always On, radio, and digital (Pandora, Facebook, Mobile In-App, Mobile browser) outlets. Peri monitored the progress and metrics of the buy including impressions, click-through rates, and value added. The campaign planned for 34,000 Comcast spots and received over 10,000 additional spots, valuing an additional $20,539 in free media exposure. The online campaign delivered 4.4 million additional impressions, equating to $6,705 in additional value for the campaign. In total, the HPVFreeCO campaign served 90,982,355 impressions over the four month time period it was in market.

In the spring and fall of 2020, the Alliance will conduct a pre/post evaluation of the adapted campaign with parents of 11-17 year olds in the 15 county region to assess recall, demand, knowledge, and attitude change towards the HPV vaccine. Earned and owned social media metrics are reported by individual county health departments and rolled up into a single evaluation. Website metrics are gathered monthly from Google Analytics. During the time the HPVFreeCO campaign was in the market, the traffic to the HPVFreeCO landing page increased significantly. Individuals spent 20 seconds on average on the webpage and there were 413,681 total sessions among 198,406 unique users from January 29-June 1, 2019.

Objective 4: Utilize sustainable analytic infrastructure to measure HPV vaccination rates.

Using IBIS, the Alliance is able to see vaccination rate trends over time, which can be utilized to inform decision making. IBIS allows DPH to stratify vaccination data by age, sex, ethnicity, and location. In June 2018, the initiation rate for HPV vaccination was 59.6% for 11-17 year old boys and girls in the 15 county region and the UTD rate was 38.5%. After 15 months of interventions in September 2019, the initiation rate was 62.3% and the UTD rate was 41.5% in the region. Overall, since the project initiation there has been an 8% increase in UTD rates and a 4.5% increase in initiation rates. This equates to approximately 17,000 more adolescents who have initiated the HPV vaccine series and 31,000 more adolescents who are UTD.

To begin the sustainability planning process, the Alliance used the Program Sustainability Assessment Tool (PSAT) that was developed by the Center for Public Health Systems Science at the Washington University in St. Louis. The tool has been tested for reliability and can be used by program to understand and plan for their sustainability. Often, the first thing that comes to mind when thinking about sustainability is funding. Funding stability is important; however, it is not the only aspect that should be considered in program sustainability planning. The PSAT helps users to understand its strengths and weaknesses of sustainability capacity across eight different domains: environmental support, funding stability, partnerships, organizational capacity, program evaluation, program adaptation, communications, and strategic planning. Upon completing the assessment, PSAT compiles scores from individual assessments into a group report. This report served as a framework for the Alliance Steering Committee's discussion regarding areas of focus for our sustainability plan.

Members of the Denver Metro Alliance for HPV Prevention (Metro Alliance) Steering Committee completed the PSAT towards the end of the third grant year, in April 2018, to inform planning of year one of the Alliance. The Steering Committee discussed PSAT results, future considerations for each of the domains, and prioritized the following action items to be applied in addition to committing to conducting the PSAT at the end of each year of the new grant cycle:

  • Strategic planning: The Alliance committed to clarify the project's logic model and identify appropriate outcome assessment measures. At the beginning of the second grant cycle, each project area worked on individual logic models that roll up into an encompassing project strategic plan.
  • Partnerships: Alliance decision makers discussed that partnerships could be more strategic by having specific goals related to specific partners. This has been executed in partnerships with CCIC, ACS, and other strategic partners of the Advisory Council.

As planned, key members and decision makers from the Alliance began sustainability discussions and planning in the spring of 2019 by taking the PSAT. Some individuals took the assessment for the second time, while new partners from the expanded region took it for the first time. Current areas of strength for the Alliance are program adaptation, program evaluation, organizational capacity, and environmental support. The four lowest scoring domains - funding stability, partnerships, strategic planning, and communications - were discussed in depth. The action steps identified were to focus on improving partnerships, community engagement, and communications through the Advisory Council and the HPVFreeCOmmunity advocate group.

Lessons learned from the Metro Alliance led to the creation of health system interventions to promote HPV vaccination. Each health system working with the Alliance creates a sustainability plan as part of the year-long intervention. A main focus for the interventions is to create structures and processes that continue after the Alliance project is complete. All of the evidence-based interventions   the Alliance offers are meant to be easily maintained after the initial work to get them initiated. Each will require some maintenance, but they should be fairly simple to continue. Health systems are also incentivized to continue because they interventions they select are often tied to key performance indicators or other quality metrics that could affect reimbursement rates and payments. Sustainability is a consideration for dental systems and clinics that work with the Alliance as well. The dental protocol was developed to have four easy steps to follow in a dental setting to increase the likelihood of maintenance over time.

The Alliance has created a sustainable model around the HPV 101 and Motivational Interviewing education sessions by using a train-the-trainer model. Alliance partners at each LPHA, CDPHE, the American Cancer Society (ACS), and on the Colorado Cancer Coalition HPV Task Force (CCC HPV TF) have been trained to administer the sessions. Additionally, health systems are encouraged to use this model and have HPV champions able to train anyone new to the health system staff. During the funding period, CDPHE, ACS, and CCC HPV TF members are able to train providers outside of the Alliance 15 county region, while Alliance funded partners are able to train providers within the region.

To address partnerships in the Advisory Council, planners created an engagement survey. The survey was administered to collect feedback from members who are both able and unable to attend individual quarterly meetings to help improve the experience for everyone involved. We asked respondents to select ways in which they would like to support our mission and has been used to clarify member expectations and planning of future meetings. The majority of members indicated they could support the Alliance by continuing to attend meetings, helping to identify barriers, disseminating materials, providing input on strategic direction, and educating stakeholders using Alliance resources, all of which contribute to the sustainability of our project. The Alliance will continue to administer engagement surveys at the end of each fiscal year for the project period.

There is significant partner and stakeholder commitment to maintain HPV vaccination efforts in Colorado and across the United States. National organizations have made HPV vaccination a priority because of the strong impact it has on cancer prevention and population health. The 2016-2020 Colorado Cancer Plan (CCP) has a goal to increase clinical interventions to prevent cancer, one of which is HPV vaccination. Continued support to increase HPV vaccination in Colorado is warranted and the 2021-2025 CCP will maintain the objective to increase clinical interventions to prevent cancer. The Colorado Children's Immunization Coalition (CCIC) is committed to promoting vaccine uptake through education, coalition building, and public policy for individuals across the lifespan. Nationally, ACS is a partner committed to sustaining HPV vaccination efforts through their HPV Vaccinate Adolescents against Cancers (VACs) programs.

In the future, the Alliance hopes to build a statewide program to unify efforts and approaches to HPV vaccination across Colorado. For financial sustainability, the Alliance will apply for another grant through the Colorado Cancer, Cardiovascular, and Chronic Pulmonary Disease Grants Program to bring the coalition statewide. The Alliance will also consider moving to a technical assistance framework to develop contracts with health systems to support quality improvement initiatives. Funding stability discussions for the Alliance will begin in the summer of 2020 when key decision makers take the PSAT again.

Colleague in my LHD