|
| |
A New Fidelity - Process Evaluation
Michigan HIV News, Spring-Summer 2004 Issue
As prevention grant funding gets fine-tuned, providing
technical assistance and capacity development assistance to agencies around the
state becomes more important. That’s the job of Amy Peterson, who has been
honing her own skills since she started in the HIV/AIDS field 12 years ago at a
CBO in Chicago. She earned a Master of Public Health degree in 2002, which added
to her arsenal of program management and training skills that she brings to her
current position with the HAPIS Prevention Community Partnerships Unit.
My job “becomes more important because there have been so
many changes,” said Peterson. Agencies are experiencing change because change
has been happening from the top down, beginning with the CDC’s Advancing HIV
Prevention Initiative and the new requirements mandated for state health
departments. “Around the same time, CDC became much more invested in using
proven interventions,” she said.
“Because of so many changes…my role really became to bring
agencies along, not only philosophically, but also with acquiring the knowledge
and skills to be able to implement these new interventions.”
Where previously HAPIS recommended that the agencies have
protocols in place to receive funding, “We now require protocols,” said
Peterson, “ these protocols outline step by step how they will do an
intervention” with predetermined materials, curriculum and sites. “Agencies need
to be consistent in their interventions and adherence to the protocols,” and
maintain their “fidelity” when implementing the intervention.
Writing the protocols was a challenge for some of the HAPIS
funded agencies. “We had heard from the agencies that they weren’t sure how to
do that,” said Peterson and so HAPIS developed a protocols workshop, held in
May, which provided assistance to those who needed it.
What’s totally new is the concept of monitoring the process.
“Are you doing the program as it was intended?” said Peterson. “We want a more
thoughtful process of to whom and why are you providing the program…and then do
that same intervention consistently.”
The HIV Event System, an Internet-based data reporting system
that HAPIS initiated three years ago with its prevention grant-funded agencies,
is an integral part of the process monitoring process. This sounds redundant and
bureaucratic, but it’s actually a very effective way to assess what services are
being provided, to whom, and where.
Process monitoring and evaluation maintain the science in
science-based prevention. Science means having controls. Controls in behavioral
science mean consistency in program delivery and “fidelity” or being true to the
original plan and protocols.
Gone are the days of the “wild-wild west” of HIV prevention,
when everyone had great ideas and programs would shoot from the hip. Today CDC
wants to fund “interventions which are tried and true science-based prevention
programs.” HAPIS provided a presentation to Michigan agencies, on the eight or
nine “interventions in-a-box” in April 2003, prior to their applications for the
2004-2007 prevention funding cycle.
But just like a cake mix, if you don’t follow the directions
(protocols), you won’t get that picture on the box. You can’t skip the eggs and
have it taste the same.
So monitoring program fidelity is new for HAPIS this year, said Peterson. And
again, this is measured through process monitoring and process evaluation. This
is not to be confused with outcome or impact evaluation.
All of this does get confusing, so Peterson coordinated a
workshop to explain the role of process evaluation and distinguish among the
different types of evaluation. In April HAPIS sponsored “Using Process
Evaluation to Improve HIV Prevention Interventions” presented in Ann Arbor by
Peterson and David Napp, who has worked as a consultant for CDC to develop
trainings in response to CDC’s ongoing interest in developing evaluation
capacity nationally. You may view an abbreviated PowerPoint presentation (PDF
file) from
that workshop.
The second change for HAPIS in this funding cycle was
“acknowledging that our agencies by and large did not have the training or
know-how to conduct outcome monitoring,” said Peterson. So they tried to
simplify the application by requesting “what types of changes you expect to see,
what direction and what type of change you are expecting with this
intervention.” HAPIS technical assistance will then help these agencies develop
tools for outcome monitoring.
So, how’s it going so far? At the 90-day assessment, even the
“high-functioning, long standing” agencies were having a problem with curricula
and protocol development. “There was a disconnect between what HAPIS thought
agency capacity was in these areas, and the reality. So we have created tools
and training to bridge the gap...,” said Peterson. The training in May was to
focus on quality assurance and protocol development. “It has been a time of
great change and learning for both HAPIS and our contract agencies, but the
quality of HIV prevention services depends on us figuring it out together.”
|
