Share the article

Health care  –  Set up a program

Evaluate progress

Monitoring and evaluation provides insight into the impact of the rational use program or interventions. It helps to see if goals are being accomplished and can identify aspects that may need improvement, which is important in the scale-up and replication of interventions.

Outputs from evaluation can be used for evaluating interventions, revising national or local guidelines, or presenting facts for underlying policy decisions. Benchmarking between departments or health care facilities can be used to encourage improvement as no one wants to be seen as the worst. However it is important to encourage improvement and promote learning from experience without fear of negative consequences.

Develop a monitoring and evaluation framework

Often evaluation is done by measuring indicators, however in addition, the collection of descriptive data can be helpful to give a comprehensive view of the extent to which changes are occurring. Unintended consequences of evaluation may include distortion of treatment priorities and promotion of unnecessary antibiotic use. To counteract this, relevant stakeholders should be involved throughout the process including indicator development, analysis and feedback. When developing a monitoring and evaluation framework, teams should consider:

  • What will be measured?
  • How data will be collected and recorded
  • Analysis and interpretation of data
  • Feedback of the results to all stakeholders
  • Acting on the results

Measurement and indicators

Both process and outcome measures should be used within the monitoring and evaluation framework. The two types of indicators give different perspectives on the success of rational use programs. Measurement through an intermittent audit can help to maintain appropriate antibiotic prescribing/use. 

Process indicators:

Process measures give a view if improvements have been made in appropriate antibiotic use practices. Some examples are:

  • % prescriptions adhering to guidelines for appropriateness and timeliness of therapy for a given infection
  • % change in the use of an antibiotic or antibiotic class
  • % prescribers who accurately applied diagnostic criteria for infections
  • % prescribers who obtained cultures and relevant tests prior to treatment and modified antibiotic choices appropriately to microbiological findings
  • % prescribers who documented the indication and planned duration of antibiotic therapy
  • % prescriptions for restricted antibiotics that are concordant with drug and therapeutics committee approved criteria

Outcome indicators:

Outcome measures inform the degree to which rational use programs or interventions have reduced costs, prevented resistance or other unintended consequences of antibiotic use. Some examples are:

  • % change in antibiotic resistance
  • % change in drug cost
  • % change in mortality
  • % adverse drug events
  • % patients with complications
  • number of patients who develop super-infections, such as C. difficile infection

Quality indicators to monitor antibiotic use in hospitals:

A study was done to develop quality indicators to be used to measure appropriateness of antibiotic use in the treatment of all bacterial infections in hospitalized adult patients, with a focus on high-income settings. A literature review was used to identify quality indicators and then experts were consulted to appraise and prioritize potential quality indicators. Eleven indicators were chosen that can be used to identify which aspects of antibiotic use there is room for improvement.

  1. Empirical systemic therapy is prescribed following the local guideline.
  2. Two blood cultures are taken before starting antibiotic therapy.
  3. Specimens for culture are taken from sites of infection before starting systemic antibiotic therapy.
  4. When culture results become available, empirical antibiotics are changed to pathogen-directed therapy.
  5. Quantity and interval for dosing of systemic antibiotic therapy is adapted to renal function.
  6. When oral treatment is adequate for the clinical condition, systemic antibiotic therapy is witched from intravenous to oral within 48-72 hours.
  7. Case notes document the antibiotic plan at the start of systemic antibiotic treatment.
  8. When the treatment duration is greater than three days for aminoglycosides and five for vancomycin, therapeutic drug monitoring is preformed.
  9. After seven days with a lack of clinical and/or microbiological evidence of infection, empirical antibiotic therapy for presumed bacterial infection is discontinued.
  10. Local guidelines are present in the hospital and an evaluation of wether an update is needed is conducted every 3 years.
  11. Local guidelines should be based on the local resistance patterns but corresponding to the national guidelines.

In MEASURE you can find tools and resources for measuring various aspects of antibiotic resistance and use (for example point prevalence survey protocols and knowledge-attitudes-behaviors-perceptions questionnaires):

Analyze results

Data collected should be analyzed by the committee and additional key stakeholders. Involving people from outside the committee can provide additional understanding of what is happening in a particular setting and helps to build ownership of finding solutions. When reviewing results, dialog and discussion can help to understand the meaning of the results. Interventions showing progress can be considered for scaling up. However when there is no indication of change or improvement, committees should discuss possible explanations and decide if changes need to be made to the program.

Communicate results and provide feedback

Results should be fed back to collaborating partners and interested stakeholders, especially management in a timely manner. Communication methods and content need to be tailored to the audience, and reporting should be transparent to promote confidence in the guidelines and overall program. Positive progress should be acknowledged and public recognition should be made of the contributions and successes of all partners involved in the work. Affirmation of hard work will provide incentive for staff to continually improve and be involved in the process.

Usage data to give feedback to prescribers: 

In Sweden a systematic monitoring of prescription data is conducted, with feedback to prescribers on their prescription patterns. Local teams from Strama, the Swedish strategic programme against antibiotic resistance, provide feedback to prescribers in hospitals and in primary care. They visit each health center in the region regularly and present collated prescription data. This is an effective and organized way to provide information directly to the prescriber and at the same time provide an opportunity to discuss the results. The Swedish work on containment of antibiotic resistance has been collected in a handbook.

Selected Resources

Resource Description
Improving Medicines Access and Use for Child Health -A Guide to Developing Interventions Manual with practical tips for those developing interventions to improve access to and use of medicines, including antibiotics, for child illness.

  • Chapter 5 describes the evaluation process
How to Investigate Antimicrobial Use in Hospitals: Selected Indicators Manual that defines indicators for antimicrobial use in healthcare settings, particular focus on hospital, patients, and prescribing practices to assess the management and use of antibiotics in hospitals. Also available in French and Spanish.
Tracking and reporting of antimicrobial stewardship in hospitals Tools to help measure and report on antibiotic stewardship activities in hospitals, from the US CDC. Additional guidance on indicators can be found here. They also provide general guidance on evaluation of health initiatives under CDC Evaluation Resources.
Tracking and reporting of antimicrobial stewardship in outpatient facilities Tools to support measurement and reporting on antibiotic stewardship activities in outpatient settings, from the US CDC.
Antibiotic Stewardship Measurement Framework, CDC/IHI (PDF) Framework that describes selected indicators that could be used to measure change concepts of the “Antibiotic Stewardship Drivers and Change Package”.
Antimicrobial Stewardship in Australian Hospitals Book that assist hospitals to develop and implement antimicrobial stewardship programs. See for example Chapter 6: Measuring performance and evaluating antimicrobial stewardship programs (PDF 1MB).
A Concise Set of Structure and Process Indicators to Assess and Compare Antimicrobial Stewardship Programs Among EU and US Hospitals: Results From a Multinational Expert Panel A set of indicators relevant to EU member states and the United States to assess and compare antimicrobial stewardship programs in hospitals in different countries and health care systems.

More from "Set up a program"

1.
Ross-Degnan, D., Vialle-Valentin, C. & Briggs, J. Improving Medicines Access and Use for Child Health—A Guide to Developing Interventions. Preprint at https://siapsprogram.org/publication/improving-medicines-access-and-use-for-child-health-a-guide-to-developing-interventions/ (2015).
1.
Centers for Disease Control and Prevention - CDC. Implementation Resources for Outpatient Settings | Antibiotic Use. CDC Antibiotic Prescribing and Use https://www.cdc.gov/antibiotic-use/core-elements/outpatient/implementation.html (2021).
1.
CDC. Implementation Resources for Hospitals | Antibiotic Use. https://www.cdc.gov/antibiotic-use/core-elements/hospital/implementation.html (2021).
1.
Public Health Agency of Sweden. Swedish work on containment of antibiotic resistance Tools, methods and experiences. (Public Health Agency of Sweden, 2014).
1.
Centers for Disease Control and Prevention - CDC. Antibiotic Stewardship Measurement Framework. Preprint at https://www.cdc.gov/antibiotic-use/healthcare/pdfs/Antibiotic_Stewardship_Measurement_Framework.pdf (2014).
1.
Pollack, L. A. et al. A Concise Set of Structure and Process Indicators to Assess and Compare Antimicrobial Stewardship Programs Among EU and US Hospitals: Results From a Multinational Expert Panel. Infect Control Hosp Epidemiol 37, 1201–1211 (2016).
1.
Systems for Improved Access to Pharmaceuticals and Services. How to Investigate Antimicrobial Use in Hospitals: Selected Indicators. Preprint at http://siapsprogram.org/publication/how-to-investigate-antimicrobial-use-in-hospitals-selected-indicators/ (2012).
1.
van den Bosch, C. M. A., Geerlings, S. E., Natsch, S., Prins, J. M. & Hulscher, M. E. J. L. Quality indicators to measure appropriate antibiotic use in hospitalized adults. Clin. Infect. Dis. 60, 281–291 (2015).
1.
Wachter, R. M., Flanders, S. A., Fee, C. & Pronovost, P. J. Public reporting of antibiotic timing in patients with pneumonia: lessons from a flawed performance measure. Ann. Intern. Med. 149, 29–32 (2008).
1.
Pulcini, C. & Gyssens, I. C. How to educate prescribers in antimicrobial stewardship practices. Virulence 4, 192–202 (2013).
1.
Nathwani, D., Gray, K. & Borland, H. Quality indicators for antibiotic control programmes. J. Hosp. Infect. 50, 165–169 (2002).
1.
Cruickshank, M. & Duguid, M. Antimicrobial stewardship in Australian hospitals 2011. (Australian Commission on Safety & Quality in Health Care, 2011).