Evidence decision-making tool for policymakers

A tool to help you use evidence to make decisions about a new or existing policy, program or other initiative based on AERO’s standards of evidence.

Download (PDF, 193KB)

Introduction

The Evidence decision-making tool assists you to:

assess how confident you are that a certain policy, program or other initiative is likely to be effective in your context
decide on next steps, including how to implement the initiative given your level of confidence and how to collect more evidence to increase your confidence in its effectiveness.

The tool can be used by an individual or a group, for example, in a planning workshop. It’s designed to be flexible, so you can use it to consider a change to an existing initiative or the introduction of something new.

The tool provides a structured way to help you consider the rigour and relevance of the evidence supporting an initiative objectively, but it’s not a set of rigid rules to follow. Sometimes it will be difficult to decide which category your evidence fits in and you’ll need to use your professional judgement to make a decision.

There is also an Evidence decision-making tool for practitioners in early childhood or school settings to consider practice or program changes.

How do I use the Evidence decision-making tool?

1. Identify your level of confidence in the practice

Read the descriptions of evidence in the first section of the tool and identify the statement(s) that best describes the evidence you currently have to show that this initiative is likely to be effective in your context. The highest level you reach is your current level of confidence.

For example, if you have heard anecdotal evidence (which is classified as low confidence evidence) and you have also read peer-reviewed research that suggests high-performing systems use this initiative (which is medium confidence evidence), then your current confidence is at level 2, medium confidence.

2. Consider how to implement the practice and collect more evidence to increase your confidence

Read the suggestions about implementing the approach in the corresponding section under 'how should I proceed'. These suggestions include how to collect more evidence if you seek to increase your confidence in the effectiveness of the initiative. For example, if you identified that your level of confidence is low in step 1, you would consider the suggestions in the low confidence section.

You can increase your confidence by finding more rigorous and/or relevant research evidence or by trialling the initiative and observing or collecting data about the effects. When your confidence increases, move up a level in the tool and consider the suggestions that match your new level of confidence.

How much confidence is enough?

Although high or very high confidence is preferable, sometimes there isn't enough evidence available to achieve these standards.

When evidence is lacking, use your professional judgement to weigh the potential benefits, costs and risks of implementing the initiative. For example, it may be acceptable to implement an initiative in which you have only low or medium confidence when not changing may do harm or when the consequences of implementing an ineffective initiative are small or have been appropriately mitigated. In these instances it is particularly important to carefully monitor and evaluate the effects of the initiative. The Evidence decision-making tool provides guidance for cautiously implementing an initiative and collecting more evidence when you have only low or medium confidence.

Remember, knowledge is constantly evolving and contexts change so the process of considering the evidence is never finished. It’s a good idea to revisit the evidence and reassess your confidence level as often as is reasonable given the importance of the decision you’re making.

Evidence decision-making tool for policymakers

How confident can I be about the effectiveness of a particular initiative in my context?

Identify the statement(s) that best describes the evidence you currently have to support this initiative. Your current level of confidence is the highest level for which you have evidence. For example, if you have both level 1 and level 3 evidence, your current level of confidence is level 3.

Level 1: low confidence

I can hypothesise why the initiative will work. I can explain step by step how the initiative will lead to positive effects in my system. However, I don’t have any data or academic research to show that this hypothesis is correct.

and/or

I have read descriptions of the initiative that provide opinion or anecdotal evidence. However, these descriptions do not provide data or references to academic research that show the initiative is associated with positive effects.

Level 2: medium confidence

Early adopters in my system think the initiative is effective, and can show me data that suggest there have been positive effects.

and/or

I have read research published in peer-reviewed academic journals or conducted by a trustworthy source such as AERO that suggests high-performing systems use this initiative. This evidence shows that using this initiative is associated with positive effects but doesn’t demonstrate that the initiative caused the positive effects.

Level 3: high confidence

I have read research published in peer-reviewed academic journals or conducted by trustworthy sources such as AERO that evaluates this initiative. This research shows this initiative causes positive effects; however, the research was not necessarily in my context.

and/or

I have read credible nonacademic sources (for example, books, websites) that said this initiative has positive effects. These sources included references to peer-reviewed academic research or research conducted by a trustworthy source such as AERO that show the initiative causes positive effects.

Level 4: very high confidence

I have read systematic reviews and/or metaanalyses that show this initiative causes positive effects over a wide range of contexts.

and/or

I have read or conducted a rigorous causal evaluation of the initiative as implemented
in my system. This evaluation shows that the initiative causes positive effects.

and/or

I have read several research studies published in peer-reviewed academic journals or conducted by a trustworthy source such as AERO that show this initiative causes positive effects in system contexts similar to mine. These studies used robust experimental designs.

Given my level of confidence, how should I proceed?

Read the suggestions in the section that matches your current level of confidence.

I have low confidence

I should implement the initiative cautiously or not at all.

If I decide to trial it, I should:

run a small-scale pilot led by experienced leaders, teachers, and/ or educators who want to trial the initiative
undertake a risk assessment to plan for and mitigate any consequences should the initiative be ineffective
closely monitor implementation to identify whether the initiative is working and manage any unintended negative consequences
be prepared to discontinue the initiative if it appears no more effective than existing approaches or if there are signs of negative consequences.

To increase my confidence, I could:

undertake a review of the research literature to identify whether there is any evidence from other systems that provides medium, high or very high levels of confidence about whether the initiative is effective
consult others who have implemented the initiative to learn if they have data or research evidence that demonstrates the initiative has positive effects.

I have medium confidence

I should implement the initiative cautiously.

When implementing the initiative I should:

allocate limited resources to the initiative and ensure that I do not divert resources from other initiatives that I have more confidence in
replicate what has been done in other systems where the initiative is associated with positive effects
undertake a risk assessment to plan for and mitigate any consequences should the initiative be ineffective
closely monitor implementation to identify whether the initiative is working and manage any unintended negative consequences
be aware that the initiative may be ineffective or have negative consequences and be prepared to discontinue it if the evidence doesn’t support its use.

To increase my confidence, I could:

undertake a review of the research literature to identify and synthesise evidence that provides high or very high levels of confidence that the initiative is effective
commission an evaluation of the initiative (which could be of a small-scale pilot) to assess whether it causes positive effects.

I have high confidence

I should implement the initiative.

When implementing the initiative I should:

ensure that I do not divert resources from other initiatives that I have more confidence in
replicate what has been done in other contexts where the initiative has demonstrated positive effects
begin implementation in places that are most similar to those where the initiative was found to be effective
carefully consider whether adjustments are necessary for my system, base them on insights from the research about how the initiative produces positive effects, and document the adjustments and why they were made
undertake a risk assessment to plan for and mitigate any unintended consequences
monitor implementation to identify whether the initiative is working and manage any unintended negative consequences.

To increase my confidence, I could conduct or commission research or evaluation to assess the effectiveness of the initiative in my system, for example, by:

comparing the longitudinal outcomes of schools or services that have already implemented the initiative with similar schools or services that have not
randomly staggering implementation and comparing the outcomes of schools or services that start implementation at different times.

I have very high confidence

I should implement the initiative as widely as possible.

When implementing the initiative I should:

allocate resources that support implementation of the initiative at scale
replicate how the initiative was implemented in the research where it demonstrated positive effects
collect and analyse data on how and why the initiative works in my system to support ongoing improvement
continually monitor the initiative to ensure ongoing effectiveness.

To increase my confidence, I could conduct or commission research or evaluation to
better understand:

why the initiative is effective, its key features and any necessary enabling conditions
whether it is equally effective for all children, students and/or educators, and if not, why this is the case.

Remember to keep monitoring the research evidence to stay up to date with evolving knowledge about effective practices, programs and policies in education.