Evidence decision-making tool for practitioners

A tool to help you use evidence to make decisions about a new or existing practice or program based on AERO’s standards of evidence.

Download (PDF, 195KB)

Introduction

The Evidence decision-making tool assists you to:

assess how confident you are that a certain practice or program is likely to be effective in your context
decide on next steps, including how to implement the practice given your level of confidence and how to collect more evidence to increase your confidence in its effectiveness.

The tool can be used by an individual or a group, for example, in a staff workshop or community of practice. It’s designed to be flexible, so you can use it to consider any type and size of change, from a small adjustment to an existing teaching practice to the introduction of a new school-wide wellbeing program.

The tool provides a structured way to help you consider the rigour and relevance of the evidence supporting a practice or program objectively but it’s not a set of rigid rules to follow. Sometimes it will be difficult to decide which category your evidence fits in and you’ll need to use your professional judgement to make a decision.

Throughout the Evidence decision-making tool you will find links to other AERO resources that provide more information to help you find and evaluate the evidence you need.

How do I use the Evidence decision-making tool?

1. Identify your level of confidence in the practice

Read the descriptions of evidence in the first section of the tool and identify the statement(s) that best describes the evidence you currently have to show that this practice is likely to be effective in your context. The highest level you reach is your current level of confidence.

For example, if you have heard anecdotal evidence (which is classified as low confidence evidence) and you have also trialled the practice and collected data that suggest it was effective (which is medium confidence evidence), then your current confidence is at level 2, medium confidence.

2. Consider how to implement the practice and collect more evidence to increase your confidence

Read the suggestions about implementing the approach in the corresponding section under 'how should I proceed'. These suggestions include how to collect more evidence if you seek to increase your confidence in the effectiveness of the approach. For example, if you identified that your level of confidence is low in step 1, you would consider the suggestions in the low confidence column.

You can increase your confidence by finding more rigorous and/or relevant research evidence or by trying the practice and observing or collecting data about the effects (practitioner-generated evidence). When your confidence increases, move up a level in the tool and consider the suggestions that match your new level of confidence.

Consider consulting colleagues within your service, school or organisation (as appropriate) before you begin or discontinue an approach.

How much confidence is enough?

Although high or very high confidence is preferable, sometimes there isn't enough evidence available to achieve these standards.

When evidence is lacking, use your professional judgement to weigh the potential benefits, costs and risks of implementing the approach. For example, it may be acceptable to implement an approach in which you have only low or medium confidence when not changing a current practice may do harm or when the consequences of implementing an ineffective approach are small or have been appropriately mitigated. In these instances it is particularly important to carefully monitor and evaluate the effects of the approach. The Evidence decision-making tool provides guidance for cautiously implementing an approach and collecting more evidence when you have only low or medium confidence.

Remember, knowledge is constantly evolving and contexts change so the process of considering the evidence is never finished. It’s a good idea to revisit the evidence and reassess your confidence level as often as is reasonable given the importance of the decision you’re making.

Evidence decision-making tool for practitioners

How confident can I be about the effectiveness of a particular practice in my context?

Identify the statement(s) that best describes the evidence you currently have to support this practice. Your current level of confidence is the highest level for which you have evidence. For example, if you have both level 1 and level 3 evidence, your current level of confidence is level 3.

Level 1: low confidence

I can hypothesise why the practice should be effective. I can explain step by step how the practice is expected to lead to positive effects for my children or students. However, I don’t have any data or academic research to show that this hypothesis is correct.

and/or

I have read/heard descriptions of the practice that provide opinion or anecdotal evidence.
However, these descriptions do not provide data or references to academic research that shows the practice is associated with positive effects.

Level 2: medium confidence

I have trialled the practice myself or know others who have trialled the practice.
I have seen data on how children or students responded to the practice that suggests it has positive effects.

and/or

I have read research published in peer-reviewed academic journals or conducted by a trustworthy source such as AERO that showed the practice was correlated with positive
effects but did not show the practice caused the positive effects.

and/or

I have read/heard credible non-academic sources (for example, books, websites, podcasts) that said this practice is effective. However, I’m not aware of any academic research to support these assertions.

Level 3: high confidence

I have read research published in peer-reviewed academic journals or conducted by a trustworthy source such as AERO that showed the practice causes positive effects. However, the research was not necessarily in my context.

and/or

I have read/heard credible non-academic sources (for example, books, websites, podcasts) that said this practice causes positive effects. These sources included references to academic research or research conducted by a trustworthy source such as AERO that showed the practice caused positive effects.

Level 4: very high confidence

I have read systematic reviews and/or metaanalyses that show this practice causes positive effects over a wide range of contexts.

and/or

I have read several research studies published in peer-reviewed academic journals or conducted by a trustworthy source such as AERO that show the practice causes positive effects in contexts similar to mine. These studies used robust experimental designs.

Given my level of confidence, how should I proceed?

Read the suggestions in the section that matches your current level of confidence.

I have low confidence

I should use this practice cautiously or not at all.

If I wish to use the practice, I should:

try it on a small scale (for example, for a short amount of time)
closely monitor how children or students respond and observe the effects
be alert to the possibility of negative consequences and have a plan to manage them
be prepared to cease the practice if it appears no more effective than existing practice or if there are signs of negative consequences.

To increase my confidence, I could:

seek advice from colleagues, asking them if they have trialled the practice and if they have either practitioner-generated or research evidence that shows the practice is effective
search for academic research that provides medium, high or very high levels of confidence about whether the practice is effective.

I have medium confidence

I should use this practice cautiously.

When using the practice I should:

aim to replicate how it was used where it showed positive effects
seek implementation advice from others who have used the practice and found it effective
monitor how children or students respond and have a plan to manage potential negative consequences
be aware that the practice may be ineffective or have negative consequences and be prepared to cease it if the evidence doesn’t support its use.

To increase my confidence, I could:

search for academic research that provides high or very high levels of confidence that the practice is effective
observe and/or collect data on the effects of the practice for my children or students over time
collaborate with colleagues to observe and/or collect data about the effects of the practice for a larger sample of children or students (be aware that if contexts differ, the effectiveness of the practice could also differ).

I have high confidence

I should use this practice.

When using the practice I should:

aim to replicate how it was used in the research where it showed positive effects
carefully consider whether adjustments are necessary for my context, base them on insights from the research about how the practice produces positive effects, and keep a record of what I have adjusted
have a plan to manage potential negative consequences from adjusting the practice
repeat this cycle until I’m confident the practice is as effective as possible in my context.

To increase my confidence, I could:

search for academic research that provides a very high level of confidence that the practice causes positive effects in contexts similar to mine
try the practice for a period of time and compare child or student outcomes before and after introducing the practice
try the practice with some children or students (randomly selected if appropriate) and compare the results
collaborate with colleagues in similar contexts to compare child or student outcomes in groups that do and do not receive the practice.

I have very high confidence

I should use this practice as much as I can.

When using the practice I should:

aim to replicate how the practice was used in the research where it showed positive effects
continually monitor my use of the practice and how my children or students respond
continue to observe and/or collect data about my children or students to ensure the practice remains effective.

To increase my confidence in using this practice, I could undertake action research to support continuous improvement, for example, to better understand:

how the practice works, its key features and any necessary enabling conditions
whether it’s equally effective for all children or students in my cohort, or in different school or service contexts, and if not, why this is the case.

Remember to keep monitoring the research evidence to stay up to date with evolving knowledge about effective practices, programs and policies in education.