Research reflection guide

• What does the research say? What policy, program or practice (i.e. approach) is being evaluated? Where and when was this evaluation conducted? How many participants were involved?  
• How was the approach evaluated? What outcomes were looked at, and how were these outcomes measured? Was there a comparison between a group of individuals who experienced the approach and a group of individuals who did not experience the approach?  
• What standard of evidence does this research meet? Is the evidence generated by the research causal or correlational? Causal evidence shows that the approach caused a change in outcomes. Correlational evidence shows that use of the approach is associated with a change in outcomes, but doesn’t rule out the possibility that the change was caused by something else, or by chance.

• What connects with my experience? What about the research is similar to my context and our current priorities? What aspects of the research are different to my context?  
• What excites me about the research? What might be possible in my context?

• What makes the approach work? According to the research, what are the key features of the approach that led to improved outcomes? What resources and organisational conditions (financial, human, logistical, curricular etc.) enabled success?  
• Would there be a benefit if I changed to this approach? What am I currently doing? What would I have to change in order to adopt this approach? Given what the research says, would any of the changes I make lead to improved outcomes? By how much do I think outcomes would improve? Alternatively, am I already doing something very similar to the approach, such that any changes might not improve outcomes further?
• What adaptations would I need to make? How aligned is this approach with existing system approaches? What about the approach will I need to change? Will any changes affect the key features? Will any adaptations make the approach less effective? Will any adaptations make the approach more effective?  
• What is the cost, in time, effort, and/or other resources, of changing? What will it cost me and/or my students to change what I’m doing? Where will this time, effort or other resources come from? If I implement this approach, what would it replace? What would be the consequences on my students of replacing my existing approach?

Should I implement the approach? Are the potential benefits worth the costs of implementation?

If yes:

• How can I rally resources to support implementation? How do I make implementation as smooth as possible? What resources and/or organisational supports do I need? How do I access these resources and/or supports?
• How will I be sure that implementation is effective? What data do I need to collect to track the effects of implementation? How will I know that any changes will be due to implementation of this new approach and not anything else?

Title: An Abecedarian Approach with Aboriginal Families and Their Young Children in Australia: Playgroup Participation and Developmental Outcomes Author/s: Jane Page, Megan L. Cock, Lisa Murray, Tricia Eadie, Frank Niklas, Janet Scull, Joseph Sparling Journal: International Journal of Early Childhood Publication date: 1 August 2019

What approach was evaluated? Is the approach described clearly enough that I could replicate it? Does the description raise any questions?

Robyn's notes:

The study explored whether the ‘Abecedarian Approach Australia (3a)’ improved early language and learning skills of Aboriginal children attending Families as First Teachers early childhood playgroups. They specifically used the Conversational Reading and Learning Games that are two main elements of the approach. Described in plenty of detail on pp.238-239. Appears to require use of a suite of 200 Learning Games that are copyrighted, and staff need to be trained to use them. It would be good to find out more about how to access the Learning Games and training.

Where and when was the research conducted? Is the research recent enough to be relevant?

Robyn's notes:

Yes it’s recent - research was done in 2 remote Northern Territory communities between 2015 and 2017 and published in 2019. 

Number of participants

Do the authors justify the sample size or discuss sample size in the limitations section?

Robyn's notes:

191 Aboriginal children in 2 communities but only 149 who had data collected. There’s no discussion of sample size. However, the authors clearly describe the sample and explain that they wanted to maximise the number of children who were eligible to participate in the study. The sample was different for different parts of the analysis.

What outcomes were measured?

Are these outcomes relevant to me?

Robyn's notes:

Outcomes measured were language development, early academic skills and motor skills. These are key outcomes for the children – so yes, relevant.

How were the outcomes measured?

Do the authors provide evidence that their methods for measurement are valid and reliable ways to measure these outcomes?

Robyn's notes:

Used a standardised instrument called the Brigance Early Childhood Screen but they adapted it to make it culturally appropriate for remote Aboriginal communities – many children didn’t speak English as their first language. The adaptations and the process of making them are described in detail in an Appendix – it appears valid.

Was there a comparison between a group who experienced the approach and a group who didn’t?

How were participants assigned to each group? Was it random? If not random, do the authors explain how the groups were similar enough for a comparison to be valid?

Robyn's notes:

No. The Families as First Teachers playgroups are provided by the Northern Territory government, and they all use the Abecedarian approach. The playgroups are available to anyone who chooses to attend – the researchers couldn’t randomly assign children to attend or not attend.

The study analysed whether children who had greater participation in the program had better outcomes than those who had less participation. The researchers refer to this as the children’s level of ‘dosage’. The researchers grouped children into low, medium or high participation based on how often they attended the playgroup (and did at least one activity) and how many Conversational Reading interactions and Learning Games they participated in when they attended.

The only information about the children is gender, age and the community they live in so you can’t tell if the groups are similar on other characteristics. And the number of children in the high dosage group is much smaller than in the medium and low groups.

What did the research find?

Robyn's notes:

The study was with Aboriginal children attending free playgroups in remote Northern Territory communities. It found that children who had higher dosage of the Abecedarian activities had better outcomes than children who had lower dosage – high was better than medium and medium was better than low.

Is this causal evidence or correlational evidence?

Robyn's notes:

Because children weren’t randomly assigned to groups it’s possible the groups were different and that something else caused the results – maybe the families who went to the playgroup less often were busier or had other reasons for not being able to attend? The study wasn’t able to account for those things. There have been randomised control trials with other cohorts of children around the world though, and the article references a small randomised trial with Aboriginal children conducted by other researchers. Even though this study isn’t designed to test causal inferences, I’m pretty confident the approach itself is evidence-based. Page 4 says that the Abecedarian approach was selected ‘because of the quality, scale and impact of the empirical research and its well developed educational focus on children from birth to age 3’.

In what ways is the research similar or different to my context?

What do the authors say about the context? Does it appear that the context was important for the results or is it likely the approach would be just as effective in a different context?

Robyn's notes:

This study was in two remote Aboriginal communities with children who mainly didn’t speak English but the Abecedarian approach has been used in many different contexts – it started in the United States. In fact, the remote context is seen as a challenge by the authors so there’s no reason to think the program wouldn’t be useful for Aboriginal students in my city location. Like the study locations, our staff are Aboriginal and we have people from the local community volunteering or on staff. The study highlights that the number of times children engage in Conversational Reading and Learning Games with adults matters. It doesn’t give a minimum dosage needed to see improvement so we need to make sure that will be okay in our context – some of our children don’t attend regularly.

What might be possible in my context? 

What do I like best about this approach? Does anything concern me? Do I feel motivated to try it in my context? Why or why not?

Robyn's notes:

This looks like something we could do but I need to investigate how to access the Learning Games and other materials, and find more guidance on implementing the activities. The researchers mention the importance of fidelity of implementation a few times – this means that it’s important that the program is implemented exactly as intended. There’s training to make sure we can do that. I’ve heard good things about Abecedarian before but didn’t know the Learning Games (which is a main element of the approach) had been adapted for Aboriginal children. Based on what I’ve read here I definitely want to find out more.

What does the research say about the key features that led to improved outcomes?

What resources and organisational features enabled success? Does it seem that this would translate to my setting? Why or why not?

Robyn's notes:

Key features aren’t really mentioned but the article says it’s important to use both the Conversational Reading and the Learning Games (not just one or the other). Though how often children and adults engage in them is important. There are special materials to use, and training in the approach. Since dosage matters it would be important that children attended the centre on enough days to benefit from the approach. High dosage was at least 80 sessions in this study which seems achievable for us. It’s also important that children engage in Conversational Reading and Learning Games in daily programs. We should plan for Conversational Reading and Learning Games throughout the day (indoors and outdoors).

What am I currently doing?

How different is this approach to what I’m already doing? How much would I be changing if I implemented this approach?

Robyn's notes:

We try to engage parents now but not in the focused way it’s done in Abecedarian. This would be more structured and intentional than the reading and educational games we currently do with the children. That could create greater consistency between educators which would be good. It would be quite a big change but it’s doable.

Based on the research and my current practice, would changing be likely to lead to improved outcomes?

Why do I think this? By how much are outcomes likely to improve?

Robyn's notes:

It’s hard to know how much this would improve outcomes. I think our children might get a higher dosage and they’re probably starting from a higher base than the children in the study (hard to tell as the article doesn’t include the Brigance scores). Also, I think we’d get better engagement from our parents/carers than in the study. So I think we should see at least as much improvement as in the study. I could talk to other centres or look for more research before deciding whether to go further with this.

How aligned is this approach with existing system approaches?

Would I need to adapt the approach for my context? Why or why not? If yes, what would I need to adapt? Why? Will this affect the key features I identified above? Could it make the approach less effective? More effective?

Robyn's notes:

No, I wouldn’t need to adapt. The approach has been used in numerous contexts and the Learning Games have already been adapted for Aboriginal children. Also implementing with fidelity seems important – there’s training and materials to use – so I don’t want to change anything.

What is the cost to me or the children/students in terms of time, effort and resources?

Where will this time, effort and/or other resources come from? If I implement this approach, what would it replace? Would I be replacing something I’m confident is effective? What would be the consequences of replacing my existing approach for the children/students?

Robyn's notes:

Financial cost for training – there’s a practitioner course and a trainer course with an Australian Uni. It looks like one person can become a trainer then train others. I’ll do some searches to find out more about these options. Maybe a grant? Working with carers is a core part of the approach that will take extra time and effort – we’ve been wanting to do more parent engagement anyway so I’m happy with that. I see this improving our interactions with children and carers rather than replacing anything.

Are the benefits worth the costs?

How have I arrived at that conclusion? How confident am I?

Robyn's notes:

I need to check financial costs before deciding. If we have the money then the benefits look worth the costs. I’m confident staff will be on board – the approach aligns with what we’re already aiming to achieve and how we work.

If I implement the approach:

How can I rally resources to support implementation? What support will I need and where can I find it?

Robyn's notes:

It would take a while to embed the whole approach and train staff – perhaps we could just try it in the 4-year-old room to start? Maybe we could embed 1 element first to gain fidelity with 1 element, and then embed the other (for example, Conversational Reading first, then Learning Games). This needs some thought.

I’ll take a proposal to the management committee next meeting to discuss once I’m clear on the cost.

How will I be sure that implementation is effective?

What data will I need to collect?

How will I know that any changes are due to the change or approach and not something else?

Robyn's notes:

We wouldn’t be able to use a screening tool like the Brigance used in the study but the observations of children that we already collect provide good data. We could also collect feedback from parents/ carers in our half yearly interviews with them. I know other ECEC services use the approach so I’ll ask them what changes they’ve seen and how they monitor monitor whether it’s making a difference.