It is fair to say that I did not expect to be writing a blog post under this title back in the early weeks and months of 2020, when the weather was terrible and we gleefully shared space and germs. But on reflection, the first part of the title ‘Evaluating online outreach’ should not be so surprising. The transposition of information and guidance for applicants to higher education from face to face settings – such as in schools, on campus or at events – to a virtual setting has been a possibility for a long time. Universities across the world have engaged in remote education practices for some time. Yet, it has taken the second part of this title ‘a global health pandemic’ to really throw university outreach into the online landscape, and in the case of 2020, to fend for itself. This has forced widening participation, outreach and recruitment teams into new and innovative – and digital – ways of working, and as a consequence has demanded that the evaluations of those activities respond in step.
Here at the University of Exeter, we are proud of our response to these unprecedented circumstances as the plethora of outreach activities that we had planned were swiftly moved to an online setting. Open days became online open days, residentials became virtual residentials and our information and guidance sessions for applicants became the Discover University series – more on that in a bit.
Yet this presented new challenges to evaluation practices which were largely caught on the back foot. Collecting data about participants, for example, was formally done as part of the face to face session with participants completing a paper questionnaire. In the online world the response rates plummet as this stage can be easily avoided. Without the use of pre-activity registration forms, there appears to be little incentive or willingness for participants to hand over personal data about themselves. Fair enough, but absent this, outreach teams do not know who is engaging and do not have the capability to track participants’ subsequent progression into higher education.
Consider as well how we assess the impact on participants of those engagements. In the short term, practitioners gain little insight into how an event has run or been received as they lack the tacit knowledge that is more easily acquired in person. As a result we become more reliant on post-activity feedback questionnaires, which face the same incentivisation problem previously mentioned. Long-term, we do not know what choices participants made around entering higher education.
There are new hurdles to online evaluation too, for instance at what point do we consider an engagement as meaningful? Is it engagement with content for one minute, 50% of the content, 100%? When events take place in person, it is uncommon for large swathes of the audience to walk out, even after a few seconds, yet this is exactly what happens in a digital environment which does not need to conform to the same social norms. When an activity takes place in person, we can usually assume that attendees have been ‘present’ for the whole session; this cannot be assumed online.
Whilst I am confident that evaluation practices will move quickly to meet these challenges, I’m afraid this post doesn’t have the answers for now. It is early days in the brave new world. What I can tell you is what we learned about online outreach from the process evaluation of our Discovery University series.
Discover University was launched in response to the necessity for distanced outreach activity. It is a collection of web-based activities and resources to help pupils make informed choices about their future and their potential progression to university. A process evaluation of the programme has recently concluded and by extension made an assessment on the effectiveness of digital modes of outreach. In this evaluation, participant and practitioner feedback identified in common the strengths and weaknesses to the online approach. The main strength seemed to be its increased reach, both in terms of geographical location and diversity of audience. In principle, digital content is received ubiquitously in locations across the planet, removing geography as a barrier to information. More than this, digital content can also be accessible to applicants outside of traditional recruitment channels, such as attendees to local schools and partner organisations. Those who do not have the means to travel to Exeter have access to the same information as everyone else and those who have additional life commitments can engage with content flexibly at times to suit them. In many ways, this makes online outreach more accessible and inclusive than face to face engagements.
However, feedback from the evaluation also identified challenges that seem inherent to online outreach; namely that the quality of engagement is potentially reduced. It is, for example, harder to provide diverse and individualised sessions when producing video content for mass audiences. There can easily be an abundance of video presentations with little interactivity leading to participants feeling ‘zoomed out’. Prospective applicants reported that whilst the information is good, they could not always get a ‘feel’ for the university via online material and it was hard to imagine what life as a student is actually like. There seems to be some value lost in the online arena compared to the traditional face to face interactions.
Given the opportunities that online outreach presents, my prediction is that it is here to stay even beyond a socially distanced society, but not at the total expense of face to face activities. It could be analogous to internet shopping for clothes (stick with me), where online, you will find the dimensions, styles, colour of clothes, but it is not quite the same as visiting the shop and trying the items on. For low investment content like generic information about student finance applications – let’s equate this to buying a pair of socks in our analogy – online content is probably sufficient. For high investment decisions, such as choosing a course or which campus to study at, you might want to visit in person to get a feel for things, like you would visit a shop when buying a fancy suit, for example. Perhaps if you are like me you will do it all online and take your chances. If the clothing analogy is starting to fray at the edges by now, the point is that a blended approach to outreach is a realistic prospect, and the next frontier for evaluation will be to understand and measure the impact of these approaches. For all the disruption the pandemic has caused for learners, it has forced universities to leap into the deep end of online outreach, and may yet point towards a more inclusive outreach offer going forward.