The following is the first in an occasional series of blogs written by colleagues working with realist methods outside the University of Exeter.  Mark and I are very grateful to Kevin Harris for his contribution.  Kevin is a Senior Lecturer at the Faculty of Business, Sport and Enterprise, Southampton Solent University, and his post is a closer look at the evaluation of The Coaching Innovation Programme…. over you to Kevin.

Hi there,

My name is Kevin Harris and I am a senior lecturer and course leader in sport development and sport policy at Southampton Solent University.

When deciding to take my PhD I was keen to do something that kept me connected with the industry I used to work in (sport for social change) and create a closer bridge between academia and industry.

Around this time I had just created the Coaching Innovation Programme which in essence mobilises student led sport, physical activity and coaching projects to residents in the community of Southampton to address community needs. For example my students have been involved with researching the needs of a community and delivering their own projects – things like combining physical literacy with maths and science in the curriculum to promote learning, all the way through to delivering sports based sessions to offer resources for homeless people. The students work with industry practitioners to address niche areas and respond to those needs.

The Coaching innovation Programme now mobilises around 30-40 of these projects every year now so this is a massive contribution to the sport development and physical activity landscape. One of the things which required and still required addressing was the unanswered questions surrounding evidence of the projects. The students would be able to refer to satisfying experiences for them, their participants and practitioners yet they would struggle to evidence the impact of their project, but even more importantly how and why their project achieved its outcomes.

This is where I saw a fantastic opportunity to apply my PhD to the CIP and come up with a monitoring and evaluation framework which would enable the students and practitioners to make sense of what they learnt from their programme. This would also address some of the issues in my field that surrounds practitioners engaging with monitoring and evaluation (M and E) which is relatively poorly carried out (Coalter, 2007). In essence I wanted to bring M and E practice closer to the practitioners and more embedded in their work.

So, then… Over the last two years I have spent a considerable amount of time reviewing certain approaches to evidence, M and E. This has been and felt like a round the world trip in itself. There is an ocean of literature and approaches out there. Given the nature of the complexity of the interventions my students are implementing it was no surprise like many of us that I found myself exploring the philosophical roots of critical realism and the emphasis of programme theory. This was after tinkering with other aspects of programme theory such as logic models whose operational logic failed to really capture what we were trying to do.

The realistic angle of programme theory and evaluation really started to take hold as it really fitted with the nature of the interventions around producing multiple outcomes for different people in different contexts and firing mechanisms. around conceptual logic. This led me to produce two models focusing on the formation of programme theory and monitoring and evaluation. The first which would take students through the steps of developing their own candidate programme theory, borrowing the principles of Pawson and Tilley’s (1997)s realist approach and combining with other aspects of operational logic by anatomising the programme Funnel and Rogers (2011). By this I simply mean outlining and breaking down the programme strategy into its components – eg activities, inputs, outputs. Made up of three stages the first model would firstly map the field and establish the context whereby students would carry out a situational analysis such as looking at the geography of the area, the needs of the participants and contact with stakeholders. This would then inform stage 2 which would enable the students and practitioners / additional stakeholders to anatomise their programme (light touch logic model) and establish the key outcomes / subsidiary theories which constitute their project.

These would usually constitute ‘if’ and ‘then’ assumptions which would lead to stage 3 which goes on to explain how and why those outcomes may come about. It is this stage I suppose that really captures the realistic lens of conjecturing CMO configurations getting to heart of explaining how and why and for whom the outcomes might work. By the end of these stages the students and practitioners have themselves a robust and rigorously constructed candidate programme theory.

Of course the next step is to then test the theory through project delivery and M and E. This is where model 2 comes in which takes the students through 6 key stages of programme evaluation within a light touch realistic approach. For example , part 1 consists of reconceptualising and refreshing programme theory, part 2 consists of developing and framing evaluation questions within the realistic lens (eg what works for whom in what circumstances and why). These questions are particularly constructed against each CMO conjecture but NOT all of them. Like Pawson 2012 states, steady your fire! Part 3 and 4 involves establishing methodological competency and agreement of methods to answer questions and part 5 covers data analysis with 6 covering the reporting of data. These 6 parts have been produced using participatory evaluation approach which has involved the students throughout via cooperative enquiry and training / facilitation (Greenwood and  Levin, 2007; Fetterman, 2005) from myself. The aim was to train / facilitate the student practitioners to be able to carry out realistic techniques for their M and E. Thus, I am testing the model.

At this stage I have just completed (nearly) my first pilot of the model by working with 6 student projects. The workshops have been delivered and supported by action learning sets engaging in discussion with the students and progressing their M and E. My aim is to reach Mphil transfer this summer by exploring the utility of the model(s), the extent of students engagement and praxis in M and E.

Key Challenges:

Firstly , teaching and stimulating interest in the area of M and E is hard, especially for young practitioners! This is particularly hard given the language used with Realist .Evalutation and in many respects students simply do not get it. The academic discourse in which it resides presents a challenge for unlocking its potential for people working on the ground. The nature of the projects themselves and the time it takes to employ a realistic evaluation has also been challenging for the students. For example, how do you uncover the generative mechanisms for change of 9 year old children? All this in addition to the many other priorities of university work load and life for the students I have been working with.

In addition, the conceptual obstacles that realsitic evaluation presents is also a major challenge. Mid range theory, Demi regularities, conjectures, mechanisms and theory riddle the literature on this. This creates major obstacles for practitioners.I don’t think that it’s the different way of seeing’ that realist approaches advocate  (eg why things work) that cause the problem. It is more about the language and understanding of how to identify mechanisms of change. I initially wondered when developing the model whether the conceptual nature of the realistic approach would be suitable for practitioners. My initial thoughts were that it should given that such an insightful method should not be constrained to academic discourses. Having posted this in the RAMESES mailing list, thankfully the guru herself Gill Westhorp stated that it’s entirely appropriate for practitioners to be introduced to the approach. Why shouldn’t practitioners engage with such techniques?

The key Gill said was to communicate it in a way that does not confuse the language and can meet the contextual needs of practitioners and how they engage. This is something I have tried to follow. By far, the hardest thing to grasp is the ever elusive programme mechanism. Having attendee the realist training workshop in Liverpool this March Justin Jagosh did a great job in explaining ways to identify a mechanism. In that programme activities within out candidate theory provide resources and opportunities and those resources and opportunities produce reasoning in the minds of the programme users. The key is identifying what resources are they … what opportunities are they and how might the participants respond to them. These are the ingredients which then produce the mechanism for certain people (for whom).

I am really keen to benchmark with people on this. I really value the realsitic approach yet promoting it within a simplistic and pragmatic way for students and practitioners is a key area for discussion.

Links to resources:

1. The Coaching Innovation Programme  http://www.youtube.com/watch?v=La1vUuMNoMU
2. Kevin Harris profile  http://www.solent.ac.uk/faculties/fbse/staff-profiles/harris-kevin.aspx

References:

Coalter, F. (2007). A wider Social role for sport.  Oxon: Routledge

Funnell, S. and Rogers, P 2011. Purposeful programme theory.  Affective use of theory of change and logic models . San Francisco, USA: Jossey Bass

Pawson, R. (2003). Nothing as Practical as a Good Theory.  Evaluation 2003 9: 471

Pawson, R and Tilley, N. (1997) Realistic Evaluation. London: Sage

Greenwood, DJ and Levin, M. (2007) Introduction to Action Research 2^nd edition – social research for social change. California: Sage

One of the problems in evaluating complex interventions can often be helping the decision maker at the ‘receiving end’ of the research sees what decisions are open to them.  The scenario where the decision maker, or review commissioner, works alongside the review team to coproduce the research is desirable.  They lend their experience and advice, researchers lend their skill and expertise.  Something that Pawson seems to be at pains to point out as a difficulty with conventional approaches to systematic review is that they frequently start with a question, which is normally turned into a neatly identified PICO, and then the review team trots merrily off into the literature like so many Hobbits on their journey to Mordor. (Okay, I made that last bit about Hobbits up, but you get the idea…)

Pawson suggests that the difficulty with doing this is that it leads to ‘explanatory impoverishment’ (Pawson, 2006. P79).  If the gate through which a question is asked and evidence gathered becomes so narrow at the outset the review suffers because interventions involving human subjects, behaviours and actions are inherently complex and complicated, and are not always readily reducible to simple effectiveness questions.  For instance, they often involve a multitude of different stakeholders (with different agenda’s, masters to appease, and targets), and rival policies vie for their attention, and a host of choices open to the targets of the intervention (ignore, subvert, substitute, acquiesce etc).  There needs to be room to look and explore these aspects so the first task of a review needs to enable the decision maker to focus wisely on that which will be most fruitful for a review.  Pawson suggests that this stage is time-consuming and ongoing, and that  ‘anyone anticipating the quick fix of a watertight review question in an explanatory review should beware that they will very soon find themselves neck-deep in alternatives.” (Pawson, 2006. p80).

So how do we navigate through the initial stage of a realist review, to enable the decision making to make the decision about what they want the review to focus on?  Well, Pawson suggests that we start by mapping out the territory, to seek important programme theories, to look at how they’ve been tried out, to explore ideas around implementation difficulties, and to look at where the programme has gone wrong (and why that might be).  Once the conceptual reconnaissance has been carried out, the review team and commissioners can plot course for the review.  And, as a bonus, the surfacing, or building of nascent programme theories will have also commenced.

So what has this got to do with Taylor Swift?  Well, I consider myself to be lucky to now have 9, yes, count them, 9 nieces and nephews.  Some of them are on Facebook now, which provides a really interesting insight into how the lives of teenagers are lived (I heard about my niece’s marriage via Facebook.  She is 13.  I told her Dad.  He already knew. Kids eh?!).  But still, at the beginning of this month, I found myself wondering how I could ‘keep up’ with them, and stay a relevant relative.

So I bought ‘Red’, a Taylor Swift album, in order to get an insight into the music (and maybe the world), of kids these days.  I have been listening to it non-stop since.  Much to the dismay and derision of my friends.  But the interesting thing about the album is that most of the songs are just the same as the songs I listened to when I was a teenager, because they are all about love and loss, and finding your perfect soul mate: ça change, plus c’est la même chose, I suppose.

And it struck me on my way into work one morning that actually, Taylor’s quest to find the ‘ideal’ partner is a process of mapping the range of options and possibilities out there.  If the album was a realist review, it would be at the very beginning, working with decision makers to enable them to appreciate the range of options open to them.  What Taylor is actually doing is discussing the kinds of decisions they might need to make; she is ‘throw[ing] light on the decisions in decision making’ (p.30 Pawson, 2005).

Each song suggests a different relationship, or phase in a relationship; and each gives us an idea of how love can misfire, go right, go bad, work out or fail miserably.  It’s a great album.

In ‘State of Grace’, we hear the story of falling for someone at first sight, the closeness you can feel to someone you don’t yet know, and the excitement of knowing that you’ll never see things in the same way again.  It’s the passionate, unbelievable way you feel when you first fall in love.  The ideal partner here is one who is ‘worthwhile’, ‘my Achilles heel’ and the one that makes you feel it is ‘good and right’.

‘Treacherous’ is a tale of illicit affection; the falling into something with eyes wide open, knowing it is not ‘good and right’, but being unable to resist.

And I’ll do anything you say/  If you say it with your hands/

And I’d be smart to walk away, / But you’re quicksand 

‘Trouble’ is probably one of Taylor’s best known songs from this album, and rightly so.  It has a catchy tune, and a heartfelt, heartrending message of the folly you feel when you continue to entertain a relationship that you knew from the start wasn’t going to go anywhere good.  Here, the focus is on the ‘trouble’ of the boy in question: his self-absorbed and self-righteous traits

no apologies/ he’ll never see you cry/

pretends he doesn’t know/ that he’s the reason why/

you’re drowning.

He is a far cry from the ‘good and right’ boy of State of Grace. However, there seems to be a pride in Taylor’s voice in this song.  Almost as if having been with a boy that was ‘trouble’, she had passed an important rite of passage.  Or indeed, that it was the very ‘trouble’ that he was which set him apart and made him attractive.

‘Begin Again’ closes the album with a summary, a contrasting of the present partner with the one that came before, indicating the faults and follies of the previous lover, and the benefits and beauties of the present;

[…] and you throw your head back laughing/ like a little kid/

I think it’s strange that you think I’m funny cause/ he never did.

Throughout the album, Taylor points out the range of different Boyfs one can get caught up with, and in telling her stories, and providing her evidence, she delivers a description of the kinds of relationships which are possible, and the sorts of decisions which may need to be made.

I sometimes wish that it was possible to use more creative means in my research… so if you see that I’ve bought my guitar into work, and instead of staring at peer reviewed literature I’m staring at a blank score… then maybe I’ll finally be on to something…  Which reminds me, must ring RKT about that musical I was planning on shared care…

This is a state of grace
This is the worthwhile fight
Love is a ruthless game
Unless you play it good and right

ça change, plus c’est la même chose, 

…and speeding down the hill came Ramone with his sparkling new ice cream van and a bullet for Molly Macs for breaking his heart.

[this is a post which was written in the summer… I have moved house now, thanks, and it’s lovely.]

So what is the connection between unexpectedly finding Haytor, and Realist Approaches?  Well, recent months have found me boring on to anyone that was unfortunate enough to be in the vicinity about house hunting.  And the thing about house hunting is that there are always more houses to hunt.  So, this picture was taken last Friday evening, as I’d just been to visit a garden flat in Ilsington, and took a left and not a right turn upon leaving, and so instead of finding myself in Sigford, I happened upon Haytor.  Which was nice.

Sorry, yes, still haven’t got to the connection.  Okay, so finding a new home – there’s always more of them out there, and the thing is, you’re always going to wonder if the one that you’ve found is the right one/best one so you can stop looking… and you’ll also find interesting and pretty things along the way (like unexpected Dartmoor) which can be distracting or crucially important….. but the search for houses really tends to be governed by external factors: how much time have you got? how much money have you got? etc etc.

….. which is a bit like realist synthesis…. it might be the objective of the realist synthesis to achieve theoretical saturation, so that we can be pretty confident that there is ‘no more’ important evidence to capture for the particular theory we are building or testing…. but the truth is, a more practical approach tends to be necessary – there will always be further evidence that would be brought to bear, to further build or refine our thinking….  so when we ‘stop’ searching, are we making a judgement which is based more on external factors to the project (time, funding), rather than the internal factors (we’ve found it all)?  When does synthesis stop?

My colleagues at UEMS who have far more experience than me in research, are going to be invaluable in helping judge when the time has come to down tools, bearing in mind there will always be more explanations, and always more houses.

I like the Beach Boys, listening to their music reminds me of the summer, and as I sit here listening to the rain thunder down on the roof of the Veysey Building I am reflecting on the summer we’ve had.

Back in June, a few of us caught up with Ray Pawson and colleagues in Leeds.  It was a useful and interesting time – there’s nothing quite like talking with people that you share a methodological sympathy with – as we discussed our present work, ideas for the future and where realist approaches may go in the longer term.

Rob’s been having fun all summer long, in Australia… but it’s not all been play- the cost-effectiveness review is almost finished, and alongside going walkabout with the family, he’s also been running workshops and talking at conferences – including a day long session on realist methods with Gill Westhorpe at the Australian Evaluation Society Conference in Brisbane last month. We’ve kept in touch and have continued work on the shared care project (obvs), and it has been an interesting, and at times, unnerving experience.

For instance, before Rob went, we identified a set of papers which we felt would offer the thickest, richest explanatory detail to enable us to test out our theories on how shared care works.  Unfortunately, when it came to data extraction, these papers just didn’t cut it – they were not truly relevant to the theories we had developed – so whilst they were helpfully descriptive and theoretical, it didn’t really matter, because they weren’t related to our theories.  This was a big disappointment, as it set the project back a couple of weeks whilst I re-screened and did further searches and retrievals in order to ensure that the studies being included in phase 2 really were going to help us test.

Then there was the data extraction form – this went through about 4 iterations, before the one being used was developed, but even so, there are still gaps, and additional fields are still being added – apparently this is not unusual so I’m not overtly concerned.  What I am concerned about it how much time it will take to data extract – even though Pawson states in the 2006 book that not every part of every study will speak to every part of every theory (i.e. one is selective, led by theory, as to what it is important to ‘extract’ from the text), you still have to read the whole thing, and extract as much as you can – so whilst it is okay to have blank fields on the DE forms, you have to be pretty sure you’ve thoroughly ransacked the article and sucked out its marrow to be confident.  But apparently that is not unusual either.  What has been unnerving is making these decisions on a semi-confident ‘we’ll see’ kind of basis.  Although you could also interpret that as ‘flexible thinking and practice’… it’s all in the branding I guess.

Mark’s been having fun all summer too.  Some of us went to the Cochrane Colloquium in Quebec, where he ran a couple of workshops on realist approaches and theory in systematic reviews (he was v. good) with Justin Jagosh from McGill University in Vancouver, and Geoff Wong from Queen Mary’s in London.  Harriet and I had an abstract on how to build a programme theory accepted for presentation and that went really well too.  The questions we were asked afterwards demonstrated people had understood what we were talking about and we made some valuable connections throughout the week for our current work and future research plans .  It was just such a shame we couldn’t have done the presentation in French.

The 21st Gala celebrations were quite an experience, and I found myself on the one hand moved (by the heartfelt farewells to a long serving cochranite) and on the other absolutely livid (for one example the portrayal of someone with mental health issues as gurning, incoherent and stupid).  But the disco dancing afterwards was ace, and Exeter totally owned the dancefloor.

So it’s been a good summer.  But it is over now.  I know this because I bought a new notebook in Paperchase yesterday.  And that always signifies the start of the autumn term.

Gala celebrations – wanna dance?

I am a big fan of Skyrim, the Bethesda Game Studios open world computer game, where you get to basically live out Nordic Saga’s from the comfort of your own home.

One of the characters you meet during adventuring across the 9 ‘holds’ of Skyrim is Wylandriah, a mage at the court in Riften.  When you first come across her, she is trying to remember the various locations of some important magical items… and upon asking her about them, she sends you off on a quest to hunt and gather.   One of my favourite things she says is “Now where did I put my spiders…?”  But the one that got me thinking the other night is “Is it ward first, then summon?”  (basically, should she defend herself, and then summon a creature to fight for her, or start with the creature, and then defend?)  Because if you’re wondering how to practically start a realist study, your focus can get mired quite quickly in trying to work out whether you should go galavanting through the literature looking for ‘mechanisms’ (those resources which an intervention or programme makes available, and how people react to them), or ‘outcomes’ or indeed ‘contexts’.

(Or you can do what we did in the shared care review and just map the complexity – but that’s another story…)

Well, I remember when I was first reading these methods that Pawson seemed to be quite focused on the outcome pattern.  And rightly so – understanding why things work, and why they don’t is, after all, one of the key reasons why people do research.  And whilst this kind of unhelpful ‘it depends’ can really mess about with a conventional approach to evidence synthesis, for the realist, this pattern of work/not work is actually really useful.

The soul and centre of an intervention is a theory which goes something like “if we do this, then this (desired outcome) will happen”.  They are based on a mechanism being brought to life through the right circumstances.  If you get the context right, then the mechanism works, if the context isn’t right, the mechanism misfires.  So the objective of attempts to achieve outcomes is to somehow jolt into life the corpse of an theory…  As Mary Shelley will tell you, you need the lightening to do this.  So a thunder storm is the necessary context.

So the bringing to life, or death of an intervention are equally useful – they indicate the circumstances within which the mechanism of action has been tried, and they provide clues to help us understand why it is that the intervention worked or didn’t work.

So if you’re wondering how to begin, start with pondering the outcome pattern of the intervention – that pattern of failure, success and all points in between are the first steps in understanding and explaining how and why it works (which after all, is the golden objective of realist synthesis).

And if you’re wondering, it is summon first, because you would have to drop your ward anyway to summon, so you may as well do it first, and then use your ward for anything which your summoned creature cannot handle.   Although, if you’ve completed the Conjuration Master Quest, you get this guy, and let me tell you, not much gets past him:

I honour my Lord by Destroying you

Here they are: Intro to realist evaluation and synthesis_Workshop May 9th 2013

For those wanting examples of realist reviews, and to know more about methodology, we’ve created a new page on reading materials.

http://blogs.exeter.ac.uk/realisthive/reading-list/

The worlds of evaluation and research synthesis are peppered with talk of theory. Sometimes we grasp for it before actually conducting research, but all too often retrospectively to provide a veneer of credibility. Are we simply unconvinced of the merits of theory? Or afraid of what this hydra-headed beast might unleash? Or simply unsure of how to use it?

For the moment, let’s just consider the first – and use food to help think it through. How could we infer how a fine mushroom risotto was cooked simply by tasting it? As with all research, we’re there ‘after the event’ but need to understand what has gone on so that we have a basis for making decisions in the future. The age-old issue arises – what we can perceive (or measure) and what we are trying to understand are some way apart. But theory provides the bridge, albeit pock-marked and incomplete, that enables us to understand that clumped rice grains will only be inadequately coated in olive oil (= inconsistent texture) and hurriedly-added tepid stock does nothing for taste. Theory is no magic bullet, but through careful and critical application we can begin to build understanding of what went on – some might even say we get to the nub of things, the ‘generative mechanisms’. Some might also say that theory (the bridge between what we can perceive and ‘what is actually going on’) is only as strong as the critical community that challenges and refines that theory. I would agree with them.

We held a bake off competition at the Veysey Building today to raise money for Comic Relief.  Based on the feedback (empty plates, full tummies), it was a success, and it raised over £160, which is excellent.

But it got me thinking, is there a middle range theory to explain how a bake off actually works?  So I thought I’d have a blog…

What is the mechanism of action?  What causes a bake off to work?

The mechanism of judging offered participants an opportunity to get involved with the heart of the enterprise (no one left out, all have valuable contributions) in a way that ignored traditional roles and boundaries within a hierarchical organisation.  For a time, the usual rules about seniority, status or power were suspended as all cakes were set out for judging by all.  The powerful combination of ‘all included, all equal’ reflects a potent human need for belonging and balance.  Once it was clear to participants that this was how the ‘resources’ were offered, their reasoning was to get stuck in.

In what circumstances does a bake off work?

The distillation of our reasoning was simple: people like to eat cake, and at Veysey, people are quite passionate about the cake they eat (another conducive/responsive context?).  AND, being researchers, there is a degree of competitiveness and perfectionism within the organisation already, which would make the idea of a competition appealing to everyone. (another conducive/responsive context).

Who were the winners and losers of the bake off?

Apart from the Amazing Helen Davey and her Red Velvet Clown Cake (who was the ultimate winner), the outcome pattern was surprisingly one tailed – I would humbly suggest that everyone had a good outcome, whether baker, judge, winner or ‘loser’…  all played a part and could share in what was a successful morning of community together.  It would be interesting to understand the experience of those that did not attend, because in a sense, they lost out…but it is likely that the context was actually constraining their involvement, rather than enabling it.

The actual outcomes delivered are quite interesting, because they can be viewed in terms of the individual participants (eaten delicious cake, so feeling full), or in terms of the ’cause’ (Comic Relief).  It is likely that participants didn’t necessarily consider these two outcomes separately though: it is something about the magnified benefit of cake and giving that creates the ultimate outcome of a bake off, which is it makes people feel good.  Some participants will probably have participated because of the blending of the individual reward (cake fullness), and the realisation of a goal for a cause.  Raising money for charity is still considered a ‘good thing’, and this was once again born out today in the bake off. I am consistently surprised that even though belts are tighter than ever, people still give generously!  Well, belts have been loosened at Veysey today.

So, for the future, to generalise the findings of this morning, perhaps we could look at how the theory of ‘status suspension’ might apply in different contexts: we could test out this theory by creating a different opportunity to suspend status, but within the same context, which also held a reward (both altruistic and intrinsic), in order to see if the same outcomes occurred.  But I hope it won’t be to do with Cake anytime soon – I’m all caked out now!

Helen Davey’s cake.  Clowns – still scary!

Well, it’s not long now before the email containing our ‘umble plan to run a workshop on realist evaluation and synthesis wings it’s merry way across the inboxes of early career researchers and postgrad students from Humanities and Social Sciences as well as the Medical School…. soon, our plans will come to pass, and we shall, once and for all, be the MASTERS OF THE UNIVERSE!!

I have always wondered precisely what is meant by Early Career Researchers – is the plan behind calling those at grades E and F ECRs some form of trying to show that at some point, we shall reach middle age, and then older age, and then die off?  It seems a funny way of looking at things, particularly if you came to academia ‘late’ like I did.  I mean, I’m only (only!) 37, so I’m definitely not ‘early’ in my career, but this is my first academic job, so I guess I am an Early Academic Career Researcher… which is even more of a mouthful.  Either way, we are hoping that the Blog, Wiki and the Introductory session and book group will gather a broad range of interest from across the University, as well as across the grades of researchers.

Looking forward to meeting you…..