Improving the Office for Students’ Standards of Evidence

Higher Education institutions, when thinking about evaluation, will first turn to the Office for Students (OfS) Standards of Evidence, as a guide for how to do it.

If you haven’t seen it, it’s really simple, and a great starting point for what evaluation looks like.

It sorts evaluation into three types: ‘Narrative’, ‘Empirical Enquiry’, and ‘Causality’. The OfS suggests that these aren't meant to be ranked. That you shouldn't chase the fancy causal design over a well-run empirical one just because it is ranked higher.

Except in practice, that isn’t how it works.

Researchers such as Fryer (2023) have pointed out that in practice, it functions as a hierarchy. That it nudges everyone towards chasing the "best" method rather than the right one for the job. And a very quantitative one at that.

And I have to reiterate it: The OfS have been explicit here that it isn’t hierarchical (see the image! It’s right there!)

But it still is talked about in that way. I think this is a problem, and one with an easy fix. So, this is my proposal for a simple way to improve the OfS Standards of Evidence.

But first…

Let me tell you about my decking…

The other week, a good friend of mine, one Ed Foster, helped me to rebuild my knackered decking in my back garden.

And of course, there is a lesson in evaluation and how much effort you put in to getting things done to a high standard.

Like one little consideration in the job, is spacing the decking boards apart. You need a consistent gap between each one, or it looks messy and starts trapping water. We didn't get the tape measure out or spend time researching ‘the best gap size between decking boards’. Ed found a flat pebble from the garden, decided it was roughly the right width, and we used that as our spacer for the whole deck.

Easy job.

Laying the boards themselves was a bigger job though.

Every board had to be measured, screwed in properly, kept level with the one before it, and many had to be cut to the right size. It wasn't super complicated, but it needed measurement this time. A pencil, one of those square things that makes sure your line is properly perpendicular, a standard saw…

We took a bit more time and more work and planning went into that bit.

Then there was the sub-frame.

The actual structure the boards sit on, and the part carrying all the weight. Get the spacing wrong and it’s annoying, but easily fixable. Get the sub-frame wrong, and the whole deck is in trouble. You're pulling the whole thing up and starting again. So that got the most attention by a long way. Measuring everything properly, getting the right sort of wood, making sure everything is screwed in properly with the right stuff… etc. etc.

It took the most time, the most thought, and the most care.

These were three different jobs, that lead to the same goal, but with three very different levels of care, thought, effort and priority. If I was meticulous and rigorous with everything, it would have taken months and cost a small fortune. If I had just eyeballed the whole thing, it would have not worked.

We prioritised and were proportionate.

It’s obvious isn’t it?

So let’s apply that logic to how universities evaluate access and participation work.

Borrowing from the Treasury

Earlier in the year, the Government reissued its ‘Magenta Book’. This is the Government’s own evaluation guidance. In it, instead of ranking evidence by method (like the OfS), it judges against four measures: usefulness, credibility, robustness, and proportionality.

That last word stood out to me.

It means matching the effort that you put in, against the size and risks of the project. A one-off, low-cost activity doesn’t need a control group, in the same way that spacing my decking boards didn’t need measuring with lasers to the nearest 0.001mm. A one-off light touch event with a handful of students doesn’t need an RCT or a 5 year longitudinal evaluation.

The idea of proportionality, I believe, is missing from the OfS Standards of Evidence. Adding this specifically changes their guidance from three ranked (but also not ranked) types, to a tiered list that builds on each other, matched to how big of a deal the thing you’re evaluating is.

I also think there are generally three types of activities that are delivered to students, from Student Support, through to Library Services, to Outreach, to what happens in the classroom.

1. Light touch interventions: The one-off / low cost and low commitment intervention. These are standalone, with no real follow-up. They don’t take up much staff time per student. If they stopped running, no senior leaders would be asking hard questions about it.
   This could be your info stand at a careers fair, a one-off webinar, or drop-in for a coffee and a chat type thing.

2. Core recurring provision: The annual funded stuff that is structured, and often across multiple sessions. Usually targeted at a specific cohort rather than ‘whoever is there’, and involves enough staff time that someone’s job depends on it running well. It will show up in annual planning, maybe KPIs and team or departmental monitoring.
   This could be your mentoring schemes, workshop series, ‘getting students in the school classroom’ type stuff.

3. Flagship programmes: The major, strategically significant, multi-year, and probably involved ringfenced funding and/or external partnerships. This is the stuff that Universities name in their Access and Participation Plans (APPs), and is a full commitment. This is high-budget, high visibility, and carries a reputational risk if it doesn’t work.
   This could be your programme that addresses an Ethnicity Degree Awarding Gap, or new support bursary for care-experienced students, or full progression programme that includes a residential and regular trips for kids from year 9 through to sixth form.

The key dividing line here isn’t necessarily cost, or student numbers, or staff time. It’s what happens if it doesn’t work or gets cut completely.

The light touch interventions, nobody really notices outside the session itself if it stops. The core provision, cutting this starts to affect staff and the cohort more widely may lose something. The flagship programmes getting axed requires a conversation with funders, governance, and is possibly a headline.

Improving the Standards

If we treat all interventions as the same, from an evaluation perspective, we become overwhelmed. So we need a judgement.

This thinking of course isn’t new. This judgement call is happening across the sector. But there is a disparity between guidance and evaluation training, and practice. And closing this is important.

We need to be proportionate, and that needs baking into the standards, not just an after-thought.

So this is what I’m proposing. A tiered system, that is proportionate.

Through this new ‘standard’ you have to consider the intervention type, to consider proportionality.

Through this, doing an RCT or complex evaluation isn’t always ‘better’. In fact, if you’re doing an RCT ‘disproportionately’, then perhaps it isn’t the best use of your time or resource.

The website is called ‘Impact Simplified’… isn’t this the opposite?

Yeah I can see that argument.

The whole point of Impact Simplified is stripping evaluation back to what is usable and understandable to a wider audience.

Swapping a simple three-type model for three tiers with an added layer of ‘proportionality’ looks (on the face of it) like I’ve just made a job bigger and more complicated.

I disagree.

Consider a recipe for dauphinoise potatoes (yes, I’m coming back to my fave spuds again). It’s only a few ingredients: cream, milk, garlic, potatoes, cheese. Sounds simple.

But hand someone a recipe that just says “layer potatoes, add milk and cream, bake”, then you’re going to get a lumpy disaster. This doesn’t tell you about how thin to slice the potatoes, or how to season it, or what kind of cheese [gruyere, because I’m fancy], or how long to bake and at what temperature.

More explanation and more guidance doesn’t make it more complicated. It makes the process more simple to understand.

“Good doesn't mean complicated, it means done well.”

So what do you think? Do we need to update the standards?