Evaluating Graduation

‍We’re in graduation season at Nottingham Trent University. Gowns, family photos, hats in the air, all of that stuff.

As I was helping out last week, handing out degree certificates, I was thinking “how do we know this worked?”. It seems like a simple question doesn’t it.

Except there is a bigger question that I couldn’t answer right away: “what is this all for?”

In between the lulls, I came up with at least four competing ideas about what graduation ceremonies achieve. There are undoubtedly more, but I want to stick with four for now.

And if graduation has 4 different purposes, then we also need to think about four different ‘causal pathways’. In order to evaluate with “graduation worked”, we need to consider the chain from the ‘activity’, to the ‘outcomes’, through to that ‘impact’. Is any of this based in theory, and then how do we evaluate from there?

So I thought I’d do that.

Thinking beyond graduation day

‍What is it for? Well, graduation doesn’t NEED to happen for you to get a degree. The final bit isn’t really about the cap and gown, and the stage, and the handshake. It’s ceremony.

And this is a good test case for evaluation because graduations happen because they’ve always been done. But it isn’t REQUIRED to pass your degree.

Students don’t even have to turn up. Those that don’t get their degree certificate in the post.

So why do we do it?

1.     Student / Consumer Satisfaction

Probably what we jump to first. We do graduations because students expect it, and because it makes them happy.

This is how we evaluate it a lot of the time, and a surface level answer to what graduation is for. Did it make the student happy? Were they satisfied?

Mapping it: A Theory of Change might go something along the lines of ‘graduation’ leads to immediate ‘positive emotional reaction’, leading to ‘increased satisfaction’ with university, ‘better perception’ of the university, with the ultimate impact of a student who has a strong ‘sense of identity’ as a graduate of that university.

Maybe throw something in there around positive impact on recruitment too, because those who have a better perception of the university might also recommend it.

This isn’t perfect, but something along those lines.

Evaluating it: A short post-event survey, split into service items (like timing, queuing, photography, quality of gowns – they were particularly good this year at NTU! – things that give you an idea of what specifically was good) and into things around meaning (like memorability, significance, pride – things that give you some idea of strength and longevity of the effect). Top it off with a ‘net-promoter score’ for good measure (you know, those ‘would you recommend this to a friend?’ questions).

This is classic consumer evaluation, but really, it misses the point doesn’t it?

My mini-Theory of Change doesn’t have any evidence behind it. This evaluation is easy and straightforward, but I don’t think it gets to the real reason we do graduations. So if your institution only evaluates graduation like this, I think it’s doing it wrong.

2.     Psychological Closure and Transition

Let’s think about this more deeply, and think about the psychology behind it.

Arnold van Gennep, an ethnographer from Ludwigsburg, came up with the idea of ‘Rites of Passage’. Basically, ceremonies like graduations help you transition from one social state to another. There are recent studies even now, like Anderson (2024) that explores what happened during Covid when graduates didn’t have the closure of a graduation ceremony.

So there is evidence of this.

Therefore when we think about ‘what is graduation for’, we can think about transition and closure. Did it help the student become a graduate? Are they ready for life as an adult?

Mapping it: A Theory of Change now goes along different lines. ‘Graduation’ leads to a ‘sense of closure’, greater ‘confidence’, and lower ‘anxiety’ about the future. This leads to the student have a stronger sense of post-university ‘self-identity’, and this leads greater successes in their professional (and maybe personal) lives.

Again, this needs refining, but we can see how this is already very different to the last one….

Evaluating it: Now we’ve changed out outcome, out evaluation needs to change too. Perhaps measures of ‘employability readiness’ or ‘wellbeing’ can be used. Maybe we can look at some quasi-experimental design, where we compare employment outcomes for those that attended graduation against those that didn’t come at all.

This isn’t perfect, because we have issues of self-selection here. But we already have a more sophisticated evaluation based on an actual theory.

3.      “You’re now part of our alumni network…”

I’ve been having a little look at the communications from a few universities, aimed at their graduating students online. Something keeps coming up: “you’re part of the alumni network now” and “you’re still part of the family”…

Graduation in some ways is about cementing students as part of the university, even now they’ve completed their course. Do graduates feel part of the family? Do they retain that sense of belonging?

Mapping it: Graduation here is about making sure students feel valued even after their courses is over. Feeling valued in the short term, leads to ‘sense of belonging’ and ‘strong attachment’ to the university, which in the long term can lead to students feeding back into the university through alumni events, mentoring, advocacy, or even giving and funding.

Evaluating it: This is a longitudinal relationship, and therefore the evaluation needs to be longitudinal too. Belonging measured before the ceremony, after the ceremony, and linked to alumni relations data at 5… 10 years… controlling for prior engagement.

This approach could be linked to one like ‘I Belong, Therefore, I Give? The Impact of Sense of Belonging on Graduate Student Alumni Engagement’ (Drezner and Pizmony-Levy, 2020).

This is recent research, and one that could form the basis of a stronger evaluation.

But it isn’t the main reason why I think graduations are so valued by universities, and are so important to students…

4.     Closing the gap

I was the first in my family to graduate from uni. For first-gen students, and for many young people who didn’t grow up in university towns and cities, this is something never experienced or seen. It was also the first time my dad wore a suit and my mum wore a nice dress, because of me.

But a graduation isn’t just about the graduate and their own psychology. It’s an external, relational, witnessed event. It certifies social mobility and success to a family, to a community.

I helped out with NTU graduation last week, and one of the moments that stuck with me was a young graduate showing her degree certificate to her younger siblings, pointing to her name, and watching them all want to touch this magical piece of paper.

That moment could be more than just a nice thing to see; it could be transformative. It might be the ‘slipstream effect’ (Wainwright & Watts, 2021) in practice.

Mapping it: A graduation that is structured to include family and community as witnesses, leads to a developed family narrative about education shifts. This leads to siblings and families ‘reimagining’ their own educational possibilities and futures, drawing on role-modelling. The ultimate impact is that siblings and younger relatives pursue HE, carried on the graduate’s ‘slipstream’.

Evaluating it: We might want to do this in two parts. Firstly, there is a quantitative side to this. Attendance and uptake, using POLAR or IMD data, and outcome data not just for the graduate but for siblings of the graduate. Can you see the outcome data for siblings, and compare against similar young people from that area?

Secondly, a qualitative element. Reflections from the gamily, focusing on significance of the event. For this evaluation, we care just as much about what the family of the graduate thinks about the day, as we do the graduate themselves.

Turning back to design

I want to take a moment here.

I think the latter is massively important. But if this is the case, that we want to bring more families in from low socioeconomic backgrounds, then we could arguably rethink how we do graduations.

For example, the cost of graduation might be working against the outcome. The Times Higher Education in 2024 found gown hire alone to be between £50 to £120 at several Russell Group unis.

£120. Blimey!

If one of the key outcomes of graduation ceremonies is to allow for everyone to participate, £120 to hire a gown is not going to do that!

Why this matters

The point of all of this isn’t about gowns.

I wanted to answer the question “how do we know graduation worked?” What I found is that you can’t answer that until you know what a graduation is for, and mapped out the theory. And then the evaluation radically changes.

These were four theories, but there will undoubtedly be more.

All of this isn’t specific to graduations. Every time we jump straight to ‘satisfaction’ as a measure, or just using the data we have to evaluate something, without thinking it through, then we risk confidently measuring the wrong thing completely.

Last week I was at NTU’s graduation. I saw satisfied students, I saw the closing of a chapter, NTU alumni being created, and I saw a family success story be written in real time.

Same event, different outcomes, different evaluations.