Is this car ‘good’…?
A couple of months ago, I attended an online event expertly delivered by Ruth Squire and the Evaluation Collective, about ethics in evaluation. At the heart of it, it wasn’t about an ‘ethical approval’ process, but about how you do evaluation ethically.
Ruth talked about academic philosopher Michael Scriven, who argued back in 1991 that value isn’t just one thing. He suggested that when we judge something, we’re often mixing up different ideas of value without realising it.
The more I thought about it, the more I realised this is a really helpful concept for people new to evaluation, and how to think about ethics in evaluation from the start.
So, let me explain it simply, by asking “is this car good?”. And let’s evaluate it as Scriven might.
This is a 1997 Nissan Micra (K11) with a 1.0ltr engine… what do you think?
On its merits
One way you can evaluate it, is by assessing its ‘merit’. This is how good something is based on its features or qualities.
The engine is small but efficient. It’s pretty tough and reliable. It can nip around a city really well because it is small and manoeuvrable.
But it has no electric windows, no air conditioning, no way to connect your phone, questionable back support in the seats, and the steering is really heavy.
So in terms of ‘merit’, it’s a mixed picture.
On its worth
Another way to evaluate it is by ‘worth’. Thinking about the cost or price of the car.
This particular car need is cheap. Really cheap. But also it needs some work, and may end up costing. It will fail its next MOT, and rust is becoming a real problem.
In terms of ‘worth’, it isn’t looking good.
On its value
A third way is to evaluate it on its ‘value’ or ‘meaning’. Considering what it means to people, or in its context.
This particular car was my first car. It isn’t just a Nissan Micra, her name is Molly.
I was 8 years old when my parents bought her new with all their savings, I remember sitting on the backseat when my mum test drove her.
When I passed my test at 17, Molly was mine. She took me on adventures, through France, up to north of Scotland, on day trips and the daily commute. She was freedom, she was fun, and a companion. I was absolutely gutted when she finally died in my mid-20s.
In terms of ‘meaning’, Molly scores very high.
With friends on a journey years ago, taking turns to drive. …it also lacked legroom!
Why does this matter for evaluation?
I’m banging on about my car, why does this matter? Let’s think about a programme, project, or intervention. Something we deliver to students and for students. You’re being asked to evaluate it.
If someone asks, “Was it good?”, what are they really asking?
Was it good value for money? (worth)
Was it well designed and delivered? (merit)
Did it matter to the people involved? (meaning)
Different stakeholders care about different answers.
Senior leaders may focus on cost.
Practitioners may focus on quality.
Participants may focus on experience and change.
Ethically, evaluation becomes risky when we only answer one of these questions and pretend it tells the whole story. We could scrap a programme that students loved, because we only think about ROI. Or, we continue to do something that a department likes to do, but in reality is badly designed and could be done better.
A simplified takeaway
Evaluation isn’t just about measuring outcomes.
It’s about being clear what kind of judgement we’re making. And sometimes, it is ethically challenging.
When we simplify evaluation, we’re not lowering standards or minimising this. We’re making our thinking visible.
Before asking “Was this good?”, try asking “Good, in what way, and for who?” That small shift can make evaluation fairer, clearer, and more useful.
Just like evaluating Molly Micra, sometimes the biggest impact isn’t the first thing you think about.
