How to find a dog, and lessons for data collection
Here’s a riddle related to ‘outcome measures’.
Elfrida is a Swedish tax inspector. Her job is to locate people who were not paying taxes for dog ownership.
What does she do to track the dogs down?
Riddle from the Lateral Podcast > Episode 15
Why am I talking about taxing dogs?
In the riddle, Elfrida needs to identify specific people. In research and evaluation, we often need to identify specific people based on a certain characteristic, trait, belief, or demographic.
So let’s have think about how she could do it?
Methods
Let’s think about some traditional methodologies.
When we talk about outcomes in evaluation, the conversation often jumps straight to one place:
“We should survey people.”
Elfrida could send a survey out to the people of Stockholm asking “do you own a dog?” But what are the problems with this?
It’s difficult to get a survey distributed
People may not answer honestly
Response rates will probably be low!
Maybe we need something more direct. Elfrida could run interviews, asking people about their pets. What are the problems with this?
It takes time to arrange and conduct
It takes skill to get the answers you need
People still may not answer honestly!
Perhaps there is existing evidence out there. Elfrida could look administrative data, like vet registration or pet insurance information. What are the problems with this?
Getting a data sharing agreement in place
Trawling through massive data sets
Perhaps only some dogs are registered!
What about something more covert. Elfrida could gather spending data on dog food and chew toys. This could be a ‘proxy measure’ for dog ownership. What are the problems with this?
Gathering this data is very difficult
People get these items from multiple sources
Perhaps people are buying these things as gifts!
The answer
Sometimes the best data to collect is the most simple.
Elfrida visited addresses where the dogs might be present… and started barking!
By barking loudly and pretending to be a dog, the dogs very often started barking back. This is quick, simple, effective!
This is the lesson for evaluation measures. Good evaluations don’t start with “what survey would work here?”, it starts with “what is the change we are trying to measure, and what is the most direct way to measure it?”
If you want to know whether a programme improves academic skills, maybe don’t just ask students whether they feel better. Look at how they structure assignments, use feedback, or manage deadlines.
That’s what simple and effective evaluation looks like.
