The Problems With Educational Research
Read, Think, Digest, Reflect, but don't swallow. That's my advice.
A research study was done on those who played recreational Russian roulette. As part of the study, researchers interviewed those who played the game. It was found that 100% who played, survived. It was concluded that Russian roulette was not that dangerous a game and didn’t warrant any action by authorities to curb the enthusiasm of those wishing to play.
For many years, I’ve published a newsletter sharing educational research, opinions, articles and more. Prior to that newsletter, many other newsletters. As someone working in education, I’ve always thought it a priority, to keep abreast with the latest information that might inform teaching and learning. I think we all should do so - as best as we can. I’ve also taught courses in educational research and design and have done my own research. See this recent article about research insights in SLA.
This commitment of mine is despite so much (at the time), in vogue, “sound” educational research later becoming dustbin fodder. Just to mention a few off the top of my head; Duckworth’s “grit”, Gardner’s “learning styles”, Dwerk’s “growth mindset”. We can also throw in things like mindfulness, personalization, the computer learning effect, mobile learning, basal readers, colored sound charts, flashcard learning and so many more.
Learning is messy. Teaching too. That’s the reason in a nutshell. It’s also why so much educational research is not reproducible. And that’s a huge problem. It’s basically, flip a coin to get approximate replication of one study. Zero educational studies have been precisely replicated, the lowest level among all the social sciences. On top of all that - we have only one-fifth of one percent (0.20) of all educational research papers being actual replication studies.
It’s a research shit show. Further complicated by all the quick shares of research findings online and calls of “science of this” “science of that” (hey! why isn’t there a science of science?). Few in education seemingly want to admit how complex it is to bottle human behavior - so full of it (materialism) our beliefs are.
I remember maybe mid 2010s, so many being excited by newly published research in the UK that found students learned better when in sock feet. I’m sure a good number of teachers tried it. It can all become rather silly, this belief that one small change will result in better learning outcomes. It doesn’t work like that.
Perhaps there are just some things we can’t bottle up, count out, put a circle around? Perhaps, it is us teachers, through the sweet nectar of time and the chewy texture of experience who might offer up the best and most sound digestion and “conclusions” - to the benefit of our students?
So, beware of believing the next presenter, at your next conference when she/he says that “X is what we need more of in education and our classrooms.”
But still teachers need to distill, read, reflect on what is researched but with the foreknowledge that when it comes to human behavior, there is a huge set of complex interdependencies at work. You aren’t able to isolate variables and know for sure, over human groups (and even individuals too) what is zooming what.
There are the normal things that make any piece of educational research problematic and which should be used to filter out responsibly any direct conclusions of studies.
Here are a few:
Small sample sizes.
Confounding variables and factors not accounted for.
Errors in data collection and sampling.
Poor contextualization and poor defining of variables.
Errors of aggregation and omission.Confounding of correlation and causation
Invalid statistical significance.Wrong tools, research design and measurement criteria
Cognitive bias. Self-selection errors of relevant data.
Errors of correlation. Improper claims.
Data visualization errors (incorrect scaling, wrong graph, data omission).
False, unvalidated conclusions.
I would add a number of other factors, outside the actual pragmatic realm of any study that we should consider. They have an impact on educational research in general.
Sponsor bias - Especially in ed-tech, companies pay (they say sponsor) research and of course, this in itself injects bias and enacts all kinds of problems into the research itself.
Researcher bias - It’s a kind of endowment effect. When you are wishing for a result, your own inner volition can alter the study. Just observing, measuring makes one part of the study, changing it. Think of it like Heisenburg’s Uncertainty Principle applied to research.
Publish or Perish - So much pressure to publish which clutters up the mind-space of those needing to shuffle through so much noise. It also creates pressure to fudge, to even falsify research. AI, the new plagarism tool on the block, further accelerators this whole dynamic.
To circle back to where I started - the Russian roulette joke. Have you heard of the fighter plane problem? I think it highlights another salient problem with research - knowing how to evaluate not just what evidence was gathered but also what wasn’t gathered. Call it the “Left Out Effect”.
Often what is missing IS the most important findings of any research (but rarely a focus or mentioned).
During World War II, fighter planes would come back from battle with bullet holes. The Allies initially sought to strengthen the most commonly damaged parts of the planes to increase combat survivability. A mathematician, Abraham Wald, pointed out that perhaps the reason certain areas of the planes weren’t covered in bullet holes was that planes that were shot in certain critical areas did not return. This insight led to the armor being re-enforced on the parts of returning planes where there were no bullet holes. This wisdom was also beneficially applied to the Skyraider during the Korean War.
I think often in research, there is a lot left out. No accounted for. Given how multi-faceted, interconnected, filled with socio-cultural influences, so so complex, our human behavior is.
So what should a poor teacher do, faced with the lying monster that is most educational research? Well, three things I might mention.
Seek out someone informed, a statistician that is well educated, versed, credible, and without any skin in the game (conclusions). This cancels out most of the media (that do a horrid job of displaying biased data results), and official organizations (who want to show data to gain a specific public response).
Look at the data yourself. Think about it. As I said, I’m no rocket scientist but if you take a common sense approach and really look beyond the tricks the magician tries to play and distract you with - you will reach your own more valid conclusions generally.
Let research inform your classroom practices over time but never control them. Teaching isn’t like building with lego. Trust your gut instincts and what works with your students. They are human beings, not experimental subjects. Don’t rush to implement the next magic silver educational bullet (like AI - with NO replicated educational research to note). But do tinker, when possible, test your own hypotheses and be a researcher in action.
You can’t bottle teaching and its magic. It just has to happen, occur, in the real, messy, huzzah of the classroom battle. What do you think?
