Teachers are often nervous when we hear school managers talk about using statistics to analyse performance. Our worries are not without reason: we have all heard anecdotal stories of the teacher whose GCSE classes ‘only’ get C grades being pushed out of the school, while the teachers who get A grades find themselves rapidly promoted. In these anecdotes, the Head ignores the fact that the ‘C’ teacher starts the course with a class of 24 who can barely read and write, and who go on to get E/F grades in everything else. Meanwhile, the ‘A’ teacher is the Head of Dept who hand-picks 6 bright sparks for his own classes and cruises through the course.
Hence the title of this post. Ultimately, you can abuse statistics to prove anything. However, with clever use, we can properly use statistics to help our students – and show that the ‘C’ teacher has done a fine job also.
We’ve all been doing this for years. We look at the Christmas exam papers and spot trends, don’t we? Class A all score full marks in Question 5, while Class B barely manage a coherent sentence – so we try to figure out what went wrong for Class B, and why? Are they a weak class, who are slow at getting through material, so this topic was missed? Did the class teacher not realise the topic was on the paper? Maybe the teacher just made a hash explaining of the topic?! When I am the one who has taught both classes, and class A clearly know something class B do not, I have to figure out whether I blundered or whether half of class B were off one day and nobody asked about homework. When we do this within our departments, it’s usually fine. We want to help each other. We want the kids to do as well as they can. If I can help the teacher in the next room with a topic they don’t teach very well, or if they can help me teach the topic I simply do not understand, then…. great! We’re all in it together.
In my worldview, of figuring out the topics you teach well and the topics that mess up the exam, there has been one piece missing (until now): exam board feedback. Exam boards might simply give you a list of grades, or maybe a module-by-module breakdown. They don’t tell you the questions your class messed up though. You know your class came out of the hall in June thinking it was a horrible paper, or that is was easy, or that all was well until the final question, but you don’t know how other schools did.
Over the years, I have marked or moderated the work of around 5,000 students from other schools. Even after one series, there are clear trends that examiners see: the topics that hardly anybody in the whole UK understands; the school where everyone is heading for an A until one particular topic trips the whole lot up; the schools where students have clearly rote-learned the standard textbook and not applied their knowledge to the scenario of the question; the schools that year-on-year produce fantastic work. Confidentiality and contracts prevent me from doing what I want to do sometimes: phone the school and talk the teacher through the paper – help a fellow professional plug the hole that pulls everyone down a grade.
Sometimes I have wished someone would talk me through the offerings my own people delivered up.
So, unless you are afraid of statistics, help is at hand (depending on the exam board you use). I do wish all boards would do this. These come from e-AQA, from my own class’s exams last summer – I can’t comment on how other boards do or don’t make this data available. For confidentiality of students, I am not identifying anyone, and removing anything that might let an individual’s details be worked out,
Firstly, we can look at how many of our people get a particular grade (there are none for ‘Our centre last year’ because we were with another board for this course):
Woohoo – all my people get A-C. But I know that already. It’s useful to know that we were miles ahead of ‘Similar centres’ and all of AQA. The page I took this graph from also shows ‘A’, ‘A to B’, ‘A to C’, ‘A to D’ and ‘A to E’.
Let’s look a little closer – the total score for the paper (green) and per module (the two in blue):
Great! We’re still beating everyone. So let’s look at the bottom module, per-question. I press ‘View components’ and see this…
(it goes on for another 9 questions, but I won’t put it all here).
Questions 1 and 2 give more good news… but something has gone badly wrong on question 3. Actually, it went wrong for the whole UK but we seem to have been worse-hit than others. At this point I can ‘View marks’ and see how each student did. I can see whether a handful of students’ poor marks dragged the class average down, or if the whole class made a mess of it. From the set of graphs for all questions, I can see whether the terrific performance is as good as it looks. Did they really outscore everyone on every question? Did they do so well across the paper, that one poor question didn’t affect things? Did they get away with poor knowledge of a third of the paper, through full marks in most other questions? Did they fluke high grades because they got lucky on the topics that came up?
Data like this gives me an idea of the topics I have taught well, and the topics I might not have not taught well. I can split the class into groups and see how the students who were in my GCSE class did in comparison to the people who only joined us for AS. Some people may have sleepless nights, wondering if the topic they rushed cost anyone a grade – this is not a good idea, as I don’t know too many teachers who get every single topic right. I don’t think I’ll have a sleepless night over this – but I will go to the paper, check the questions that gave problems, and figure out what happened.
One single mark is unlikely to affect the overall range of grades your class scores. But, a mark or two here and there, over four modules, could be significant. If I can figure out the topics that gave problems, and plug a hole or two, I can help someone’s future. If I modify how I teach the topic they all messed up on, I can help a lot of people’s futures.
