A Nightmare in the Examination Hall

GCSEs! They’re terrible, aren’t they? Unfit for purpose? They must be, because the press told us so (ably assisted by various strategically placed press releases from the Education Secretary). Children the country over are suffering after unfair changes to grade boundaries left thousands with a D when previous benchmarks would have left them with a C. Proof, if proof be need be, that the entire GCSE system (introduced in 1988 by the Conservative Party, of all people) is entirely corrupt and unfair, right?

What EXACTLY is the problem this year?

FryExams

GCSEs are far from perfect, but as usual, the press (and the government) were (perhaps deliberately) telling a very simplistic and generalised version of what was going on.

According to my scouring of the TES forum on results day, and government regulator Ofqual’s official report, the issue of the changed grade boundaries affected two out of three GCSE English qualifications only. English Literature was unaffected, while English Language and English Language and Literature had problems. But only at the Foundation (lower) Tier, apparently (all those worried about introducing a “two-tier education system” might want to remember that GCSEs already do this). And, of three major exam boards across England, only from two of them (“primarily AQA and Edexcel”, says Ofqual).

To put that into perspective, that means that, out of dozens of subjects being examined, this problem affected only one. And that one has, in essence, eighteen separate qualifications (three English qualifications across three major boards, Foundation and Higher Tier for each), of which four were at fault. And each of those four was made up of three modules, not all of which had the grade boundaries dramatically shifted. Suddenly doesn’t look like the damning critique it appeared, does it?

The other issue that all the papers fail to mention is that these particular GCSEs were being awarded for the first time this year, initially by a small group in January then a much larger one in June. Under such circumstances, it’s fairly common for government regulator Ofqual (and their predecessors QCA) to send an observer to the awarding meetings where grade boundaries are decided, in order to monitor standards.

Ofqual’s initial assertion as to the reason for this issue is that the standard was set wrongly in January. It suggests that Ofqual weren’t properly monitoring the awarding in that first series for the new qualifications, which would be unusual. Equally, the boards concerned must have some culpability for setting the boundaries generously themselves, but Ofqual’s monitoring of this is the final arbiter, and the very reason for its existence.

This rather gives the impression that they allowed the first awarding of a new qualification to either be monitored sloppily, or not monitored at all. Ofqual is a fairly new and untried regulator, rushed into existence with alarming haste by the incoming Coalition government in 2010. With this in mind, you start wondering whether it’s the exams that are the problem, or the purported guardian of their standards. Of course, that’s all a bit fiddly for a big, emotive press story about children being unfairly treated by the thousand, and doesn’t fit the political narrative.

What should we do, Mr Gove?

clip_image002

So, after a couple of weeks scandal, Mr Gove (Education Secretary and part time Pob lookalike) has given us his verdict on What Should Be Done with GCSEs. And unsurprisingly, his judgement based on all the evidence is… to do what he always said he wanted to do anyway.

So, a new ‘English Baccalaureate’ (must be good, it sounds classy), comprising the core subjects of English, Maths and the Sciences, each to be tested in one humongous three hour exam at the age of sixteen, with no more coursework. And each subject to be administered by only one exam board each, to combat the (apparent) problem of competition driving standards lower.

Longtime readers of this blog will know that I’m no fan of the current government (not that I have a lot of time for the Opposition either), but taking a step back from partisan politics, is any of this a Good Thing? And more pertinently, if it is, for whom is it Good?

Board to death

Evil Exams

To take the latter point of Gove’s plans first – no more competition between exam boards? I actually think that’s rather a good idea. It’s a bit of a first for a Tory minister to acknowledge that the great god competition actually lowers standards in any situation; perhaps they could try extending that philosophy to the likes of water supplies, railways, bus services…

Still, I digress. It always seemed a nonsense for any real competition to exist when all of the competitors must, essentially, supply the same product meeting the same standards. The press narrative for a couple of years now has been that boards can only compete by offering “easier” exams, thereby giving schools a greater proportion of good results and a better place on the league table.

This is, generally speaking, bullshit. When the government’s standards regulator is doing its job properly, it must ensure that all qualifications in the same subject at the same level offer a parity of challenge. Put simply, if anyone’s caught offering an exam that’s “easier” than anyone else, they face potentially losing the ability to offer it at all. It’s quite common for disenchanted schools, facing a year of bad results, to take their business to another board – only to find next year’s results just as bad, if not worse.

So if all exams are the same, how can you have competition? It boils down to other areas; customer support, teacher training, learner resources and so on. The quest for each board to better the others here, with a finite budget, is what can lead to a stretching of resources and consequent problems the like of which we’ve already seen.

The elimination of competition should therefore be a Good Thing. And so it is, but only in part – boards will still have to compete to be the only one offering each subject at GCSE level. My preference, discussed in a previous post, would be for one board covering all subjects across the country, a system which works well in other countries such as Australia.

Still, competition every few years to offer a subject is better than competition all the bloody time, with each board mercilessly trying to grab a bigger slice of the market. The worry is going to be the initial scramble for licences, particularly with players like Edexcel, which has the financial might of its parent company, multinational publisher Pearson, behind it.

In order to be fair, the process of settling who gets to offer which subject absolutely must be completely transparent and open to public scrutiny. Edexcel’s status as part of a profit-driven multinational gives them an unfair advantage over not-for-profit boards like OCR. And in other areas of the Coalition’s frenetic quest to outsource all things public, we’ve seen private companies like Pearson assert the mantra of “commercial confidentiality” to cover all manner of sins in their negotiations. If this isn’t to be another case of ‘lobbying’ (read ‘paying off the minister concerned with a promise of a juicy directorship on retirement’), the process must be entirely open to scrutiny and investigation.

OK, OK… but what about the exams?

FinalExam

When I was sixteen, I did O Levels – which worked in just the way Mr Gove is so keen on. I wasn’t convinced of their validity even then. A massive, nerve-wracking exam taken after weeks of frantic revision really only assesses what you’ve reminded yourself of recently and can remember on the day. GCSEs, while far from perfect, were designed to combat this with a process of continual assessment throughout the course, introducing the element of coursework to counter the criticism that plenty of intelligent people aren’t actually that good at exams.

Traditionalists have always had a bit of a problem with coursework; and in some ways they have a point. Mainly done without supervision, it was particularly open to plagiarism, a problem that’s intensified with the rise of the internet. The worry now is that entire coursework essays can be cribbed from Wikipedia; or even that certain, ahem, unscrupulous online companies actually offer to do it for you – for a fee, of course.

A halfway decent teacher, though, should be able to spot if work he/she is marking is written by someone other than the pupil they’ve been teaching for the past couple of years. If, that is, they’re not completely frazzled by their workload. Because for teachers, the problem is that coursework effectively means they’re marking students’ exams themselves, and that’s a lot of work – especially in larger schools, where the marking must be moderated by a more senior teacher and sometimes revisited if it’s not up to scratch.

The problem of plagiarism, at least, was supposed to be addressed by the introduction, in these new GCSEs, of ‘Controlled Assessment’ – basically doing coursework under supervised classroom conditions. Being a major change, it caused a lot of disquiet in the teaching professions, but it could have been a change for the better. Sadly, we’ll never know, as it was condemned for replacement after just one year due to the combination of press furore and political ambition. It may have a chance to prove itself in the next couple of years, as the ‘English Baccalaureate’ isn’t due to start until 2015, but its fate is already sealed.

So, assessment will go back to one, externally marked , terminal exam for each subject. I’m sure teachers will be very happy at the reduction in their already massive workload that will result from removing internally assessed work. But as a former exam board employee, I can testify that there was already a huge problem recruiting examiners for the examined units that already exist. Remove internally assessed ‘coursework’, and whatever board/s is/ are left will need many many more examiners.

Given the difficulty recruiting enough for the current level of externally marked work, I can see this being a logistical nightmare. Possibly the reduced workload caused by removing internal assessment will alleviate pressure on teachers, but I’m far from sure it will spur them on to become external examiners. And so Gove’s much-loved final exams may find themselves with a significant paucity of people to mark them. If you try to get exams marked without a sufficient amount of examiners, that’s when standards really suffer.

More generally, I’m not so sure about Gove’s emphasis on memorizing facts, figures and dates. Rote learning is important, of course – you can’t build an argument without facts to construct it from. But I worry that he’d rather have schoolchildren reciting the list of English monarchs without ever thinking about history.

The crux of it is that, while GCSEs could certainly have done with some fundamental reform, Gove’s changes simply push the system back to what he presumably fondly remembers from the 1950s. Hearkening back to a non-existent ‘Golden Age’ is certainly no basis for a programme of education – it’s been tried already, and the world has moved on.

So what should be done?

Abertay_Space_School_AR

I think there’s a real need to have a proper debate about the fundamentals of assessment – what we’re trying to achieve/quantify and how – going down to the absolute basics rather than modifying existing systems or hearkening back nostalgically to earlier ones. We need to properly challenge received wisdom on this issue, and do it entirely separately of political ideology.

For a start, since it’s been mooted that all children should stay in education until the age of eighteen (keeping them off those pesky unemployment registers), do we need a terminal exam at sixteen at all? GCSEs, like O Levels before them, were meant to quantify achievement at the level when children might leave school and go to work. If they’re not doing that, is there any point having them? Other countries, whose children stay in full time education until eighteen, manage perfectly well with tests taken at that point.

Which then leads us on to the question of A Levels. Among other things, GCSEs are used as a measure of whether a student is apt enough to take an A Level in a particular subject. But students don’t take A Levels in every subject; if terminal tests are taken at eighteen, they would necessarily include subjects that might not otherwise have been taken. Not everyone does Maths or English at A Level, for instance, but if all testing happened at the age of eighteen, they would have to. So that would render A Levels redundant too.

Which then, logically, brings us to Higher Education. With A Levels gone, how will universities assess the ability of their applicants? There’s already a problem that universities have to judge on the basis of predicted grades rather than actual results, and for years the idea has been floated of issuing results earlier, to give a more concrete idea of prospective students’ abilities. In practice, it’s unworkable – marking periods are already crushingly short, and to issue results significantly earlier would mean taking the actual exams much earlier, leaving less time to teach the course.

But maybe we shouldn’t assess by testing at all. Maybe there should be some other process of continual assessment throughout children’s schooling, from primary school onwards. And while we’re about it, do we need schools to be divided into a primary and secondary model at all? Again, other countries do it differently, some with more grades of school, some with less.

Also, should tests (if we have them) be norm-referenced (based on percentages of each cohort getting certain grades) or criterion-referenced (based on how well you actually know the subject you’re being tested on)? And why have grades in their current form? It’s always seemed unfair that a difference of one mark can move students from that all-important C to the doom-laden D. Why not express results in percentages of marks gained, as some countries do?

These are all questions that need to be asked. And ideally they need to be asked by educational experts, and not politicians. Labour’s Shadow Education Secretary Stephen Twigg has pledged that, should Labour return to power in 2015 (which is looking increasingly likely), they won’t implement Gove’s proposed changes at all.

That might be good news for those who cleave to the current system (which may not be a good thing either), but it means that for several years the entire educational system will be in turmoil, exam boards frantically designing new qualifications and tendering for licences to deliver them, while the poor overworked teachers must yet again begin training to deliver a new style of course – for the second time in three years. As always, the first group of students to take the test will be terrified of that leap into the unknown. And all for naught, if Labour get in and Twigg keeps his word.

It’s the clearest illustration ever of why politicians should be kept out of education altogether. Apart from the fact that they tend to know nothing about the subject, the constant demand to imprint your political ideology onto the education system means that it changes every time the government does, often for the worse. Teachers never know whether they’ll need retraining every five years, while students end up with incompatible results from completely different qualifications, that offer little comparability to prospective employers.

So if we really want reform, and we want it for the better, let’s keep political ideology out of it altogether and leave it to the experts – teachers, academics, you know, people who actually do the educating. Because Gove’s time trip to 1956 doesn’t strike me as much of an improvement.

The trouble with exams…

Gove

Those hard hitting undercover reporters at the Telegraph have been at it again today. After nearly scoring rather an own goal in the Murdoch Sky takeover by entrapping Vince Cable into saying inflammatory things with two pretty young ladies, it seems they’ve now been looking for another potential long running story by latching onto the perceived corruption of the British exam system. Their undercover reporters went to “13 meetings organised by (exam) boards used by English schools”, looking for evidence of what can be painted as corruption. And guess what? After looking pretty hard, it seems that they’ve found some.

The video on the Telegraph website, secretly filmed at one of Welsh exam board WJEC’s teacher training sessions, certainly looks damning. In it, senior examiners appear to tell the attending teachers what areas of the GCSE History syllabus will be covered in the upcoming exam paper. Because this fits the narrative the Telegraph are trying to shape, they comment that their investigations showed this to be a routine occurrence; that “teachers were routinely given information about future questions, areas of the syllabus that would be assessed and specific words or facts students must use to answer in questions to win marks”.

Fresh from his somewhat bizarre expenditure in sending schools across the country new copies of the King James Bible with a foreword he wrote (perhaps at God’s urging), chinless Education Secretary Michael Gove has been quick to leap onto a potentially non-controversial (for the government) issue about which he can be seen to be “doing something”. Eager for any opportunity to get his less than aesthetically pleasing mug into the papers, Gove has commented that such incidents "confirm that the current system is discredited", and has ordered new qualification standards watchdog Ofqual to conduct an investigation, the results of which may, politically, be a foregone conclusion.

The trouble is that, like so many recent media storms, this isn’t as simple an issue as the Telegraph are making out, and their pejorative language isn’t helping to clarify the issue. The sinister, X-Files-style air of conspiracy is generally enhanced when they claim that what they’ve been investigating was “series of secretive exam seminars, which are thought to have rapidly grown in popularity in recent years”, and for which shady examiners charge “up to £230 a day”.

Speaking as someone who used to work for one of the country’s leading exam boards, and administered one of these qualifications directly, I’m afraid I have to pour a little cold water over the Telegraph’s Oliver Stone-ish picture of a sinister cabal of exam boards and examiners conducting backroom deals in anonymous hotels. The “secretive exam seminars” they refer to are in fact very standard training days for teachers in how best to teach the courses prescribed by the board. Every exam board uses them, and the obvious advantage of having senior examiners present the courses is that they generally set the standards for the marking, and so are in the best position to advise on how students can get good marks.

Yes, these ‘sinister’ meetings can indeed cost £200 or so a day (though better financed exam boards sometimes offer them gratis). But the Telegraph story’s implication that this is some kind of bribe trousered by corrupt examiners is, frankly, bollocks. It’s charged by the boards, goes into the overall budget for training and marketing, and used to fund further training programmes. It is not, however much the Telegraph would like it to be, used by corrupt examiners to fly back and forth to shadowy meetings held by the Bilderberg Group, the Masons and the Illuminati.

But let’s look a little closer at the video evidence of the particular case that has sparked so much fury this morning. Are these examiners, as the Telegraph implies and so many people have clearly inferred, stepping over a line and revealing the questions in the upcoming exam to the privileged few teachers whose schools are prepared to fork out the training money?

Well, as with the Clarkson debacle last week, the selective clips used in some news outlets (including the BBC) certainly make it appear that way. But to give the Telegraph credit, they’ve been even-handed enough in posting a much longer clip that gives a better sense of context to the remarks. The oft-quoted soundbite (it’s in big letters on the Telegraph’s front page) is “We’re cheating. We’re telling you the cycle. Probably the regulator will tell us off.” That certainly sounds like someone who knows he’s doing wrong.

But if you look at the whole clip, it’s clear that the trainer is telling teachers that a particular topic is likely to come up, rather than a specific question on that topic. At this point, I should mention (as none of the press have) that every qualification is underpinned by a very carefully worded exam board document called a ‘specification’. Having been involved in developing and drafting these for new qualifications myself, I can tell you that they are obliged to cover the exact details of the course, and that no exam question can be set on a topic outside the areas of study contained therein.

Now I’m not familiar with the exact details of the WJEC History GCSE specification, but judging by the clip, this is a compulsory question which may be asked on one of three potential topics. As topics cannot be repeated from year to year, and there are three of them, anyone could check the past papers (freely available on boards’ websites, incidentally) and make a reasoned guess as to how these three topics will cycle round. In this case, these are newish qualifications, though, so the information couldn’t be surmised yet. But it could be within a year or so.

Undoubtedly this examiner is crossing a line in explicitly spelling it out to teachers, which illustrates how amazingly careful you have to be in wording what you say. But what he’s doing is simply telling the attendees what they could, in all likelihood, have worked out for themselves. It’s not the same as telling them the actual question, which is likely to be a bit more specific than “write something about Germany between 1933 and 1939”.

One teacher then asks whether, as educators, they should be teaching the children all the topics, rather than just the ones likely to crop up in the exams. And the trainer says that in an ideal world, they would indeed. But he acknowledges that teachers, driven by the tyranny of league tables and the political pressure to improve results year on year, may well find this an impossible demand; despite sailing way over the line, he’s trying to help – and these are the sort of questions teachers always ask in these training sessions, for precisely these reasons.

The second presenter is trying to spell out a section relating to modern US Presidents, and has incurred the wrath of the Telegraph by saying “off the record” that no question is likely to come up relating to “the Iraq war”. Again, this is clearly crossing a line, but not to the extent that it might appear. The examiner has just said that this section “now extends to 2000”, and points out the unlikelihood of questions coming up relating to “Clinton or Bush”. OK, it sounds as though the paper should cover Clinton. But which Bush, and which Iraq war? I’ll agree, the most likely interpretation is the ones from 1991, but there’s some doubt there.

Still, having said that, both trainers do seem to cross a line between helpful hints and actually breaching confidentiality. As Chief/Principal Examiners, they would be the ones responsible for setting the questions, and would have the necessary insider knowledge; this is clearly grounds for disciplinary action.

But speaking as someone who’s organised and on occasion presented such training, this debacle shows very clearly how careful one has to be, as a board representative, in wording what you say. Teachers, under pressure to deliver good results, often leap on potentially different interpretations of training speeches or even specifications themselves to justify their claims that markers have treated their students unfairly, and in representing an exam board you have to be conscious of the fact that anything you say is likely to be dissected with the sort of linguistic attention rarely seen outside a legal chamber. I’m not sure that what we’re seeing with these two trainers is outright corruption; more, it seems to be incompetence at knowing how far they can go in their statements as representatives of their exam board.

All of this, however, has been leapt on by the Telegraph and now our esteemed Education Secretary as evidence that the entire system of exam boards is riddled with corruption, and must be fundamentally reformed. The first contention is clearly nonsense – this is one qualification at one level from one (fairly small) exam board. WJEC don’t have the dominance of AQA, OCR or Edexcel, but even they presumably offer dozens, if not hundreds of qualifications at each of a variety of levels, each comprising multiple modules with their own exam papers. Discovering this kind of indiscretion in two senior examiners is like saying that all male film stars must be gay on the evidence that Rupert Everett and Neil Patrick Harris are.

But the idea that the system could do with fundamental reform? I actually think there might be something to that. The trouble with this kind of media shitstorm is the tendency for politicians (who usually have no personal experience of the issues surrounding education) to act in a knee jerk way and change the system for change’s sake, without considering whether the changes are the right ones. Every incoming government (and this one is no exception) wants to radically change the education system; in part, it’s to stamp their own authority in it, and in part it reinforces the standard party mantra that everything the other party did must be wrong. But that doesn’t mean that what they’re doing is right. As Sir Humphrey once spelled out the politician’s creed in Yes, Minister: “Something must be done. This is something. Therefore we must do it”. Or, more populist, Jeff Goldblum’s admonition to Richard Attenborough in Jurassic Park, “your scientists were so preoccupied with whether they could that they didn’t stop to think if they should”.

The most obvious bone of contention is that there are three (major) exam boards, and they’re in competition with each other. This should, logically, appeal to the Conservatives, with their mantra that free market competition guarantees quality. The trouble here is that, in education, standards should be consistent for everyone; how, then, should exam boards compete for schools’ business? Oh sure, there’s cheaper (or free) teacher support, or the advancement of new, innovative, technical systems. But ultimately, what schools want from an exam board is for a high proportion of their pupils to get good grades. With this being the sole criterion for choice, the logical progression is that boards can only compete by tacitly implying that schools will get better results with them – ie, that their courses are simpler and their exams easier. This is, of course, against the letter of the law; but it’s amazing the sophistry that can be employed to circumvent this.

In practice, if the mode of competition is to lower standards, the ultimate result is that well-worn phrase “a race to the bottom”. The ever-increasing numbers of students achieving high grades seem to bear this out; surely it should be impossible for these to increase every year without some lowering of standards?

Again, the reasons for this trend are more complicated, and have as much to do with politics as education standards. Under the current system, the marks required to get particular grades can vary, according to senior examiners’ assessment of an exam’s difficulty. These are set at a meeting presided over by a Chair, who is ultimately the person responsible for the qualification’s standards. The trouble is that, to meet government regulatory requirements, the Chair may be (and frequently is) obliged to overturn the examiners’ judgments to maintain the percentage of entrants at particular grades – this is seen as the only way to guarantee consistency of standards.

But a statistical standard really shouldn’t be how we judge the quality of education. A fundamental misunderstanding that it’s possible to quantify the unquantifiable is, in my view, at the root of much is what is wrong with the current system. That’s what gave us the overly simplistic school league tables under the Thatcher government, the continual tyranny of statistics under the Blair government, and now the addle-headed English Baccalaureate under the current one. None of these take into account the complexities of the issue; in their attempt to boil down so many things into a set of judgmental statistics, it’s along the lines of the oft-heard complaint against the Conservatives – that they know the price of everything and the value of nothing.

How to change it, then? To be fair to the Coalition, they have got all sorts of committees investigating that very question, although politicians rarely listen to reports that contradict their pre-existing prejudices (just ask the Advisory Council on the Misuse of Drugs). One common suggestion is to scrap the element of competition and simply have one, publicly owned, exam board. This is the case in many countries (Australia for one) with higher recognised educational standards than this one. On the face of it, that has a lot of merit – one standard, applied consistently to every school, with no ‘dumbing down’ and as much independence from ill-qualified politicians as possible.

In practice, unfortunately, it seems impossible. We have a long entrenched system of competing boards, originally sponsored by the Universities (OCR still is). The practicalities of buying their interests (though OCR and AQA at least are classed as charities) would be highly complex and expensive. And at a time when the government are busy introducing the most nakedly ideological Conservative policies since Mrs Thatcher, it hardly seems likely that a Conservative-led government would suddenly spend masses of public money to, effectively, nationalise a whole sector of employment. Not to mention the fact that they’re keen on introducing even more privatisation to education, with Free Schools and the ability of corporations such as McDonalds’ and JCB to award qualifications – initiatives started under the more-Tory-than-Tory New Labour.

Oh well, if we can’t do that, why not try removing politics from education? Altogether? Reduce the demand to boil results down to meaningless statistics for populist reasons; if no votes depended on the percentage of students getting an A, I suspect we’d soon see a fall in those levels to reflect real standards. But while political popularity is tied to such statistics, it would be a ‘courageous’ politician who would stand up and say “what we need is fewer people doing well in exams”.

A properly independent publicly funded education system, regulated independently with the minimum of government oversight and organised by actual educational professionals would be a real step forward. You’d think that would find favour with the laissez-faire, keep-the state-out-of-things Conservatives. But again, sadly where public money is concerned, the politicians are convinced that only they can make the best-informed decisions – a timeworn fallacy that’s applied to governments of every party over the years.

I appreciate that this has been a long ramble on the subject, but to return to the original point – two bad apples does not a rotten barrel make. The system we have is far from perfect, and I’m always glad when the debate about fundamental changes is spurred. But this knee jerk media-generated scandal is entirely the wrong way to go about that debate, and it’s appalling (but not surprising) that policy makers should so easily leap onto the mob bandwagon. There is scope for real change in this area, but I suspect we won’t see anything I’d consider improvement as a result of the Telegraph’s scandal mongering.