Tuesday, 20 May 2014

Response to Ofqual Consultation on Reforming GCSEs in MFL and ALs

Ofqual consultation - MFL GCSE assessment: Urgent need for response from as many people as possible - especially Question 10!

The Ofqual consultation regarding assessment of GCSE languages for the GCSE subject content to be first taught 2016 and first tested 2018 is now published and responses are needed by

 Friday 23rd May 2014.
 

I think it is a well constructed document stating clear rationale for its proposals.  After each explanatory section, there are clear statements with option of responding 'strongly agree, agree, disagree, strongly disagree and don't know/no opinion' or 'Yes/No'.  It is optional to add text. 

Go to page 40 to see the response section Pages 40-52 containing the 26 questions which you can fill in and send off if you prefer not to do it online.

I urge you to register your response so that you can support good decisions or challenge poor decisions. It may seem daunting, but it would be sufficient just to indicate the extent to which you agree to 26 statements. (So possibly just 15 minutes).  In my experience, organisations DO take notice of us when enough of us make the point.

I have written a response on behalf of our school and there is a copy to view / download here (stored on my google drive).  I have also copied and pasted it below. 

Question 10 - An end to mixed tier entry?

To me, the most important  element to challenge is the proposal that students should be required to enter for EITHER higher OR foundation-tier assessments, but not a combination of the two. So it would no longer be possible to enter, say, speaking and writing foundation, and reading and listening higher level.   This would be detrimental for a significant number of 'mid-range' students, and especially punishing for those with dyslexia.  Please communicate this to Ofqual by filling in the consultation, and 'strongly disagree' with this proposal which is in Question 10.

There is an opportunity to give reasons for your responses.   I have  taken the opportunity to reiterate points made to Ofqual in the past on Marking criteria in the context of MFL, [download word doc here] and I did a quick  analysis of previous GCSE and O level 'test types' to help me answer the section on validity, reliability and fairness.  You can view/download it here:  analysis of the relative validity, reliability and fairness of current and former GCSE test types

Background

The DfE consulted on GCSE subject content and assessment objectives for Modern Languages in June 2013.   I gave a full response in my blogpost here.

The DfE published the GCSE subject content and assessment objectives for Modern Languages in April 2014 with the following explanation on the front cover: 'Final content for this subject will be puiblished when Ofqual has consulted on and made decisions about assessment arrangements for GCSE languages'.  I have created a 9-page word document which compares directly the June 2013 with the April 2014 criteria [Download here]


RESPONSE COPIED AND PASTED BELOW: (26 questions highlighted to aid skimming!)


1.         Reading skills should be assessed using exams set and marked by the exam boards. To what extent do you agree or disagree this statement?
(x ) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer:
Setting valid and reliable reading exams is a highly skilled job. This is the most straightforward and fair way of ensuring fair comparison across the cohort. 
In order to make the test a valid test of reading, the questions need to be in English.  Testing using the target language makes the test of a discrete skill invalid (e.g. a candidate may understand what is written in a text, but because they may not understand or misunderstand the rubric, they may fail to demonstrate this OR a candidate may make a guess at the answer and write it in the target language without understanding its meaning.    It is vital to have questions in English (as for the 'harder language' exams)
Having said this, we assume that the DfE criteria cannot be changed now, so would urge consideration of the following principles when applying the requirement to use a percentage of target language testing:
(1)   Give great thought as to which type of questions can maintain validity the closest e.g. ensure that questions set are of a lower language complexity than the language being tested so that it is totally reasonable to expect that candidates can understand the question and know what they have to show they can do.
(2)   Given that the decision has been made not to have mixed skills testing, you need to be constantly asking 'is this question testing skills other than the one we are meant to be testing?' Do not penalise for the quality of written response.  Test only for evidence that the text has been understood.
2 Writing skills should be assessed using exams set and marked by the exam boards. To what extent do you agree or disagree this statement? 
( x) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer…
Taking the task in examination conditions with a high level of control (invigilators, silence) ensures fairness.  Making comparative judgements about quality of writing is a highly skilled job.
Although we do not recommend target language testing, we are pleased that the phrasing used with respect to the written element seems to comply with the principle that pupils need to be able to understand very clearly what they are expected to do.
3 Listening skills should be assessed using exams set and marked by the exam boards. To what extent do you agree or disagree with this statement?
( x) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer…
Setting valid and reliable reading exams is a highly skilled job. This is the most straightforward and fair way of ensuring fair comparison across the cohort. 
In order to make the test a valid test of listening, the questions need to be in English.  Testing using the target language makes the test of a discrete skill invalid (e.g. a candidate may understand what is written in a text, but because they may not understand or misunderstand the rubric, they may fail to demonstrate this OR a candidate may make a guess at the answer and write it in the target language without understanding its meaning.    It is vital to have questions in English (as for the 'harder language' exams)
Having said this, we assume that the DfE criteria cannot be changed now, so would urge consideration of the following principles when applying the requirement to use a percentage of target language testing:
(1)   Give great thought as to which type of questions can maintain validity the closest e.g. ensure that questions set are of a lower language complexity than the language being tested so that it is totally reasonable to expect that candidates can understand the question and know what they have to show they can do.
(2)   Given that the decision has been made not to have mixed skills testing, you need to be constantly asking 'is this question testing skills other than the one we are meant to be testing?' Do not penalise for the quality of written response.  Test only for evidence that the text has been understood.
4 Speaking skills should be assessed by non-exam assessments, using tasks set and marked by the exam board. To what extent do you agree or disagree this statement? 
(x ) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer…
External examining would ensure the highest level of control, but I appreciate that this would be extremely costly.  
5 What considerations need to be taken into account to make sure students’ speaking skills are assessed in a way that is:
Please see appendix for detailed analysis of the extent to which current and past assessment types for all skills are valid, reliable or fair.  We summarise principles below.
a)        valid
·    Ensure that assessments test what they are meant to be testing and allow all pupils of all abilities to demonstrate what they know, understand and can do, across the full range of speaking skills (communication, independent interaction, pronunciation & intonation, fluency/confidence, accuracy), across the full range of authentic situations (e.g. ones which are highly predictable to ones which are not predictable) and across the full range of authentic 'support' which can be given (e.g. glossaries of phrases/words, wordlists/dictionaries, prompting from a sympathetic native speaker through to  surviving with minimal support)   
·    Design range of test types to suit the aspect of the skill being tested e.g. supplying a common text to be read aloud, thereby testing pronunciation only.
·    Recognise that the ability to understand a question is not a test of speaking ability, so ensure that tasks and assessment criteria do not penalise candidates unduly for this (e.g. allow re-wording, prompting, quickly moving on to another question where candidate can demonstrate abilities, do not penalise hesitation when a candidate has not instantly understood a question).
·    Allow a reasonable amount of time / number of opportunities for candidates to show what they know understand and can do.  [Try to avoid a high-stakes '7 minutes' at the end of the course if at all possible]
………………………………………………………...………………………………….
b)        reliable,
Common shared criteria for all exam boards. 
(See appendix on marking criteria - response to Ofqual questions).
·         Establish very clear common subject criteria with interpretation shared by exam boards
·         Ensure there is a clear and explicit reference to a benchmark (e.g. the GCSE core subject specification), that they reflect the subject criteria and content fairly (do not include descriptors which describe AS/A2 performance)
·         Ensure that criteria are clear, transparent and as objective as possible to avoid a subjective response (include full descriptions of all strands for each 'band' - do not have gaps; be very clear about how apparently 'subjective' qualities are tested (e.g. 'spontaneity', ''creativity' , 'independence') and if they cannot be defined, do not include them.
·         Indicative content (samples) to demonstrate meaning of quantitative and qualitative words (e.g. long/short, basic/ higher, complex / simple / original; if samples cannot be given, omit the words.
·         Ensure consistency and lack of contradiction in descriptions of bands of performance
·         Consider how to deal with language which may not conform to the definitions of a scale of  levels .. e.g. someone may have grasped something which may be seen as 'complex' (e.g. perfect tense) but may make mistakes in something defined as 'simple language' (e.g. basic present tense) Recommendation: Take a positive approach.   Reward what is demonstrated, allow the benefit of the doubt (e.g. do not penalise for incorrect ambitious language or incorrect understanding of a question (listening comprehension).
·         Identify what range of language, grammar and structures constitutes language worthy of 'higher levels' (avoid obsession with tenses / opinions and justifications which can lead to an extremely 'formulaic' approach in order to 'tick the boxes') [Note that the 2008 criteria specifically did not 'require'  three tenses for every piece of written work in order to qualify for higher grade.  Higher level language can be represented in more sophisticated ways than just using three tenses, as evidenced in authentic literature]
·         Require a field test to establish how fairly the mark scheme can be applied and a moderation process which ensures that if 2 people mark the same piece of work independently they reach exactly the same / agreed tolerance of x points
·         Be aware of the risk of a marking scheme which prevents candidates form getting top marks e.g where 4 sub-criteria of a skill are each marked out of 5, and where examiners are reluctant to award the top band, there will be a 'bunching' of marks around 16/20  [This has been identified as one of the possible reasons for the relative lack of A* at A level]
·         Ensure a  common approach across examining boards to any 'limiting' factors (e.g. to what extent is accuracy penalised if communication is weaker ) [This has been an issue at A level, where one board will not award marks from the top band for accuracy if the top band for content is not reached, whereas another board does allow this]
·         Ensure a common approach to how to deal with pupils who are over-ambitious, e.g.  how can you give equal credit for what they know understand and can do and avoid penalising them  for attempts to be more 'original' which may lead to a higher proportion of inaccuracy compared with the work fo a candidate who has adopted a more 'standard' approach
·         Ensure that tasks match the criteria
………………………………………………………...………………………………….
c)         fair?
Ensure equal conditions for all candidates (same rules, whatever the board, with respect to predictability and support; system for checking schools' compliance with rules when teacher is examiner (e.g. opening of test papers, degree of support)
Ensure equal standard of marking, whatever the board (e.g. through having common criteria and cross-board moderation checking, including teachers in standardisation meetings) - see above
………………………………………………………...…………………………………..
6 How might any aspects of the proposed assessment requirements impact on:
a)        the costs, and
Conditions requiring higher levels of control / supervision / invigilation and a recording of all tasks increase staffing costs [cost of cover when teachers need to administer tests simultaneously to protect confidentiality, less flexible time slots mean that centres cannot take advantage of specific periods when teachers have lower contact time].  Currently no invigilators needed for test and only one task needs to be recorded formally so the cost is covering the teacher lesson.
External examiners would be more costly to the exam board, however, this method has been used in the past.   Teachers are currently 'unpaid' examiners for the exam boards.
……………………………………………………...……………………………
b)        likely take-up of new modern foreign language GCSEs?
If all tasks are unpredictable and have no support, they will be extremely difficult for lower attainers who will not be able to demonstrate the lower order speaking skills they have e.g. ability to communicate with use of a guide book / glossary, ability to pronounce, ability to 'recall' and memorise (note that memorisation is a valid skill)
There is a danger that we revert to the situation pre-current GCSE when teachers were complaining bitterly about candidates having to memorise 16 topics.  This can be very off-putting for all abilities
If the ML exam ends up being more expensive than other exams, this could be a disincentive for managers to encourage students to take languages
7 The outcome of the speaking component should contribute to a student’s overall grade. To what extent do you agree or disagree this statement? 
( x) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
1.1      Please give reasons for your answer
We strongly agree with point 2.14 "For modern foreign language GCSEs the ability to speak the language is a key aspect of the qualification. Modern foreign language GCSEs are designed to develop and recognise students’ skills in a second or additional language. This is in contrast to English language GCSE which is designed to be taken by English speakers. "
8 All assessments (reading, writing, listening and speaking) in new modern foreign language GCSEs should be tiered. To what extent do you agree or disagree this statement?
(x) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer
Firstly, we must challenge your assertion in 3.7.  Much thought has been given to the rationale for tiering. (Helen Myers was heavily involved in the last GCSE revision as one of the two teacher representatives invited by QCA to meetings with exam boards,  and tiering certainly was not adopted because of  'historical practice'.  It was based on considering what was the fairest way of assessing candidates of all abilities.
A single exam covering all grades lasting, say, 40 minutes would not give adequate opportunity for students to demonstrate what they know understand and can do and would be particularly damaging for the least able for whom under half would be applicable.  (However, the more able do not suffer from 15 minutes of relatively easy questions)
The most valid and reliable method for assessment would be to revert to the method in 1988.  All take short foundation tier in each skill.  It is then optional to take a further higher tier paper in any or all of the skills.  From the candidate point of view there is no 'gamble' as to which level to take, and  no feeling of being 'pigeon-holed' at an early stage of the course.  In our experience, higher level students are not demotivated by sitting tests which they find easy, and are intelligent enough to understand the reasons for the system when this is explained.     If lower level students attempted the higher and found it difficult, they were not demotivated as they had already taken the foundation tier and succeeded in it.  From the exam board point of view, there is no problem about comparability between foundation and higher levels to secure the same grade, and obviously no need for an overlap section).  Thus, the approach we strongly recommend is to have a '2-part' test for each skill.  This would take up more time and involve additional marking, but this is achievable (-it worked in 1988-) and electronic means of marking could speed up the process of marking.  The academic benefits far outweigh the organisational disadvantages.
Failing this, we would strongly support tiering, with an overlap between higher and foundation tiers.  It would be essential for listening and reading in order to give candidates the maximum opportunity to demonstrate what they know understand and can do. (e.g. if equivalent grades G - C were tested in just half the paper, say 20 minutes, this would not be a valid enough 'sample').
It is less essential in speaking and writing where feasibly a more open-ended task could be set with guidance on 'points which could be included'.  However, we would welcome the opportunity to have more focussed tasks which are more accessible to the lower attainers and do not rely on an 'open-ended' response and memorisation.  Below we summarise the ways in which different tiers could make tasks more relevant for the ability profile by offering a range of predictability and support.
Foundation speaking: read aloud a pre-learnt passage; carry out a predictable role play with support of a glossary; carry out a predictable conversation from a defined range of topics with support of a glossary
Higher speaking: carry out a less predictable role play with limited glossary; carry out a less predictable conversation/simulation from a wider range of topics with limited / no glossary
Foundation writing: Write a list of notes (with support of glossary); reply to a formulaic straightforward letter (with glossary) [All structures and topics required from foundation list only]
Higher writing: Write answer to a more challenging letter (with limited glossary) frpm a wider range of topics; write a document demonstrating higher level functions and structures (to interest, persuade, inform, convince) and expressing directed content (so in the form of 'translation')
Foundation reading: Tasks based on texts which reflect foundation topics and structures; translation into English with support of glossary
Higher reading: Tasks based on texts which reflect higher topics and structures and require more skills of inference; translation into English with support of limited glossary
Foundation listening: Tasks based on passages which reflect foundation topics and structures (defined); Opportunity to listen to texts more frequently / passages broken up more
Higher listening: Tasks based on passages which reflect higher topics and structures (defined); Longer passages; Requirement to infer meaning
9 All available new modern foreign language GCSEs should be tiered. To what extent do you agree or disagree with this statement?
( x) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer
We agree with the rationale you give in 3.19 We are providing a General Certificate of Secondary Education, rewarding what people achieve through being at secondary school.   We should be encouraging non-native speakers to be learning community languages.  (e.g. especially those in the caring / service industries e.g. nursing)
10 Students should be required to enter for either higher- or foundation-tier assessments but not a combination of the two. To what extent do you agree or disagree this statement?
( ) Strongly agree
( ) Agree
( ) Disagree
( x) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer
A significant number of students have SEN with respect to literacy and can perform significantly better in speaking and listening tasks.  They would be heavily disadvantaged by such a system. 
Para 3.29 is not necessarily logical.  Candidates can be very 'thrown' by a paper where they do not understand very much, and can 'underperform' as they 'give up'.  In mock exams we frequently find pupils who gain a grade D in the higher and a grade C in the foundation.
Para 3.31 seems controversial.  What evidence do you have to suggest that Category 2 students are deterred from improving their reading and writing skills because of the option to take a foundation paper?
If the proposal to remove the option of mixed tier testing were taken forward, we believe it would have a very serious impact on the popularity of and take-up of languages.  It would not pose a problem for the more able of-course, but would pose a further deterrent for the less able, who would be forced to take foundation papers in all skills if they had relative weaknesses in any one of the skills. 
11 For the listening, reading and writing assessments 20 per cent of marks, and for speaking 50 per cent of marks, should be allocated to questions or tasks that are common in any series to both the foundation and higher tier assessments. To what extent do you agree or disagree with this statement?
( ) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( x) Don’t know/no opinion
Currently the overlap for listening and reading allows for G - C and E - A* - so the equivalent of three out of the available 8 grades.  This seems to work adequately.  On a 1-9 point scale therefore, we would expect the overlap to be 2-3 points.
12 Do you have any further comments on the tiering of modern foreign language GCSEs? …………………………………………………………………………………………………………………………………………………………………………………………
13 New GCSEs in ancient languages should be assessed wholly by examination. To what extent do you agree or disagree with this statement?
(x ) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer…as for modern languages (see above)
14 New ancient language GCSEs should not be tiered. To what extent do you agree or disagree with this statement?
( ) Strongly agree
() Agree
() Disagree
( ) Strongly disagree
( x) Don’t know/no opinion
Please give reasons for your answer
Your points in paras 5.2, 5.3 and 5.4 explain the situation well, and reflect the position in our school where only the most able and committed take Latin.  However, if a tiered approach were to be considered on balance more appropriate, we would not complain at all!
Availability of modern foreign language GCSEs
15 Modern foreign language GCSEs in a wide range of languages should be available in the future. To what extent do you agree or disagree with this statement?
( x) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer…
GCSE is the recognised 'currency ' for employment and opportunity and this should be as widely available as possible
16 Modern foreign language GCSEs for which there is low demand should be available in the future. To what extent do you agree or disagree with this statement?
( x) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer…
GCSE is the recognised 'currency ' for employment and opportunity and this should be as widely available as possible
…………………………………
17 Modern foreign language GCSEs should be available for students who are existing users of the language. To what extent do you agree or disagree with this statement?
() Strongly agree
( x) Agree
( ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer…
This can reward students for their abilities, and encourage them to maintain and even improve on home language (e.g. requiring all four skills)
18 In your opinion would schools and colleges be willing to pay a higher fee to enter students for modern foreign language GCSEs for which there is a lower demand?
Yes
19 What, if any, steps do you think Ofqual should take to secure the availability of GCSEs in a range of modern foreign languages?
It seems reasonable to come to an arrangement with the Awarding Organisations such that they 'share out' the provision for less widely taught languages
20 A range of ancient language GCSEs should be available in the future. To what extent do you agree or disagree with this statement?
( ) Strongly agree
( ) Agree
( ) Disagree
( ) Strongly disagree
( x) Don’t know/no opinion
Please give reasons for your answer…
This does not affect our school
21 What, if any, steps should Ofqual take to secure the availability of GCSEs in a range of ancient languages?
…………………………………………………………………………………………….
22 A disabled student should obtain an exemption for no more than 40 per cent of the available marks for a modern foreign language GCSE. To what extent do you agree or disagree with this statement?
( ) Strongly agree
( ) Agree
(x ) Disagree
( ) Strongly disagree
( ) Don’t know/no opinion
Please give reasons for your answer…
We are not sure why 40 % has been selected.  Could there be an allowance for recognition of achievement in any of the four skills attempted according to the protected characteristic affected? It seems arbitrary that it is possible to accredit short course listening and speaking OR short course reading and writing but not to allow other combinations.
23 We have identified a number of ways the proposed requirements for new GCSEs in modern foreign languages may impact (positively and negatively) on persons who share a protected characteristic. Are there any other potential impacts we have not identified?
Yes
If yes, what are they? …
 
(1)   Para 7.15 rightly identifies the benefits of tiering for those who wish to learn languages for religious purposes, but if prevented from mixing those tiers, they could be prevented from demonstrating the full range of their skills
(2)   Dyslexic students would suffer greatly from a requirement to take all foundation or all higher papers.  Typically they do relatively much better in listening and speaking than in reading and writing.  A common pattern for them is to enter higher listening (for example being able to gain a high 'C' and foundation reading
24 We have not identified any ways by which the proposed requirements for new GCSEs in ancient languages may impact (positively or negatively) on persons who share a protected characteristic over and above those impacts that apply to the changes to GCSEs generally. Are there any potential impacts we have not identified?
No
If yes, what are they? …………………………………………………………………..
25 Are there any additional steps we could take to mitigate any negative impact resulting from these proposals on persons who share a protected characteristic?
No
If yes, what are they? …………………………………………………………………..
26 Do any of the proposals or options being considered have financial or wider resource consequences, positive or negative, for:
Schools                        Yes
Exam boards               Yes
Others                           Yes/no
Please explain your response…
Provision of someone to read aloud texts for visually impaired would add additional cost to staffing and rooming. 
 




No comments:

Post a comment