El Guiniguada, 30 (2021), pp. 193-206
Print ISSN: 0213-0610 – eISSN: 2386-3374
Servicio de Publicaciones y Difusión Científica Universidad de Las Palmas de Gran Canaria
Servicio de Publicaciones y Difusión Científica (SPDC), Universidad de Las Palmas de Gran Canaria, Parque Científico-Tecnológico, Edificio Polivalente II, C/ Practicante Ignacio Rodríguez, s/n
Campus Universitario de Tafira, 35017 Las Palmas de Gran Canaria, Spain
eISSN: 2386-3374
10.20420/ElGuiniguada.2013.333 (doi general de la revista)
Journal information, indexing and abstracting details, archives, and instructions for submissions: http://ojsspdc.ulpgc.es/ojs/index.php/ElGuiniguada/index
Implementación del MCER en las rúbricas de dos principales certificados de inglés: Cambridge FCE y Trinity ISE-II
Universidade da Coruña
DOI (en Metadatos y en Sumario Revista)
Recibido el 01/07/2020 Aceptado el 13/04/2021
El Guiniguada is licensed under a Creative Commons ReconocimientoNoComercial-SinObraDerivada
4.0 Internacional License.
RESUMEN
El Marco Común Europeo de Referencia (MCER) se desarrolló con el objetivo de promover una base común para la creación de currículos educativos y servir como guía en la elaboración de materiales didácticos, exámenes de certificación e instrumentos de evaluación. El MCER se ha implantado en España a través de diferentes leyes educativas y ha propiciado la introducción del enfoque comunicativo o nuevos instrumentos de evaluación como rúbricas. Sin embargo, casi veinte años después de la aprobación del marco, pocos son los estudios que han revisado de qué manera se ha adaptado el marco. Es aquí donde la presente investigación se sitúa, con el objetivo de comprobar cómo se ha implementado el MCER en las rúbricas para examinar la destreza escrita de dos de los principales certificados de inglés: el Cambridge Assessment English FCE y el Trinity College ISE-II.
PALABRAS CLAVE
INGLÉS COMO LENGUA EXTRANJERA, MCER, EVALUACIÓN, RÚBRICA, FCE, ISE, CERTIFICADOS OFICIALES DE INGLÉS
ABSTRACT
The Common European Framework of Reference (CEFR) was developed by the European Council with the intention of providing a comprehensive basis for the creation language syllabi and curriculum guidelines, together with the design of teaching materials, language certificates and instruments of assessment. The CEFR has been implemented in Spain through different education laws and has prompted the introduction of the communicative approach and the use of new instruments of assessment such as rubrics. Nonetheless, almost twenty years after the CEFR was passed, not many researches have been conducted on how the Framework has been implemented. It is from this line where the current research stems as it intends to check how the CEFR has been adapted in the rubrics used for the assessment of the writing skill in two main English Certificates: the Cambridge Assessment English FCE and the Trinity College ISE-II.
KEYWORDS
EFL, CEFR, ASSESSMENT, RUBRIC, FCE, ISE, ENGLISH OFFICIAL CERTIFICATES
INTRODUCTION
The Common European Framework of Reference (henceforth, CEFR) was developed by the European Council with the intention of providing a comprehensive and transparent basis for the creation and design of language syllabi and curriculum guidelines, together with the design of teaching and learning textbooks, worksheets, tasks and, in short, teaching materials, and the assessment of the foreign language competence.
As cited in the Framework, assessment is used in the document “in the sense of the assessment of the proficiency of the language user (Council of Europe, 2002, p. 177). As a result, the framework is a valuable source that can be checked by teachers in order to construct the specifications of a task. Furthermore, “the scales provide a source for the development of rating scales for the assessment of the attainment of a particular learning objective and the descriptors may assist in the formulation of criteria” (2002, p. 179). Thus, descriptors can be used as basis for the creation of rubrics, they can be consulted to stablish assessment criteria and, at the same time, they can be useful to give the students feedback since they allow them to obtain a clear assessment of their performance, with positive and negative aspects.
The Framework is also a practical instrument to “relate national and institutional Frameworks to each other” and “map the objectives of particular examinations and course modules using the categories and levels of the scales” (2002, p. 182). This means that the Learning Standards used in a determined level in one institution are very similar to the ones used in another institution in a complete different geographical point. It also implies that a student with a certified B2 level should have the same competence in a language as any other citizen in the European Union with a B2 certificate, independently of the institution that has granted it.
The use of the CEFR for the teaching and assessment of English as a Foreign Language (henceforth, EFL) has been implemented in Spain through different education laws, the last one called Ley Orgánica para la Mejora de la Calidad Educativa, known as LOMCE (2013). This adaptation has triggered several changes in the teaching-learning process, among them, the implementation of the communicative approach or the use of new instruments of assessment such as rubrics or portfolios.
In consequence, a great deal of research has emerged in connection with the teaching and assessment of EFL in recent years. However, almost twenty years after the CEFR was passed, there have been few studies conducted to review the adjustment of materials, methodologies, evaluation system and instruments of assessment aiming at check how they have adapted to it.
It is precisely on this line where the current comparative analysis can be placed as it intends to examine two rubrics used as instruments of evaluation in two official English certificates. The objective is no other than checking whether those instruments do actually follow the guidelines established by the Framework.
WRITING AND RUBRICS
Writing has been one of the main skills to be taught and assessed in modern languages. Despite its importance, writing is a skill that is generally not mastered neither by native nor non-native speakers. Indeed, Allen et al. (2014) indicate that “students in the US struggle to reach proficiency levels throughout their high school years” (p. 125). This is due to the complexity that the process of writing implies.
Even though L1 and L2 writing processes are alike, they show some differences in lower-level processes. L1 writers have automatized processes like syntactical constructions and lexical access that L2 learners often regard as high-demanding. (De Keyser in Allen et al., 2014, p. 126). Moreover, L2 students tackle some constraints when writing, for example, the mental translation they try to do.
Concerning the assessment of the writing skill of L2 learners, there are several criteria that normally influence the score such as the range of vocabulary, the organization, cohesion and coherence, grammatical resource and the communicative achievement. Some other aspects such as spelling, originality or punctuation may be considered as well. In the past, writings were scored based on the teacher’s criteria, who had to come up with a mark without specific guidelines to support his or her decision (Brooks, 2012, pp. 229-230). As a result, writing score was not only imprecise but also it was heavily influenced by the mistakes made by the learner. Nowadays, it is evident that in order to consider a composition to be outstanding, it is not enough that it does not contain grammatical or spelling mistakes. It must also show a great grammatical and lexical resource, a good punctuation, development of ideas and cohesive devices and discourse markers. Therefore, finding a tool that allows us to measure all those aspects was needed.
Taking into account the different aspects to measure when writing is being assessed, rubrics seemed to be an appropriate tool as they “enable an evaluator to convert a given quality of student work into a letter grade, percentage or level” (Mark Brook in Frydrychova, 2011, p. 391). According to Gavin Brooks, rubrics were first used as a tool to analyse writing in 1912 (2012, p. 229).
Different researches have been conducted in relation with the use of rubrics to the assessment of writing, specially focused on the effects that providing students with a rubric and explaining them how it works prior to the exam have. Researches like the one carried out by Todd Sundeen (2014), or the one conducted by Anthony Becker (2016) worked with different groups in which treatment groups were showed and explained the rubric and control groups were not. Results proved the benefits that having access to the rubric in advance had on the students’ results. Laurian & Fitzgerald’s study (2013) also revealed that students obtained higher scores once they were shown a rubric.
Although rubrics are used all over the world, in Spain they are still starting to be used as an assessment tool and occasionally as a teaching tool. A research conducted by Velasco-Martinez & Tojar in 2015 on the use of rubrics in Spanish universities indicated that only 4% of the rubrics were used in the branch of arts and humanities and that it was mainly used to assess essay writing (36%). Since the introduction of rubrics in our education systems is still recent and scarce but finally on the rise, it is high time rubrics were further discussed as there are still many enquiries about their application, effectivity, uses and objectivity which need to be addressed.
METHODOLOGY
The Council of Europe is responsible for the implementation of a set of actions oriented towards language research from a qualitative approach. In the year 1971 in the symposium “The teaching of Modern Languages in Adults’ Education” established, among other aims, that the language teaching should specify realistic and appropriate objectives based on an assessment of the necessities, characteristics and skills of students. It was also stated that teaching must be planned as a coherent whole, by covering all the objective’s specifications, the assessment of learner achievement and the effectiveness of the system, and that everything implies a coordinated effort by education administrations, materials, examiners, inspectors, teachers and students, etc as they all should share aims and assessment criteria. (Vez, 2011, p.89)
The approach employed for this investigation is therefore qualitative. Daniel Madrid (2001) refers to this kind of research which does not use numerical data extracted from reality but instead tries to interpret and describe the reality in detail through words (p. 12).
The main technique has been the analysis of documents, particularly, the two rubrics that had been selected and that are used for the assessment of the writing papers in the two chosen institutions: Trinity College and Cambridge Assessment English. For Berelson, the analysis of content is a research technique for the objective, systemic and qualitative description of the content of communication” (in Gil, 2011, p.182).
Following Hernández Sampieri et al. (2010, p. 418), eight steps were determined for this comparative analysis. The first step was the exploration of data; in these steps the two rubrics from two well-known institutions of English certificates in Spain were chosen: those of certificates Trinity College ISE-II and Cambridge Assessment English FCE. The rubrics selected were intended to assess writing competence for level B2 of CERF that corresponds to the high-intermediate and it is the most frequently required by companies and institutions for either work or study.
Figure 1.
Writing rubric for Cambridge FCE
Figure 2.
First part of rubric for the first task of writing Trinity College ISE-II
Figure 3.
Second part of rubric for the first task of writing Trinity College ISE-II
Figure 4.
Rubric for the second writing task Trinity College ISE-II
Moreover, in this step the framework was also studied carefully in order to determine which indications were given for the creation of rubrics. It is obvious that a good rubric must be valid, reliable, effective and relevant. Nevertheless, it is very frequent to find rubrics in textbooks that do not follow the aforementioned criteria. Rubric’s validity is determined by how well it assesses what it has to assess (Phelan and Wren, 2005, para. 2). Regarding reliability, it is “the degree to which an assessment tool produces stable and consistent results” (para 1.) Relevance comes by the fact of whether the instrument is appropriate for the assessment of a concrete task and also by the descriptors and criteria it consists of. Finally, a rubric is considered to be effective when it follows all the above mentioned as well as its ability to provide the students with a suitable feedback, that is, which is the level of performance and which aspects should be improved.
Together with all these aspects that conform a good rubric, the CEFR guidelines to create grading scales must be bear in mind too. The Framework highlights that it is essential to build a feasible tool, so a rubric should not contain more than five criteria. As far as the descriptors are concerned, it is detailed that they must be positively worded, and they must be brief and avoid vagueness.
Second step was the structuring of data in recording tables. With this purpose, some tables to collect the main data from rubrics were designed. That way, rubrics were classified in terms of typology according to how the performance is measured, according to their scale, according to their theme, according to their level of application, according to their function, according to the scorer and according to the channel. Moreover, their relevance and validity were studied and whether or not they followed the main guidelines established in the Framework.
Figure 5.
Recording table
Next, the data dump from the recording tables was carried out. Once the tables were completed, the fourth step was to set patterns or concepts from the data obtained. In
order to do so, recording tables were contrasted so that those aspects shared could be found and the divergences from the framework could be pointed out.
In the fifth step data were set in a real context. In this case, it was clear that we were in a European country, Spain, and hence the guidelines from the CERF must be followed. Furthermore, the English Certificates selected have several calls a year and allow that many students are assessed at the same time. This prompts the necessity for a rubric not only effective but feasible, that means, one that can be handy to assess quickly many students.
The sixth step consisted in the rebuilt of data in order that they can be studied as a whole, that is, we started assessing the rubric in small parts such as its descriptors, its number of criteria, etc. and now we had to assess them in general and answer the question: is this rubric effective?
Once the rubrics were valued as a whole, it was time to look for connections with other results of learning and, finally, to formulate theories based on the findings as it will be explained in the conclusions section.
FINDINGS
Firstly, it is worth to highlight that the rubrics analysed are available for everybody on the webpages of both English certificates, unlike in some other institutions such as the ACLES or some Spain’ Schools of Languages, where the rubrics are not always accessible to candidates. Several studies have proved the importance and benefits that providing students with a rubric in advance have. Students are allowed to know what it is expected from their performance, which criteria will be assessed and in which way and their performance improves considerably when they do (Laurian and FitzGerald, 2013). The webpages of these institutions do not always show data about the validity or reliability of their exams, with the publication of index like the Cronbach Alpha or SEM, which would be very recommendable, and it would ease greatly the comparative analysis of the certificates. In this sense, the Cambridge Assessment English is one of the few that shows complete transparency and publish not only the figure of the above-mentioned indices but also reports on the studies related to them.
After the comparative analysis of the rubrics used to assess the writing skill in the Cambridge Assessment English FCE and the Trinity College ISE-II papers, some similarities were found such as the fact that both rubrics are analytic and not holistic. This is presumably due to the fact that analytic rubrics allow us to measure different aspects of the text that has been produced by the candidate, for instance, organization, lexical resource, grammatical resource, coherence and cohesion, etc. Their results obtained are therefore much more accurate and precise than the ones obtained with a holistic rubric, as research has shown (Sundeen 2014; Becker 2016).
All the rubrics that have been analysed use a numerical scale. Even though it is true that, grosso modo, all the rubrics used to assess papers in English Certificates often use numerical scales since they are much more precise; if a qualitative scale were used in conjunction with the numerical score, the students could receive a feedback besides the mark. Nevertheless, a qualitative score to be truly meaningful, should be followed by an examiner’s comments. This is so because a pass, fail or good do not add much more information than a number but if the scorer wrote a small comment together with it, it would be really helpful for a learner.
According to its application, the Cambridge Assessment English FCE’s rubrics is a skill-focused one. This implies that the rubric works for the assessment of one skill, the writing skill in this particular case, but for all the writing tasks. By contrast, the Trinity College ISE-II writing skill is assessed with two different rubrics, one per task, hence it is a task-specific rubric. This kind of rubrics are much more tailored to the task but when there are several tasks and several rubrics to use, the scorer’s job becomes less agile, more tedious and slower.
The rubrics analysed are, according to their function, proficiency rubrics as they aim at determining the learner’s level in order to decide whether or not to grant the level’s certificate. Finally, rubrics have different criteria in terms of number, though they are similar in content. While the second task writing rubric of the ISE-II consists of three criteria: task fulfilment, organisation and structure and language control, the FCE’s writing rubric has four: “content”, “communicative achievement”, “organisation” and “language”. However, the “task fulfilment” criterion of the ISE-II encompasses the aspects that the FCE’s criteria of “content” and “communicative achievements” measure. This is different in the case of the rubric used for the first writing task in the ISE-II which contains just two criteria.
The contrast of the rubrics with the guidelines set by the Council of Europe in the CEFR has brought up interesting findings. As it has already been stated, the Framework establishes the use of instruments that are really feasible. In order to be so, the grading scale must have between four and five criteria, it must be convenient and handy, which means it has to be one-page long with descriptors that are not terribly long since a huge number of students will be assessed with the same rubric in each Certificate’s call.
Cambridge Assessment English’s writing rubric for FCE is less than one-page long and measures four criteria. As a result, it is a feasible one. On the other hand, the rubric for the first writing task of the Trinity College ISE-II is presented in two pages, which is not very handy. The one used for the second writing task is one-page long though. According to the criteria, the first one contains only two and they are two generic: “reading comprehension for writing” and “task fulfilment”. It must be said that the task is not common for the assessment of writing because it consists in filling in a scheme and it measures both the writing and the reading skills at the same time. With regard to the second task, it is a composition, the most frequent task to assess the writing skill. In spite of its suitable disposition in one page, the rubric is too long, so it is not handy either.
The Framework also explains how the descriptors must be: they have to be positively worded, brief and precise. The findings of the current research point out that none of the rubrics that were studied are positively worded. In the Trinity College ISE-II rubrics, level one descriptors contain expressions such as “poor achievement of the communicative aim”, “errors frequently impede understanding”, “poor awareness of the writer-reader relationship”, “inadequate evidence of grammatical range”. On the same line, the Cambridge Assessment English FCE rubric also contains expressions like “irrelevances and misinterpretation of task may be present” in level one. However, there is a clear attempt for using positively worded descriptors, even in the lower levels, with expressions such as “uses the conventions of the communicative task in generally appropriate way to communicate straightforward ideas” (descriptor of communicative achievement level one”, “text is connected and coherent, using basic linking words and limited number of cohesive devices” (descriptor of
organisation level one), or “uses everyday vocabulary generally appropriately, while occasionally overusing certain lexis”, “uses simple grammatical forms with a good degree of control” (descriptor of language level 1). As far as the briefness of descriptors, rubrics used by the ISE-II clearly do not respect this principle as most of the descriptors are between ten and fourteen lines long while the descriptors contained in the FCE’s rubric are five to six lines long on average.
To conclude, none of the descriptors analysed in the Trinity College writing rubrics can be considered as imprecise or vague as they describe the level of performance with accuracy and the levels are perfectly escalated with the above and below ones. But the Cambridge writing rubric does contain some vagueness since level 0, 2 and 4 are not described individually but with a “performance shares features of Bands X and X”. This denotes certain inaccuracy that is not desirable in this kind of assessment instruments.
CONCLUSIONS
The research conducted can be concluded by making a general reflection upon what the findings obtained mean within the area of study. The main goal of this study was to determine whether the CEFR has been correctly implemented in the instruments of assessment from some renown English Certificates in Spain.
The findings of the comparative analysis and the contrast with the guidelines written in the Common European Framework show incongruencies and shortcomings in those rubrics used for the assessment of the writing skills in some English Certificates of high-intermediate level or B2.
To begin with, none of the rubrics that have been scrutinised follows completely the guidelines of the CEFR. The rubrics for the assessment of writing in the ISE-II are not handy, their descriptors are not positively worded, and although they are accurate and clear, they are not brief. On the flip side of the coin, the rubric for the assessment of writing in the First Certificate is handy, the descriptors are almost always positively written, and it is brief, but in contrast, it is too vague in some of the levels.
The fact that both certificates share a fail at not always using positively written descriptors in the lower levels of their grading scales as the Framework sets, may lead to the reflection on whether this is entirely possible. After all, being able to describe a negative performance with words that are positive and precise to be able to differentiate it from the above levels and that, at the same time, in a brief way, requires being a true master of the language.
Another inconvenience of the Framework is that all the scales that are included but one are holistic and they just indicate the different levels of a language in the four skills: speaking, writing, reading and listening. Consequently, the institutions themselves must create their own analytic rubrics from the CEFR. They must decide on the criteria, the scale and formulate the descriptors for each of the levels and criteria. Thus, rubrics tend to be very different and are not always valid, effective or reliable. Besides, having to create an analytic rubric for a level, let’s say B2, from a holistic rubric in which each descriptor refers to a different level of the scale (from C1 to A1) brings about very diverse rubrics despite being all based on the Framework. If the CEFR contained examples of analytic rubrics for all the skills, even if those could be tailored to the exam and tasks required, resulting rubrics would be much more alike in terms of criteria or the statement of their descriptors. They would be more reliable and so would be the English Certificates themselves. The inclusion of analytic
scales would also facilitate examples of how to create descriptors positively worded that are simultaneously brief but precise.
The Common European Framework of Reference for Languages is extremely useful for assessment. The Council of Europe has carried out an essential unification of language competence levels in a continent that promotes their citizen’s mobility for working, studying and travelling and in a society that each day is more and more globalised. Its application may not be, notwithstanding, as feasible as it should. This is completely normal and, therefore, the Framework should be reviewed and improved with the goal of easing to the extent possible its implementation in the current education reality. At the same time, it should be updated to give response to the new problems or realities that may come up. Like languages themselves, the Common Frameworks should be always in constant development.
REFERENCES
Allen, L. K. et al. (2014). L2 Writing Practice: Game enjoyment as a key to engagement. Language Learning and Technology, Vol. 18, No.2, Jun., pp.124- 150.
Becker, A. (2016) Student-generated scoring rubrics: Examining their formative value for improving ESL students’s writing performance. Assessing Writing, Jul.
Brooks, G. (2012). Assessment and Academic Writing: A look at the Use of Rubrics in the Second Language Writing Classroom. Kwansei Gakuin University Humanities Review, Vol. 17, pp. 227-240. Recuperado de: core.ac.uk/download/pdf/143638458.pdf
Cambridge English Language Assessment (2016). Cambridge English First Handbook for Teachers for exams from 2016. CambridgeEnglish.org, Recuperado de: www.cambridgeenglish.org/images/167791-cambridge-english-first-handbook.pdf
Council of Europe (2001). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge University Press.
Frydrychova, B. (2011). Evaluation Writing in English as a second language.
Procedia_Social and Behavioral Sciences, Dec., pp. 390-394.
Hernández Sampieri, R. et al. (2010). Metodología de la Investigación, 5ª Ed.
McGraw Hill, ISBN: 978-607-15-0291-9
Laurian, S. & Fitzgerald, C.J. (2013). Effects of using rubrics in a university academic level Romanian literature class. Procedia. Social and Behavioral Sciences, Elsevier, 76, pp. 431-440.
Madrid, D. (2001) Introducción a la investigación en el aula de la lengua extranjera. Metodología de investigación en el área de filología inglesa, editado por María Elena García Sánchez and María Sagrario Salaberri, Universidad de Almería, Secretariado de Publicaciones, 363, pp. 11-45. Recuperado de: www.ugr.es/~dmadrid/Publicaciones/Introduccion%20investigacion%20aula- Sagrario%20y%20Elena.pdf
Phelan, C. & Wren, J. (2005-2006). Exploring reliability in academic assessment, Iowa: UNI Office of Academic Assessment, University of Northern Iowa, Recuperado de: www.uni.edu/chfasoa/reliabilityandvalidity.htm
Sundeen, T. H. (2014). Intructional rubrics: Effects of presentation on writing quality. Assessing writing, Elselvier, 1st of Apr. 2014, pp. 74-87.
Trinity College London. (2017) Integrated Skills in English (ISE) Guide for Teachers
— ISE-II (B2). Trinity College web page, Online edition Jun. Recuperado de: https://www.trinitycollege.com/qualifications/english-language/ISE/ISE-II-B2-resources/ISE- II-B2-Guides
Velasco-Martínez, L. & Tójar, J.C. (2015) Evaluación por competencias en educación superior. Uso y diseño de rúbricas por los docentes universitarios. AIDIPE (Ed.), Investigar con y para la sociedad, Bubok, Vol. 2, pp. 1393-1405, Recuperado de: avanza.uca.es/aidipe2015/libro/volumen2.pdf
Vez, J. M. (2011) La Investigación en Didáctica de las Lenguas Extranjeras. Educatio Siglo XXI, Vol. 29, No.1, pp. 81-108. Recuperado de: digitum.um.es/xmlui/bitstream/10201/27149/1/La%20Investigación%20en%20Didáctica%20 de%20las%20Lenguas%20Extranjeras.pdf