{"id":526,"date":"2013-05-10T12:14:41","date_gmt":"2013-05-10T11:14:41","guid":{"rendered":"http:\/\/www.uefap.com\/blog\/?p=526"},"modified":"2013-05-10T12:14:41","modified_gmt":"2013-05-10T11:14:41","slug":"assessment-criteria","status":"publish","type":"post","link":"https:\/\/uefap.org\/blog\/assessment-criteria\/","title":{"rendered":"Assessment Criteria"},"content":{"rendered":"

Assessment Criteria<\/b><\/p>\n

I ran a workshop at a BALEAP conference several years ago about assessment criteria. I was particularly interested in the difficulty of marking assignments, especially when the English course is part of a degree course and the marks contribute to the student\u2019s degree classification.<\/p>\n

The purpose of the workshop was to look at ways of using profile forms to assess such writing assignments. I was not satisfied that we came to a satisfactory solution at that time, and I’m still not happy with it.<\/p>\n

Typically writing is marked using some kind of profile form where students are given marks for criteria such as grammatical accuracy, organisation, spelling, cohesion and coherence, referencing etc. There are many problems associated with using such profile forms and this is what I wanted discuss. These problems need to addressed formally when the course is part of a degree. Some of the problems are:<\/p>\n

    \n
  1. what happens if the student ignores the question?
    \nwhat happens if the student doesn\u2019t answer the question completely?
    \nhow do we handle content?<\/li>\n
  2. what happens if the student writes too may words?
    \nwhat happens if the student doesn\u2019t write enough words?<\/li>\n
  3. what do we do about plagiarism?<\/li>\n
  4. how important is accuracy?
    \nhow important is handwriting?
    \nwhat about word-processing mistakes (failure to handle the computer)?
    \ncan one mark sheet be used to help markers to be consistent and also to give feedback to the student?<\/li>\n<\/ol>\n

    I did not really want to look at the problems of constructing such profiles, but the problem of what these profiles can and cannot handle. There have been several important publication justifying the use of these profile forms. Jacobs, Zinkgraf, Wormuth, Hartfiel, & Hughey (1981), for example show that holistic scoring works. But what criteria should we use and how should they be weighted? One of the earliest uses of such profile, the Foreign Services Institute, uses Accent, Grammar, Vocabulary, Fluency, Comprehension at 6 levels for spoken language (Oller, 1979, pp. 320-326; Valette, 1977, pp. 157-161). For writing the following criteria have been used:<\/p>\n

    Mullen (1980)
    \nControl of English structure,
    \nCompositional organisation,
    \nQuality of writing,
    \nAppropriateness of vocabulary<\/p>\n

    Hughes (1989, p. 95) ELTS
    \nCommunicative quality
    \nOrganisation
    \nArgumentation
    \nLinguistic accuracy
    \nLinguistic appropriacy<\/p>\n

    Bachman & Savignon (1986) – Oral
    \nGrammar
    \nCohesion
    \nSensitivity to register<\/p>\n

    Hamp-Lyons (1991a)
    \nContent
    \nArgument
    \nText structure features – cohesion
    \nEvidence of planning – coherence
    \nLanguage<\/p>\n

    Hamp-Lyons (1991a) MELAB
    \nTopic Development
    \nOrganisation
    \nConnections
    \nSentence Structure
    \nVocabulary
    \nMechanics<\/p>\n

    Hamp-Lyons (1991a) Michigan Writing Assessment
    \nIdeas & Arguments
    \nRhetorical Features
    \nLanguage Control<\/p>\n

    Jacobs, Zinkgraf, Wormuth, Hartfiel, & Hughey (1981)
    \nContent
    \nOrganisation
    \nVocabulary
    \nLanguage use
    \nMechanics<\/p>\n

    Hamp-Lyons & Henning (1991)
    \nCommunicative quality
    \nInterestingness
    \nReferencing
    \nOrganisation
    \nArgumentation
    \nLinguistic accuracy
    \nLinguistic appropriateness<\/p>\n

    Hamp-Lyons (1991b)
    \nCommunicative quality
    \nOrganisation
    \nArgumentation
    \nLinguistic accuracy
    \nLinguistic appropriacy<\/p>\n

    However, this is what EAP teachers think. In this context, Bridgeman & Carlson (1983) surveyed undergraduate English faculty and subject specialist faculty to find out what they thought were important..<\/p>\n

    They found that in order of importance, the English faculty thought the following criteria were important:<\/p>\n

    1. Paper organisation
    \n2. Development of ideas
    \n3. Paragraph organisation
    \n4. Addresses topic
    \n5. Overall writing
    \n6. Sentence structure
    \n7. Appropriate to audience
    \n8. Assignment requirements
    \n9. Quality of content
    \n10. Vocabulary usage
    \n11. Punctuation\/spelling
    \n12. Vocabulary size<\/p>\n

    Whereas the subject specialist faculty prioritised the criteria differently<\/p>\n

    1. Quality of content
    \n2. Assignment requirements
    \n3. Addresses topic
    \n4. Development of ideas
    \n5. Appropriate to audience
    \n6. Paper organisation
    \n7. Overall writing
    \n8. Vocabulary usage
    \n9. Paragraph organisation
    \n10. Sentence structure
    \n11. Punctuation\/spelling
    \n12. Vocabulary size<\/p>\n

    It is useful to compare these findings with the criteria used by the various authors mentioned above. These can be summarised:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
    English Lecturers<\/b><\/td>\nSubject Lecturers<\/b><\/td>\n<\/td>\na<\/b><\/td>\nb<\/b><\/td>\nc<\/b><\/td>\nd<\/b><\/td>\ne<\/b><\/td>\nf<\/b><\/td>\ng<\/b><\/td>\nh<\/b><\/td>\n<\/tr>\n
    1<\/td>\n6<\/td>\nPaper organisation<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    2<\/td>\n4<\/td>\nDevelopment of ideas<\/td>\n<\/td>\n?<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    3<\/td>\n9<\/td>\nParagraph organisation<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n?<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    4<\/td>\n3<\/td>\nAddresses topic<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
    5<\/td>\n7<\/td>\nOverall writing<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    6<\/td>\n10<\/td>\nSentence structure<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    7<\/td>\n5<\/td>\nAppropriate to audience<\/td>\n?<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/tr>\n
    8<\/td>\n2<\/td>\nAssignment requirements<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
    9<\/td>\n1<\/td>\nQuality of content<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
    10<\/td>\n8<\/td>\nVocabulary usage<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    11<\/td>\n11<\/td>\nPunctuation\/spelling<\/td>\n<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
    12<\/td>\n12<\/td>\nVocabulary size<\/td>\n<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
    English Lecturers<\/b><\/td>\nSubject Lecturers<\/b><\/td>\n<\/td>\na<\/b><\/td>\nb<\/b><\/td>\nc<\/b><\/td>\nd<\/b><\/td>\ne<\/b><\/td>\nf<\/b><\/td>\ng<\/b><\/td>\nh<\/b><\/td>\n<\/tr>\n
    9<\/td>\n1<\/td>\nQuality of content<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
    8<\/td>\n2<\/td>\nAssignment requirements<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
    4<\/td>\n3<\/td>\nAddresses topic<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
    2<\/td>\n4<\/td>\nDevelopment of ideas<\/td>\n<\/td>\n?<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    7<\/td>\n5<\/td>\nAppropriate to audience<\/td>\n?<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/tr>\n
    1<\/td>\n6<\/td>\nPaper organisation<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    5<\/td>\n7<\/td>\nOverall writing<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    10<\/td>\n8<\/td>\nVocabulary usage<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    3<\/td>\n9<\/td>\nParagraph organisation<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n?<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    6<\/td>\n10<\/td>\nSentence structure<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n\u2713<\/td>\n<\/tr>\n
    11<\/td>\n11<\/td>\nPunctuation\/spelling<\/td>\n<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
    12<\/td>\n12<\/td>\nVocabulary size<\/td>\n<\/td>\n<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n\u2713<\/td>\n<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    a. Mullen (1980)
    \nb. Hughes (1989, p. 95) ELTS
    \nc. Hamp-Lyons (1991a)
    \nd. Hamp-Lyons (1991a) MELAB
    \ne. Hamp-Lyons (1991a) Michigan Writing Assessment
    \nf. Jacobs, Zinkgraf, Wormuth, Hartfiel, & Hughey (1981)
    \ng. Hamp-Lyons & Henning (1991)
    \nh. Hamp-Lyons (1991b)<\/p>\n

    Similarly, Norton (1990) compares students’ understanding of what lecturers want in assignments with the lecturers own views. There is some disagreement, but mostly they are in agreement:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
    Mean ranking of students’ essays and tutors’ criteria for assessment of<\/td>\n<\/tr>\n
    <\/td>\nMean ranking<\/em><\/td>\n<\/tr>\n
    Criteria<\/em><\/td>\nStudents(N = 98)<\/em><\/td>\nTutors(N = 6)<\/em><\/td>\n<\/tr>\n
    Answer the question<\/td>\n1.5<\/td>\n1.0<\/td>\n<\/tr>\n
    Content\/knowledge<\/td>\n1.8<\/td>\n–<\/td>\n<\/tr>\n
    Relevant information<\/td>\n2.3<\/td>\n2.0<\/td>\n<\/tr>\n
    Understanding<\/td>\n2.4<\/td>\n1.3<\/td>\n<\/tr>\n
    Wide reading<\/td>\n2.9<\/td>\n3.7<\/td>\n<\/tr>\n
    Structure\/organisation<\/td>\n3.3<\/td>\n2.8<\/td>\n<\/tr>\n
    Argument<\/td>\n3.5<\/td>\n1.5<\/td>\n<\/tr>\n
    Evaluation\/own views<\/td>\n3.9<\/td>\n3.5<\/td>\n<\/tr>\n
    Presentation\/style<\/td>\n4.2<\/td>\n3.5<\/td>\n<\/tr>\n
    English\/spelling<\/td>\n4.8<\/td>\n4.0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    What is common to these findings is that that subject lecturers value content. I do not think that most EAP marking profiles that I have seen deal with this adequately. Having a category giving, say, 20% to content does not seem to me to be satisfactory, as it is possible to ignore the content – or the question – totally and still pass the English exam. When I talk to examiners about this they always \u00a0agree with me and say that – of course – they take that into account when marking,but the marking profile forms they use do not take it into account. Should they? Can they? How?<\/p>\n

    References<\/b><\/p>\n

    Bachman, L. F. & Savignon, S. (1986). The evaluation of communicative language proficiency: A critique of the ACTFL oral interview. The Modern Language Journal, 70, <\/i>380-390.<\/p>\n

    Bridgeman, B. & Carlson, S. B. (1984). Survey of academic writing tasks. Written Communication, 1<\/i>, 247-280.<\/p>\n

    Hamp-Lyons, L. & Henning, G. (1991). Communicative writing profiles: An investigation of the transferability of a multiple-trait scoring instrument across ESL writing assessment contexts. Language Learning, 41,<\/i> 337-373.<\/p>\n

    Hamp-Lyons, L. (1991a).Scoring procedures for ESL contexts.. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts <\/i>(pp. 241-276).<\/i> Norwood, NJ: Ablex.<\/p>\n

    Hamp-Lyons, L. (1991b). Reconstructing \u201cAcademic writing proficiency\u201d. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts <\/i>(pp. 127-153).<\/i> Norwood, NJ: Ablex.<\/p>\n

    Hughes, A. (1989). Testing for language teachers. <\/i>Cambridge: Cambridge University Press.<\/p>\n

    Jacobs, H. L., Zinkgraf, S. A., Wormuth, D. R., Hartfiel, V. F. & Hughey, J. B. (1981). Testing ESL composition: A practical approach.<\/i> Rowley, Mass.: Newbury House.<\/p>\n

    Mullen, K. A. (1980). Evaluating writing proficiency in ESL. In J. W. Oller & K. Perkins (Eds.), Research in language testing<\/i> (pp. 160-170). Rowley, MA: Newbury House.<\/p>\n

    Norton, L. S. (1990). Essay writing: What really counts? Higher Education, 20,<\/i> 411-442.<\/p>\n

    Oller, J. (1979). Language tests at school.<\/i> London: Longman.<\/p>\n

    Valette, R. M. (1977). Modern language testing <\/i>(2nd ed.). New York: Harcourt Brace Jovanovich Inc.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Assessment Criteria I ran a workshop at a BALEAP conference several years ago about assessment criteria. I was particularly interested in the difficulty of marking assignments, especially when the English course is part of a degree course and the marks … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[3],"tags":[14,18,28],"class_list":["post-526","post","type-post","status-publish","format-standard","hentry","category-eap-general","tag-assessment","tag-eap","tag-higher-education"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/uefap.org\/blog\/wp-json\/wp\/v2\/posts\/526","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/uefap.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/uefap.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/uefap.org\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/uefap.org\/blog\/wp-json\/wp\/v2\/comments?post=526"}],"version-history":[{"count":0,"href":"https:\/\/uefap.org\/blog\/wp-json\/wp\/v2\/posts\/526\/revisions"}],"wp:attachment":[{"href":"https:\/\/uefap.org\/blog\/wp-json\/wp\/v2\/media?parent=526"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/uefap.org\/blog\/wp-json\/wp\/v2\/categories?post=526"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/uefap.org\/blog\/wp-json\/wp\/v2\/tags?post=526"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}