Skills and Criterion Referenced Assessment: Turning LWB236 Real Property A Law into a Subject for the Real World [2007] JSPL 6; (2007) 11(1) Journal of South Pacific Law 56

	Home \| Databases \| WorldLII \| Search \| Feedback Journal of South Pacific Law

SKILLS AND CRITERION REFERENCED ASSESSMENT: TURNING LWB236 REAL PROPERTY A LAW INTO A SUBJECT FOR THE REAL WORLD

In 2006, the LWB236 Real Property A Law teaching team in the School of Law at the Queensland University of Technology (QUT) is replacing a research essay assessment task with a more authentic assessment task that will better equip students for the real world. The new assessment task is a drafting exercise and memorandum of advice that will embed and assess more lawyering and generic skills. In particular, it will develop legal research skills, legal analysis skills, drafting skills, written communication skills and document management skills. It will also give the students the opportunity to develop autonomy and self-confidence.

In addition to the new assessment task, the teaching team is also implementing a new assessment regime, that is, criterion referenced assessment in accordance with the new QUT Assessment Policy. The outcome of this will be increased validity, reliability and transparency of the assessment task. The criterion referenced assessment sheet was designed to indicate the alignment between the new assessment task and the learning outcomes of LWB236 Real Property A. The design was informed by a criterion referenced assessment sheet used in a first year undergraduate core law subject, LWB143 Legal Research and Writing, and the one used in LWB237 Real Property B, to recognise that law students incrementally develop their skills as they progress through the law degree. The successful implementation of criterion referenced assessment will depend on the measures taken to ensure that there is a shared understanding of the criteria and performance standards between the markers and students.

This conference paper will discuss these plans for turning LWB236 Real Property A into a subject for the real world in 2006.

Law students are not a homogenous group because they enrol in a law degree with diverse backgrounds and varying skills. However, the literature suggests that they have a common view that their degree ‘will better enable them to succeed in professional employment, assist them to make career changes, strengthen their potential for a more personally fulfilling life, or some combination of these’.^[1] This student demand has driven the need for law schools to focus on embedding and assessing skills. Skills that are essential to practice law are commonly referred to as lawyering skills, whereas those skills that may be transferred to many contexts are commonly referred to as generic skills. Skills should not be learned in a one-off or haphazard manner. This conference paper considers how students should incrementally develop skills across three levels as they progress through the law degree. It also discusses how LWB236 Real Property A fits within this integrated approach to embedding and assessing skills.

To assess skills, Queensland University of Technology (QUT) is in the process of implementing criterion referenced assessment as opposed to norm referenced assessment. It is anticipated that all law units will implement criterion referenced assessment by the end of 2007.^[2] LWB236 Real Property A will be introducing it in semester 1 2006. This conference paper distinguishes criterion referenced assessment from norm referenced assessment. In particular, criterion referenced assessment results in greater validity, reliability and transparency. The designers of the criterion referenced assessment sheets will need to take an integrated approach to implementing criterion referenced assessment to ensure that there is a logical progression of skills across the three levels.

Christensen and Kift unpack the development of skills into three levels. At level 1, students are ‘instructed on the theoretical framework and application of the skill, usually at a generic level. This skill may be practised under guidance and feedback provided. Assessment will usually include a critique of the skill as practised’.^[3] Level 1 is notionally the equivalent to the first year undergraduate core units in the law degree. Level 2 builds on level 1 and is notionally the equivalent of the second year undergraduate core units. It requires ‘a degree of independence... This may involve some additional guidance at an advanced level of the skill, an environment in which to practise the skill in a real world legal scenario, and feedback to students on their progress. Students will be encouraged to reflect on their performance and on ways to improve. At this level, individually or within a group, a student should be able to complete a task utilising a range of skills in relation to a simple legal matter’.^[4] Level 3 builds on level 2 and is the equivalent of the third and fourth year undergraduate core units. It requires students to ‘draw on their previous instruction and transfer the use of the skill to a variety of different circumstances and contexts without guidance. Students should be able to adapt and be creative in the ways they approach the context and use particular skills. Reflection on performance will be a key aspect. At this level, individually or within a group, a student should be able to complete a task utilising a range of skills in a complex legal matter for a knowledgeable and critical audience.’^[5]

LWB236 Real Property A is a second year undergraduate core unit. It is notionally a level 2 unit. Previously, it embedded and assessed skills in a compulsory 1500 word assignment on native title that was completed in teams.^[6] The students were also required to complete a research methodology indicating their research strategies, resources consulted and the outcomes of their research. They were also required to peer review another team’s research methodology and reflect on their teamwork skills. After reflecting on this assessment task, the teaching team decided to introduce an assessment task that was authentic and learner-centred.

The new assessment task planned for 2006 is a drafting exercise and a memorandum of advice. It is more authentic and learner-centred because it provides the students with autonomy to practise lawyering and generic skills in a real world legal scenario. It will develop skills at level 2 and thus build on the skills learned in the first year of the undergraduate law degree in LWB143 Legal Research and Writing, for example, legal research skills, legal analysis skills, written communication skills and document management skills. To support the development of these skills at level 2, students will be referred to additional resources. These skills are further developed at level 3 in LWB434 Advanced Research and Legal Reasoning.

The new assessment task will also introduce students to drafting skills, for example, drafting documents relating to the transfer of title. Drafting skills are not embedded or assessed in the first year of the law degree at QUT. Thus, the development of drafting skills in LWB236 Real Property A will be at level 1. To support the development of drafting skills, students will be instructed on drafting principles and provided with the necessary scaffolding. The assessment task requires the students in a memorandum of advice to critique drafting skills as practised. The students will be required to complete the assessment task by themselves and in this way it will develop their self-confidence. Consequently, the new assessment task is more authentic and will better equip students for the real world. Drafting skills are further developed in LWB237 Real Property B. After determining the type of assessment and the skills to be assessed, it is necessary to consider whether the assessment will be norm referenced or criterion referenced.

Norm referenced assessment ranks a student’s performance against their peers in a particular cohort. A marker using a norm referenced approach to assessment must rely on some sort of criteria to attain raw scores that are fitted into a pre-determined distribution,^[7] which is commonly referred to as a “bell curve”.^[8] For this reason, it could be described as a hybrid of both criterion referenced assessment and norm referenced assessment, but the dominant marking strategy is norm referenced.

The alternative to norm referenced assessment is criterion referenced assessment. This involves marking students against clear and attainable criteria and performance standards as opposed to their peers. It might also be described as a hybrid of both criterion referenced assessment and norm referenced assessment because the marker should monitor the spread of results to ensure that the marks are not clustered at the extremes.^[9] When monitoring the spread of the results, the marker should reflect on whether the assessment task had the appropriate degree of complexity and whether the markers had a shared understanding of the criteria and performance standards with the students.

In 2006, LWB236 Real Property A will be implementing criterion referenced assessment. The benefits of this approach include greater validity, reliability and transparency.

The validity of an assessment task measures the ‘desired learning outcomes’.^[10] The validity of an assessment task using a norm referenced approach to assessment cannot be determined by analysing the pre-determined distribution of marks because it is possible that the student who received the top score did not achieve the desired learning outcomes. The raw scores would need to be analysed. The validity for norm referenced assessment depends on how the marker calculated the raw score. This is likely to be based on some sort of criteria even though it may be clandestine.

Criterion referenced assessment is more valid than norm referenced assessment because it specifically indicates the alignment between the assessment criteria and the unit objectives, that is, the desired learning outcomes. This alignment is demonstrated in the criterion referenced assessment sheet to be used in LWB236 Real Property A in semester 1 2006, which is extracted in Appendix 1. The assessment criteria are presented in the first column. The criterion referenced assessment sheet refers to the relevant unit objectives, which are set out in full in the LWB236 Real Property A study guide. The weightings attached to the criteria depend on the importance of the desired learning outcome.

Students are advised whether they have met the desired learning outcomes by ticks in the appropriate performance standard, individual feedback on the assessment item and additional comments at the bottom of the criterion referenced assessment sheet. This personalised feedback will also be supplemented with meaningful generic feedback on the online teaching site. The markers can use this feedback to inform their future teaching and learning approaches in the unit. As a result, the reliability (consistency) of feedback is vital.

The reliability of an assessment task measures whether the same student is marked consistently if they are marked more than once, and whether two markers mark a student’s performance consistently. Norm referenced assessment has been criticised for being unreliable. It treats the knowledge and skills of cohorts from year to year consistently and does not acknowledge that a cohort in one year may be better than the cohort in another year. Using the norm referenced approach to assessment means that a particular student may pass in one year, but fail in another year.^[11] The Centre for the Study of Higher Education recognises that norm referenced assessment is unfair to small cohorts because it exaggerates the difference between the students and is also unfair to large cohorts because it understates the difference between the students.^[12]

In contrast, criterion referenced assessment involves a prescriptive marking regime. It establishes performances standards for each criteria. In the exemplar in Appendix 1, the performance standards are presented across the page, that is, “excellent”, “very good - good”, “satisfactory” and “poor”. The term “excellent” usually equates to a mark within the range of 85-100 per cent. The term “very good - good” usually equates to a mark within the range of 65-84 per cent. The term “satisfactory” usually equates to a mark of 50-64 per cent. The term “poor” equates to a mark less than 50 per cent. Each performance standard specifies the weight attached to it. Allocating a narrow range of marks or a single mark to each performance standard will lead to greater reliability.

Defining each performance standard is a difficult process, which is refined in light of experience. The key is to anticipate the strengths and weaknesses in the student attempts at the assessment task. These strengths and weaknesses need to be articulated so that there is a clear limit between each performance standard. The process becomes more difficult as the number of performance standards increase. When drafting the “excellent” performance standards there is a need to avoid using descriptors that are almost impossible to achieve, for example, “All relevant issues considered”. There is also a need to make sure that the descriptors appropriately reflect the level of the performance standard, for example, “superficial analysis” would be inappropriate for the “satisfactory” performance standard.

Further, the assessment will be more reliable if each marker has a consistent understanding of the words used in the performance standards, for example, “persuasive”, “predominantly” and “basic”. One strategy that can be used to ensure that there is a consistent understanding of the criteria and performance standards is to invite peer review of the criterion referenced assessment sheet before it is released to students. This will give the markers a sense of ownership over the criterion referenced assessment sheet and generate interest in it. Another strategy is to provide the markers with marked examples of assessment using the criterion referenced assessment sheet. This will give the markers a greater understanding of how to apply the criterion referenced assessment sheet and advise them of the types of comments to be provided to students. In addition to the markers having a shared understanding of the criteria and performance standards, the students must also have a consistent understanding with the markers. This shared understanding is better achieved under criterion referenced assessment because it is more transparent.

The transparency of an assessment task measures whether the students understand what they are required to do. Norm referenced assessment does not clearly indicate to students what they need to do to be awarded a certain mark because they are marked against their peers. As a result, it forces students to be more competitive.^[13] This is an undesirable learning outcome for students because working independently and teamwork skills are both important generic relational skills. Criterion referenced assessment encourages students to focus on the desired learning outcomes as opposed to competing with their peers. It is transparent because it clearly articulates to the students the criteria, performance standards and how these are weighted. The transparency of the LWB236 Real Property A criterion referenced assessment sheet would be increased if the markers explained the wording of the criteria and performance standards to the students, for example, the definition of “predominantly” and “persuasive”. Further strategies to increase transparency would be to provide students with examples of marked assessment using criterion referenced assessment and to ask the students to apply the criteria and performance standards to a piece of assessment.

INTEGRATED APPROACH TO ASSESSING SKILLS AND IMPLEMENTING CRITERION REFERENCED ASSESSMENT ACROSS THE THREE LEVELS

The designers of criterion referenced assessment sheets need to recognise how a specific skill they are assessing is placed in the context of other units in the law degree. In particular, they need to know whether and how the skill has been previously assessed in earlier units in the law degree. They also need to know whether and how the skill is assessed in later units in the law degree. This knowledge is essential in taking an integrated approach to assessing skills and implementing criterion referenced assessment across the three levels.

The design of the LWB236 Real Property A criterion referenced assessment sheet has been informed by the units assessing the same skills earlier and later in the law degree. In particular, LWB236 Real Property A builds on the level 1 skills (legal research skills, legal analysis skills, written communication skills and document management skills) developed in LWB143 Legal Research and Writing and thus the design of the LWB236 Real Property A criterion referenced assessment sheet was informed by a criterion referenced assessment sheet used in LWB143 Legal Research and Writing. These skills are further developed at level 3 in LWB434 Advanced Research and Legal Reasoning. However, as at semester 2 2005, LWB434 Advanced Research and Legal Reasoning had not introduced criterion referenced assessment with descriptors for the performance standards. Further, drafting skills are not embedded or assessed in the first year law units. Thus, LWB236 Real Property A is the first unit to introduce drafting skills. Even though LWB236 Real Property A is a second year unit, it will be introducing drafting skills at level 1. The LWB236 Real Property A criterion referenced assessment sheet was informed by the one used in LWB237 Real Property B as that unit does assess drafting skills.

The designers of criterion referenced assessment sheets should aim to use the same number of performance standards in all units. QUT has seven grades of assessment, but the designers of criterion referenced assessment have chosen not to include seven different performance standards because this is much more onerous and is not worthwhile (or possible in whole or half marks) if a particular criterion is lowly weighted; for example, if it is worth a maximum of two marks. LWB143 Legal Research and Writing and LWB237 Real Property B provided four performance standards and LWB236 Real Property A followed suit.

However, after reflecting on the criterion referenced assessment sheets used in these units there is an area that can be continuously improved, that is, using terminology consistently in the units. For example, the middle performance standard should not be called, “satisfactory”, “sound” and “fair” in different units. It should be called one of these terms in all units. Similarly, other terms like “superficial analysis” or “writing style” should be attributed the same meaning across the units. This will enhance the shared understanding of the criteria and performance standards by markers and students.

Another area that needs to be explored is whether skills are being incrementally developed at levels 2 and 3, if the expectations of the level 1 skill development are too high. For example, the “excellent” performance standard used in LWB143 Legal Research and Writing (at level 1) for legal citation is, ‘All references correct and conform with style guide’. The word “all” suggests that something slightly less than perfect would not be excellent, which is unreasonable and perhaps an unrealistic expectation, particularly of students at level 1. If all references are correct at level 1, there is no scope for the students to incrementally develop citation skills at levels 2 and 3. There is also no scope for the designers of criterion referenced assessment sheets at levels 2 and 3 to incrementally expect more of the students. The criterion referenced assessment sheets implemented in level 2 and 3 units cannot simply repeat the performance standards implemented in level 1 units. Each level should build onto the previous level to demonstrate the logical incremental progression of the assessment of skills.

An integrated approach to the design of criterion referenced assessment sheets across the law degree needs to be taken. A best practice model of how this can be achieved for the development of a particular skill is presented in Appendix 2. For example, “excellent” at level 1 is only worth “very good to good” at level 2 and is only worth “satisfactory” at level 3. The outcome of this integrated approach will be to demonstrate how a student is incrementally assessed on a skill as they progress through the law degree.

The new assessment task and criterion referenced assessment regime in LWB236 Real Property A will better prepare students for the real world because it is more authentic and develops more skills. It will also lead to greater validity, reliability and transparency. However, there is still room for improvement and the LWB236 Real Property A criterion referenced assessment designers will be striving to adopt an integrated approach to the assessment of skills and implementation of criterion referenced assessment.
APPENDIX 1 – LWB236 REAL PROPERTY A – DRAFTING EXERCISE AND MEMORANDUM OF ADVICE ASSESSMENT CRITERIA AND FEEDBACK SHEET (20%)

STUDENT NAME:..............................................................................

Criteria	Excellent	Very Good - Good		Satisfactory	Poor	Marks
Drafting Skills – Unit objectives 10 and 11 Maximum 6
Understanding and application of drafting principles	Drafting demonstrates an excellent understanding of drafting principles; no obvious drafting errors; drafting complies with the law and meets client’s requirements; drafting is concise, does not include irrelevant information and has not produced unwanted results 5.5-6	Drafting demonstrates a good to very good understanding of drafting principles; precedents have been slavishly followed without creative thinking; meaning is apparent, complies with law, meets client’s requirements, but could be more concise, includes irrelevant information or has produced unwanted results 4-5	Genuine attempt to comply with good drafting principles; drafting is satisfactory in meeting some of the client’s requirements, but is deficient in some respects 3-3.5		Limited or no demonstrated understanding of drafting principles; drafting does not sufficiently satisfy the client’s requirements or comply with the law <3	/6
Legal research and analysis – Unit objectives 8, 9, 10 and 11 Maximum 8
Analysis of the issues in light of the relevant law and appropriate reliance on authorities to support arguments made in memo	Comprehensive level of analysis of the issues in light of the relevant law; no irrelevant issues considered; appropriate authorities relied upon and applied correctly 7-8	Persuasive level of analysis of issues in light of relevant law; some irrelevant issues considered; the appropriate authorities could have been applied in a more convincing manner 5.5-6.5	Satisfactory level of analysis of issues in light of relevant law; some issues missed; some appropriate authorities overlooked or incorrectly applied 4-5		Superficial or no analysis of the issues in light of the relevant law; arguments supported with no legal authorities or inappropriate legal authorities <4	/8

Legal writing and written communication – Unit objectives 9, 10 and 11 Maximum 6
Appropriate structure and use of memo formalities (e.g. To:, From:, Re:, Date) and summary of advice	Very professional structure; memo formalities included; headings and subheadings are consistent, concise, appropriately prioritised and flow logically 2	Predominantly professional structure; memo formalities included; headings and subheadings could flow more logically or be more consistent or concise 1.5	Some professionalism and an effort to structure; missing some memo formalities; headings and subheadings in need of refinement or inclusion 1	Limited or no professionalism; missing memo formalities; limited or no use of meaningful headings 0	/2
Legal citation conforms with the prescribed style guide and footnotes used appropriately	References in body and footnotes conform with style guide and there are no obvious errors in them; footnotes used appropriately 2	Predominantly references in body and footnotes conform with style guide; footnotes used appropriately 1.5	Generally references in body and footnotes conform with style guide; some footnotes include substance; more footnotes need to be included 1	Limited or no references in the body or footnotes conform with style guide; limited or no footnotes used appropriately 0	/2
Writing style including appropriateness for the intended audience, plain English, grammar and spelling	Very fluent writing style in plain English and appropriate for the intended audience; no obvious spelling, grammar, punctuation or typographical errors 2	Predominantly fluent writing style in plain English and appropriate for the intended audience; some spelling, grammar, punctuation or typographical errors 1.5	Generally meaning apparent but writing often not fluent; inappropriate use of legalese; several spelling, grammar, punctuation or typographical errors 1	Not appropriate for the intended audience; meaning unclear, complicated legal language used; several spelling, grammar, punctuation, typographical errors and lack of proofreading 0	/2

TOTAL: ......./20
APPENDIX 2 – BEST PRACTICE MODEL FOR INTEGRATING THE ASSESSMENT OF A PARTICULAR SKILL AND IMPLEMENTATION OF CRITERION REFERENCED ASSESSMENT ACROSS THREE LEVELS

^[∗] Lecturer, School of Law, Faculty of Law, Queensland University of Technology (QUT). GPO Box 2434, Brisbane Qld 4001 Australia, [email protected].
^[1]Teaching and Learning Committee Australian Technology Network, ‘Executive Summary’ in Report on Generic Capabilities of ATN Network Graduates (2000) http://www.clt.uts.edu.au/TheProject.htm#Executive.Summary (Accessed 20 July 2007).

^[2] QUT Teaching and Learning Committee, CRA Implementation Plan for Assessment Policy (2003)
http://www.library.qut.edu.au/academics/cra-infolit.jsp (Accessed on 25 October 2005).
^[3] S Christensen & S Kift, ‘Graduate Attributes and Legal Skills’ (2000) 11(2) Legal Education Review 207, 219.
^[4] S Christensen & S Kift , above n 3.
^[5] S Christensen & S Kift , above n 3.
^[6] Native title will continue to feature prominently in the unit, but will be assessed on the end of semester exam as opposed to a compulsory essay.
^[7] Centre for the Study of Higher Education, (2002) A Comparison of Norm-referencing and Criterion-Referencing Methods for Determining Student Grades in Higher Education
http://www.cshe.unimelb.edu.au/assessinglearning/05/normvcrit.html (Accessed on 25 October 2005).
^[8]Centre for the Study of Higher Education, above n 7.
^[9] QUT, Manual of Policies and Procedures (2003) cl 9.1.3 http://www.qut.edu.au/admin/mopp/C/C_09_01.html (Accessed on 25 October 2005).
^[10] QUT, above n 9.
^[11] Centre for the Study of Higher Education, above n 7.
^[12] Centre for the Study of Higher Education, above n 7.
^[13] S Jackson, A Project to Facilitate the Implementation of Criterion-Referenced Assessment in the School of Law (2004) https://olt.qut.edu.au/udf/FELLOW09/gen/index.cfm?fa=getFile&rNum=1638031&nc=1 (Accessed on 25 October 2005).