{"id":1330,"date":"2021-05-17T05:58:42","date_gmt":"2021-05-17T12:58:42","guid":{"rendered":"https:\/\/edtechuvic.ca\/edci565\/?p=1330"},"modified":"2021-05-17T08:49:51","modified_gmt":"2021-05-17T15:49:51","slug":"alignment-and-the-assessment-triangle","status":"publish","type":"post","link":"https:\/\/edtechuvic.ca\/edci565\/alignment-and-the-assessment-triangle\/","title":{"rendered":"Alignment and the Assessment Triangle"},"content":{"rendered":"
\n
2021 <\/span><\/div>\n
Image: Jamie Morrison on Unsplash<\/a><\/span><\/div>\n<\/div>\n
\n

 <\/p>\n

In this topic we are going to look at some strategies to help you plan your assessments to increase the validity and reliability of the inferences that you draw from the evidence of learning produced during your course. If you are going to have confidence in the judgments that you make about learners and their learning, it makes sense to put some significant thought into how you come to those judgments. Lipnevich et al. (2020) report that over 78% of syllabi they examined indicated a reliance on summative exams for determining at least some of the grades assigned to learners, so it seems safe to say that exams are a popular assessment instrument, possibly because they are familiar to many instructors, they appear to be relatively easy to create and administer, and selected-response tests are super easy to score, especially if automated.<\/p>\n

In this topic, however, we hope to equip you with some ideas and strategies for ditching your selected-response test. After all, it is unlikely to be giving you accurate information about the learning happening in your course.<\/p>\n

To begin, let’s think about the components of a well-considered assessment plan. The National Research Council (2001) writes about the idea of an assessment triangle<\/em> comprised of three equal and interdependent parts: cognition,<\/em> or a model of the construct to be learned; observation<\/em>, a performance assessment of some sort, and; an interpretation<\/em>, or inference<\/em> based on the data gathered in the performance.<\/p>\n

\"\"<\/a><\/p>\n

Assessment Triangle (National Research Council, 2001)<\/p>\n

The assessment triangle is a tremendously useful tool in planning for assessment because it gives us a clear pathway from cognition<\/em> to observation<\/em> to inference,<\/em> as well as clear guideposts to ensure that an assessment plan is properly aligned<\/em>.<\/p>\n

Cognition<\/h3>\n

The foundation of the triangle, cognition<\/em>, is understood as a model of the knowledge and skills in a domain, mastery of which wold represent competence. This model includes not only the content of the domain but also how learners typically engage with learning the content and in which order. For example, learners typically learn addition<\/em> prior to learning multiplication<\/em> in mathematics. This model might be quite simple at early levels of learning, but extraordinarily complex as the ability level of the learners increases.<\/p>\n

Most often, in designing learning experiences, this modelling takes place during the process of identifying and refining learning outcomes, at the very beginning of the process (although it may be a much more detailed process in some instances, such as for domains which require licensure, like nursing or accounting). Engaging in this process clarifies the domain of the course or learning experience and is both limited by the nature of the domain, and also delimited by the choices made by the instructor. The process also makes explicit the appropriate cognitive levels in relation to the outcomes such as learners being able to recall, describe, compare, critique, analyze,<\/em> or create.<\/em><\/p>\n

Observation<\/h3>\n

The NRC report describes the observation<\/em> component of the assessment triangle as being a “set of specifications for performance tasks that will elicit illuminating responses from students” (2001, p. 48). It is critical that this performance task be directly and explicitly linked (aligned<\/em>) to the cognitive model represented in the learning outcomes in order to ensure that the inferences derived from the data are valid representations of the learner’s ability in the domain.<\/p>\n

In the design process, this is done by mapping assessments (i.e., performance tasks, portfolios, test items, etc.) to the learning outcomes and to the identified cognitive skill. For example, if an outcome is “Learners will be able to apply<\/strong> critical theory to issues related to gender in schools.”,<\/em> then the assessment only requires learners to recall<\/em><\/strong> and describe<\/em><\/strong> critical theory, then there is a lack of alignment between the cognitive model and the observation of learner ability.<\/p>\n

Interpretation or Inference<\/h3>\n

The NRC describes assessment as a process of “reasoning from evidence” (p. 42). As we learned in the previous unit, and reiterate here, measuring learning is at best<\/em> an imprecise estimation<\/a>. In large-scale assessments, psychometricians use detailed statistical models such as Classical Test Theory (CTT) or a number of different item response theories (IRT) to calculate the probability<\/em> that a learner has obtained a particular level of ability in relation to the cognitive model. The trouble for classroom assessments, which are administered to groups that are much smaller than the required N<\/span><\/span><\/span><\/span><\/span>. As such, classroom assessment inferences are more often inferred based on more qualitative or intuitive models.<\/p>\n<\/div>\n

The inferential model used in classroom assessments is important because the inferences we make form the judgments we communicate to the University with respect to how well learners have obtained the required knowledge or skills in the domain. In the design process, instructors should think carefully about the rubrics they use to guide their inferences and also about how much confidence they can have in their inferences.<\/p>\n

Rubrics<\/h3>\n

The UVic Undergraduate Grading Scale <\/a>reflects a rubric that can be used to guide instructor interpretations of learner creations. It provides five clearly delineated categories (4 passing categories and 1 failing) of achievement and rich descriptions of each category to guide inferences. Another rubric that you might consider using is the SOLO Taxonomy (Biggs & Collis, 1982). SOLO stands for Structure of the Observed Learning Outcome, and it also describes five clearly delineated categories which can be used to guide inferences about student work.<\/p>\n

\"\"<\/a><\/p>\n

The SOLO Taxonomy<\/p>\n

Planning for Assessment: The Blueprint<\/h3>\n

In the next few posts, we will work through the process of blueprinting a course in preparation for developing and producing the content. Central to this blueprint process is ensuring that the cognitive model<\/em> of the domain of your course is in alignment with the observations<\/em> you will make which will provide evidence to support your inference,<\/em> leading finally to a judgment in the form of a letter grade.<\/p>\n

\n

Optional Further Reading<\/h3>\n

Read Chapter 2 of Knowing What Students Know: The Science and Design of Educational Assessment,<\/em> available online for free at this link<\/a> and compose a short post about something you found to be important, or unclear, or perhaps a question you have on the backchannel. This might be a great time to join the backchannel (if you haven’t already) and introduce yourself!<\/p>\n

References<\/h2>\n
\n
Biggs, J., & Collis, K. (1982). Evaluating the quality of learning: The SOLO taxonomy<\/i>. Academic Press.<\/div>\n
<\/div>\n<\/div>\n
\n
Lipnevich, A. A., Guskey, T. R., Murano, D. M., & Smith, J. K. (2020). What do grades mean? Variation in grading criteria in American college and university courses. Assessment in Education: Principles, Policy & Practice<\/i>, 27<\/i>(5), 480\u2013500. https:\/\/doi.org\/10\/ghjw3k<\/a><\/div>\n<\/div>\n
\n
\n
<\/div>\n
National Research Council. (2001). Knowing What Students Know: The Science and Design of Educational Assessment<\/i>. The National Academies Press. https:\/\/doi.org\/10.17226\/10019<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

2021 Image: Jamie Morrison on Unsplash   In this topic we are going to look at some strategies to help you plan your assessments to increase the validity and reliability of the inferences that you draw from the evidence of… Continue Reading →<\/a><\/p>\n","protected":false},"author":3,"featured_media":1333,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-1330","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-assessment"],"_links":{"self":[{"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/posts\/1330","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/comments?post=1330"}],"version-history":[{"count":11,"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/posts\/1330\/revisions"}],"predecessor-version":[{"id":1348,"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/posts\/1330\/revisions\/1348"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/media\/1333"}],"wp:attachment":[{"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/media?parent=1330"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/categories?post=1330"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/edtechuvic.ca\/edci565\/wp-json\/wp\/v2\/tags?post=1330"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}