Volume 11, Issue 2
July 30, 2010
Beyond Portfolios: Using Technology to Enable Course Level Outcome Assessment
In their book Disrupting Class, Christensen, Horn, and Johnson (2008) assert that innovation occurs when an organization discovers a new model for delivering goods or services to non-consumers which creates a market on a new plane from the predominant existing market. The rise of the online, for-profit university served as such a disruptive change to the higher education community.A key contribution of the online medium has been its sharp focus on learning outcomes along with its ability to document everything that takes place in the classroom. This article demonstrates a new approach to student learning assessment with a focus on course outcomes which are mapped to program goals to provide meaningful data to educators as we look for ways to improve the student learning experience in both fully online and blended learning environments.
Methodology
Barr and Tagg (1995) were among the first to discuss the shift from an instruction to a learning paradigm nearly 15 years ago. In the intervening years, the pendulum has slowly shifted toward a focus on outputs (learning outcomes) and away from teaching inputs which have historically driven quality improvement efforts. The ultimate solution will involve a balanced approach that includes incentives both for good teaching and for student performance on learning outcomes.Outcome based education proponents believe that a focus on content area knowledge is superseded by a commitment to making sure students demonstrate proficiency against stated learning outcomes.Content is still important but faculty and course authors must use it to design student assignments that yield observable and measurable results that prove students have met the goals of a learning outcome (Jansen, 1998). When implemented correctly, students know exactly what they will be expected to learn as part of a course or program and faculty receive valuable data to inform needed changes in curriculum and instruction.
Figure 1 below illustrates a typical university learning outcome hierarchy. Institutions begin with broad mission statements which are supported by program level goals, as well as by course, task or unit level outcomes. Accreditors have historically focused on program level assessment; however, with the increased calls for accountability, they are beginning to ask for more discrete outcome assessment at the course level.
Figure 1. Typical Higher Ed Outcome Hierarchy
A major challenge with program level portfolio assessment is that the evaluation takes place after the student has finished the course where the assignment was produced, so it's usually too late to remediate if there were performance gaps. This assessment model encourages faculty to assess within the course which provides more discrete information on exactly where students are excelling and where they are struggling. This approach also allows for remediation towards underperforming outcomes before the student completes the course.
Program level assessment can then be supported by two separate data streams. The first involves course level outcome performance data that is aggregated and averaged up to the program level via a curriculum mapping hierarchy that is defined by teaching faculty and populated with data by the faculty of record. The second data stream involves direct program level assessment against the same artifact but uses a separate, higher level evaluation rubric and external evaluators. This parallel assessment method provides a way for departments to monitor inter-rater reliability between course faculty and independent program level evaluators.The In-Course outcome performance report illustrated in Figure 2 below shows the data that is available to faculty who use the Pearson LearningStudio Learning Outcome Manager solution. Learning statement IDs are displayed across the top and students are listed along the left. Looking at the table horizontally allows the instructor to evaluate individual student performance against all outcomes in the course while a vertical analysis provides a view into howall students are performing against the outcomes within the course. This data can also be reported across multiple terms and sections, allowing for significant outcome performance analysis over time. In addition, institutions can use this data to monitor at-risk students which should lead to improved in-course retention rates.
Figure 2. In-course Outcome Performance Report
The final step in creating a solid culture of assessment is to close the loop by actually evaluating data that is collected and by making changes to curriculum or instruction where needed to improve student performance levels. This allows departments to track changes made to courses over time to demonstrate how these changes have impacted student learning. Incorporating technology to support this process involves some up-front setup along with ongoing maintenance; however, this work is often offloaded from teaching faculty which allows them to focus on providing substantive feedback to students while gathering more discrete data to diagnose underperforming students before it's too late.
Conclusions
A dual session at the 2009 Higher Learning Commission Annual Meeting in Chicago entitled "Facilitating Assessment of Student Learning: Insights from HLC Team Mentors" presented a series of fundamental questions along with characteristics of success and risk which they've identified over several years of working with over 500 universities on best practice assessment. This session was led by mentors who have guided university leaders that participate in the HLC's Academy for Assessing Student Learning.The session handout includes an excellent summary of what was presented and is a good start for developing a best practice assessment program. The fundamental questions begin with a recommendation to get broad ownership among all campus stakeholders on the development of the institutional mission all the way down to the course level learning outcomes. The mentors then followed an outcome through the entire creation, delivery, evaluation and stakeholder reporting process suggesting best practices and pitfalls to avoid for each step.
A key takeaway from this session was a strong recommendation that university assessment committees should simplify evaluation processes and only collect the amount of data that they can analyze. A common pitfall is that institutions are swimming in so much data that they don't know where to begin. One example given to support this recommendation was that departments should consider collecting data on a limited set of courses and cycling through the curriculum over a period of years instead of trying to hit every course each year. They also suggested simplifying outcome rubrics to ensure that faculty can provide quality feedback on a limited set of criterion instead of burdening professors with complex schema that require more time than is feasible for quality evaluation.The most critical step is that faculty need to get engaged in the design and implementation of a campus assessment process. As I consult with institutions, nearly everyone identifies gaps that are embarrassingly obvious once faculty engage in the development of an assessment plan. The important thing is that these gaps are identified and corrected which is what continuous improvement is all about.
Brian McKay Epp, M.Ed.Academic Trainer and Consultant
Pearson eCollege
First published in the Proceedings of Global Learn Asia Pacific 2010 Conference
References
Barr, R., & Tagg, J. (1995). From Teaching to Learning - A New Paradigm for Undergraduate Education. Change, November/December, 13-25. Retrieved from http://ilte.ius.edu/pdf/BarrTagg.pdfChristensen, C. M., Horn, M. B., & Johnson, C. W. (2008). Disrupting Class: How Disruptive Innovation Will Change the Way the World Learns. New York: McGraw-Hill.
Fager, J., Adler-Kassner, L., & Verschelden, C. (2009, April). Facilitating Assessment of Student Learning: Insights from HLC Team Mentors. 2009 NCA HLC Annual Meeting, Chicago. Retrieved from http://www.ncahlc.org/download/annualmeeting/09Handouts/fager_GMON0130c_GMON0230c.pdfJansen, J. D. (1998) Curriculum reform in South Africa: a critical analysis of outcomes-based education, Cambridge Journal of Education. 28 (3) p. 321-331. http://www.tandf.co.uk/journals/titles/0305764X.asp
McKay Epp, B. (2010). Creating a Culture of Assessment: Technology Enhanced Outcome Management. In Z. Abas et al. (Eds.), Proceedings of Global Learn Asia Pacific 2010 (pp. 414-419). AACE.
Instructor's Tip
Online instructors, especially those who are new to teaching online, can have a strong concern that they will fail to make a connection with their online students.Using announcements in your online course can be an effective contribution in your efforts to be "present" in your course and assist with creating a continuous instructor-student connection.
A recent enhancement to the Announcement tool in Pearson LearningStudio allows faculty to adjust the default look of an announcement subject line to draw students' attention to the announcement area or to a particular entry more effectively.When adding or editing an announcement, you will now notice a small-version of the visual editor next to the subject-line entry field. You can use the options in this visual editor to apply formatting enhancements to your announcement subject line such as Bold, Italics, Color, or a combination of these. You can also apply additional enhancements by utilizing HTML tags in your announcement subject line.
Try using one of the following HTML tag examples (with your announcement title inserted between the tags) to enhance the default look of your announcement subject line.Example #1:
Use this code to have your announcement subject scroll across the line area:<marquee>Announcement Subject Line</marquee>
Example #2:
Use this code to create a scrolling subject line with bolded, larger text:<marquee><strong><font size="3">Announcement Subject Line</font></strong></marquee>
Example #3:
Use this code to create a yellow background for a fixed announcement subject line:
<span style="background-color: yellow;"><strong>Announcement Subject Line</strong></span>
Announcement Subject Line
If you like to use the announcement tool in Pearson LearningStudio to post an Instructor welcome to the course, key policies and procedures, assignments and due dates or other important course information, take advantage of this recent enhancement to help you capture your online students' attention more effectively.
Note: As a further enhancement, the announcement character limitation has been doubled to 500 characters in order to accommodate the use of HTML tag characters. Please keep in mind that script tags and java script will not run in the announcement subject-line field.Rachel Cubas , M.Sc.
International Academic Trainer & Consultant
Pearson eCollege
Related Links:
Contact Us | Careers | Privacy Policy | Terms of UseCopyright 1999-2009 Pearson eCollege®
Phone: (888) 884-7325