Assessing Assessment Tools

“To transform learning and teaching, we must start by transforming assessment, feedback and reporting,”

– Kathleen Donohoe

There is considerable potential for multimedia technologies to make feedback richer and more personal and for a wider range of learner skills and attributes to be demonstrated through assignments involving, for example, e-portfolios, blogs and wikis. In addition, online tools can support peer and self-assessment in any location and at times to suit learners – the value of peer and self-assessment in developing learners’ ability to regulate their own learning is increasingly recognised” (JISC, 2010: 11)i *

Technology enhanced assessment incorporates far more than computer-based assessment, incorporating opportunity for ‘stealth assessment’ (Oldfield, Broadfoot, Sutherland and Timmis), summative and formative assessment, valuable, continuous feedback loops and immersive experiences, all integrated to create a deep learning experience.

The nine considerations and criteria below provide an oversight of long identified requirements, aligned with current and emerging innovation and technical capabilities. They should be considered from the perspectives of students, educators, administrators and leadership, at school, jurisdictional and/or national level. All stakeholders can benefit from meeting the criteria below, but for many, these will be game changers.

These considerations should be the benchmark expectations of effective technology (platforms, systems, software, apps, devices) to support assessment, backed by pedagogical practice:

Is equity supported through:

• Consistent experience and benchmarking of results across the state, acknowledging the disparity that exists.
• Access to undertake assessments online and offline, with or without network access, and through network disruption.
• Meeting accessibility compliance and universal design across the platform and assessment activities, for all users – including capabilities for text or image-based tasks.
• Offline task development and marking capability.
• An array of options for providing feedback (voice, commenting).
• Anonymous candidature marking, ensuring transparency and reducing bias.

To what degree does the technology support:

• Invigilation capabilities that ensure effortless oversight of attendance, progress during assessment and completion, for all students.
• Online monitoring of all candidates during assessments if required (without overloading teachers) e.g. utilising traffic light systems alerts.
• Improved plagiarism detection and resolution through invigilation monitoring and alerts, mechanisms for students to test their compliance.
• Academic integrity through locking down of restricted applications during assessment.
• Privacy and security for data around student and their results, through relevant profiles integrated across the system.
• Security around assessments and marking requirements.
• The capability for all users to electronically log procedures and report on their own case history for training or audit purposes.

How agile and adaptable are the assessment platform, bank and tasks:

• To allow and address context, individual, localised or jurisdictional requirements?
• To compliment, and integrate into existing infrastructure, through minimal change management and rapid implementation strategies.
• To suitable assessment methodologies for your level, subject, compliance of policy.
• Customisation requirements for personalisation.
• In keeping with the wide range of assessment methodologies available. Simulations, MCQs, vocational education requirements.
• To support the current and emerging array of assessment tools and capabilities, with consideration of age, level, stage, devices and operating systems?
• To support and align summative and formative assessment?

Is validity and reliability supported through evaluation and reporting:

• Tracking all assessments and making use of the data available to determine and improve reliability, relevance and validity of assessment items at local and strategic level?
• To support feedback loops encouraging teacher improvement.
• To inform strategic oversight, initiatives, implementation, benchmarking and scaling of exemplar practice at school, jurisdictional level.
• For required audit and reporting processes.
• On task statistics, supporting comparators on achievement, identification of areas of weakness for a student or group of students, in syllabus items or in the assessment task.
• To inform effective strategies and provide feedback on teaching approaches and initiatives.
• To determine exemplar marking guides.

Is there capability and transparency for:

• Tagging assessment tasks and questions against syllabus outcomes, learning outcomes, level of difficulty / challenge, general capabilities and across faculty, to support reporting on coverage and connection: syllabus, skills etc.
• Alignment to frameworks such as learning progressions.
• An approval process supporting the addition or amendment to the bank of assessment tasks.

How seamless is:

• Access through a single login with other elements of the eLearning ecosystem and the linked administrative systems.
• The experience, traversing learning activities, assessments and feedback, do they inform each other?
• The assessment experience in the event of network or other failure?
• Teacher oversight of progress etc during assessment to inform interventions or dis-engagement.

Defined by:

• What level of training is required? For whom and at what stages?
• The minimal work required by teachers, leadership and support staff through assessment design, test bank building, testing and marking?
• The capabilities and support for ‘closing the loop’, utilising the data to inform changes to or scaling of practice.
• The assessment banks and their scalability, sustainability and adaptability, locally and strategically?
• A user-friendly experience, navigation and interface – from the perspective of all stakeholders?

Can the technology effectively support:

• Alignment with other systems to ensure transparency over all aspects of learners.
• Adaptability during the learning experience, informed by achievement, skill, motivation and resilience.
• Effective and efficient capabilities, whilst considering initial and on-going cost considerations (licensing etc) and support requirements.
• Assessment tasks that provide immediate feedback (with or without teacher intervention).
• The entire assessment journey, supported by question banks, randomisation, immediate feedback to reinforce content, inform next / future assessment tasks, ubiquitous access to assessment tasks and results?
• Agile assessment journeys that react to achievement in previous assessments, current assessment, across subjects, as required.
• Practice tests to motivate and build confidence.
• The provision of a horizontal and vertical picture of student achievement and improvement – across years and across subjects.
• Increased learner agency, autonomy and self-regulation.
• Teacher professional learning and development, to inform development required, then through the development process;
o To assess professional skills and attitude to provide evidence of appropriate everyday pedagogical competences.
o To manage professional learning to record and reflect on activities undertaken.

Does the technology support:

• Collaborative learning (for teachers and students).
• Co-creation and sharing of modules and assessment banks, designed, maintained and evaluated at local or strategic level.
• Communities of practice focusing on assessment, informed by review and data.
• One to many capabilities: Self-assessment, Peer assessment and group assessment, supporting demonstration of individual and group work, continuous assessment reinforced by monitoring, encouragement and review, personal development and learning throughout the course, non-bias feedback and a mix of assessment patterns.
• Collaboration in assessment design between schools, across jurisdictions.

Technology enhanced assessment – moving forward

Oldfield, Broadfoot, Sutherland and Timmisii recommend several areas for future research:
• “Cultivate new assessment practices based on principles and theories of learning
• Develop new assessment tools that reflect pedagogical principles
• Construct new responses to the current emphasis on high-stakes summative assessment
• Respond to ethical challenges presented by the use of digital technologies in assessment
• Consider new contexts relevant to assessment using digital technologies, including learners’ lives and social, cultural, educational and technical backgrounds”
As the technical capabilities evolve and technology enhanced assessment becomes the norm, these issues will be addressed to ensure an adaptive, seamless experience for students, teachers and administrative support. This must be a co-created and co-implemented vision between pedagogical experts and technical experts, considering and harnessing all the above, ratified by agile policy reflecting current and emerging capabilities.
i JISC, 2010: 11, cited in Oldfield, Broadfoot, Sutherland and Timmis
ii Oldfield, Broadfoot, Sutherland and Timmis: Technology Enhanced Assessment: Review of the Literature Available at: https://www.bristol.ac.uk/media-library/sites/education/documents/researchreview.pdf

Kathleen Donohoe
Director, School Learning Environments and Change at School Infrastructure NSW

Kathleen is the Director and Founder of Leading Thinking International. She supports school and industry to reimagine learning experiences across physical, virtual, and social learning space, bringing national and international teaching, academic, research, and executive experience. She is an Honorary Research Fellow at the University of Melbourne, MGSE, with the ILETC Project. Kathleen has vast experience developing and implementing strategy for transformation, curriculum renewal, and physical virtual and social learning space, including as Director, Futures Learning at NSW Department of Education.

*ref:https://www.linkedin.com/pulse/assessing-technology-enhanced-assessment-kathleen-donohoe/

Learn more about Elumina’s online exam and assessment solutions here: Exam Management