Short Duration Online Assessment Considerations

Schools that wish to offer one or more on-line traditional exams and/or one or more on-line open book exams in AY 2020/21 are asked to request approval from AQSC. Approval is conditional on the proposing School providing assurance that it has made or assured adequate provision for the support of all students taking the assessment. Therefore, it is worth considering whether your learning outcomes can be assessed by a means other than a Short Duration Online Assessment. It’s worth seeking advice and guidance from CHEP, the Faculty’s Associate Dean for Education, and your Deputy Head of School for Education.

It’s also worth keeping in mind that the University has little experience with Short Duration Online Assessments delivered remotely to students’ own devices; computer-based exams run centrally in the past have focussed on the use of controlled environments such as Workstation computers and Faculty laptops.

In terms of support, while ServiceLine is available to resolve technical queries there are no current plans to provide dedicated and scalable support to “in flight” Short Duration Online Assessments.

Contact your Deputy Head of School Education to find out more about this process.

Be mindful of those students outside of the UK time zone when selecting certain times for tests to be available .  Allowing the assessment to be taken at any time of day will be more inclusive for those outside of the UK.

ExamStart is a tool used to secure computer-based, invigilated exams. It only works on University workstations within the University network. It can only be used by students who are physically attending exams on campus.

OnDemand is the new name for Questionmark Perception. It is a multiple choice / defined response system that integrates with our ExamStart system.

A limited number of licences for OnDemand are available to the University. To manage this demand those who have already used OnDemand in the past may continue to do so. New authors should only be recommended OnDemand if no other tool can serve the specific requirement needed (such as Review and Record if purchased).

At least four weeks from a University mandate and the provision of a charge code for purchase.

Blackboard allows for tests to be built within a Blackboard course. A number of question types are available and tests can be deployed with a variety of options. Tests reside within their own course and can be delivered only to students on that same course. Tests may be copied from one course to another by instructors. Blackboard tests had been recommended only for formative use previously and OnDemand for summative use. As such iSolutions does not have experience supporting the use of Blackboard tests for summative use.

When making a Blackboard test, instructors are completely in control of all aspects of the setup and deployment of the test.  Conversely with OnDemand there has been a technical checking process because the scheduling of the assessment is done by iSolutions.

In Blackboard there are two main question types for receiving text entry.

Essay: Blackboard state that there is no word limit.

Short Answer: Blackboard state that there is no word limit.  You can set how many lines the answer box should have, up to 6 lines.  This set an expectation of how long the answer is expected to be.

In OnDemand there is no word limit.

Our strong recommendation is to avoid using short answer and essay question types in Blackboard or OnDemand assessments.  It is much more efficient to use Blackboard Assignments, eassignment, or Turnitin if you require students to write short answers or essays.  There is also a current issue when writing feedback for short answer or essay questions in Blackboard tests: the text editor does not show the buttons for formatting text, so only plain text can be entered when writing feedback for these question types.

Refer to the comparison table below for further information on question types.

In Blackboard and OnDemand tests all question types except for Short Answer, Essay, and File Response are marked automatically.  It is much more efficient to use Blackboard Assignments, eassignment or Turnitin if you require students to write short answers, essays or upload files.

Refer to the comparison table below for further information on automatic marking of different question types.

No. When creating a test using a file import process, only a spreadsheet based file can be used. You may find it more practical to use the PDF exam paper as the basis for a Blackboard assignment, where students could upload a scan of their answer which could then be marked on line using Bb Annotate, which includes pen based annotation. The use of Blackboard tests tends to require a bit of thinking first about which question type is most appropriate, and whether the question should be changed in any way to better suit the potential of using a Blackboard test. Creating computer-based assessments like these tends to be front loaded in terms of effort, with the return on investment coming over time from less work required for marking.

Not in Blackboard tests.

In OnDemand you can divide assessments into sections, using question blocks, but students cannot move between those question blocks once a block has been submitted.

In Blackboard and OnDemand tests, yes.

In Blackboard and OnDemand tests, yes.

In Blackboard tests, yes.  In Blackboard you can upload a video file up to 2500 MB into the question text box, but without using Panopto students may not be able to play that video.  However due to the recurrent issues that students report of problems where they have to “click to play” videos hosted on Panopto, it is not recommended to use videos in questions without thorough testing before hand.  The use of video increases the internet bandwidth required to complete a test.

Videos are not recommended in OnDemand. Videos are normally hosted externally and we do not recommend embedding external content.

In Blackboard tests both the questions and the order of answers to questions (where applicable) may be randomised.

Blackboard: You can build pools of questions, these are contained within the course in which you are building them.  You can then build tests that take for example 5 questions at random from pool A, 10 questions from pool B, and 2 questions from pool C.

The use of question pools is for confident advanced users because diagnosing issues with assessments based on question pools is by its nature more difficult.

OnDemand: Question banks are supported, but normal advice is to use the pool to make a static exam per sitting so that if a question is incorrect none or all of the students are disadvantaged.

Step by step guides are available on elearn.southampton.ac.uk

Blackboard: Leave the “show test results and feedback to students” options empty in the Test Options. Set the ‘hide from students’ setting for column in Grade Centre and make sure the score is not included in the running total.  See https://elearn.southampton.ac.uk/blackboard/assessment/hiding-or-showing-marks-to-students/
OnDemand: Yes.

Blackboard: You can create questions in excel, using a specific format specified by Blackboard, save it as a tab separated text file and then upload to Blackboard.

OnDemand: Yes.

Blackboard: See https://elearn.southampton.ac.uk/blackboard/testandsurveyresults/ and https://elearn.southampton.ac.uk/blackboard/itemanalysis/

OnDemand: view this guide

General approach:

Typically tests tend to focus on the ability to recall and organise information.  Design test questions to focus more on higher order and critical thinking that requires the application, analysis, and evaluation of learning, rather than simple knowledge recall.

Blackboard: with the current functionality and features available to us it is very difficult to prevent cheating.

Measures that would require additional cost and are not currently available at UoS:

Lockdown the browser:  A commercial add on is available which will lock down the test taker’s computer.  This still does not prevent test takers from photographing the screen during the test, or working with others or accessing resources using another device.  The use of lockdown browsers tends to introduce increases in support calls from test takers and tends to restrict the type of device they may use to take the test.

Online proctoring: While there are many commercial online proctoring services there are just as many guides on how to bypass them.  The use of online proctoring may also have privacy implications and would not necessarily have been part of what test takers believe they “signed up for”.

Mitigation techniques for Blackboard tests:

1. Use question pools and randomisation.

While building question pools is much more effective in OnDemand, you can build them in Blackboard.  Question pools can be used to make it more likely that test takers will receive different questions when taking the test.

When building the questions, you can select for the answers to be displayed in a random order, for example in multiple choice and multiple answer questions.

When deploying the test you can select that the questions are shown in a randomised order.

2. Ensure no feedback is shown once the test is complete.

When setting the deployment options set not to show any feedback to students, such as correct answers or what the possible answers were.

3. Make the test available during a tighter time period.

The less time that the test is available, either through using the availability criteria or by setting a timer, the less time there is for the questions to be shared between test takers.   The other side of this coin is that the use of timers or limited availability restrictions can have a negative impact, see the risk minimisation table.

4. Set questions to appear one at time.

If questions are shown all at once it is much simpler for a test taker to take a photo of the whole test and share it quickly onwards.

5. Create a “pre-test” that has to be completed before the real test becomes available.

You could create a “pre-test” with questions such as “I will follow the University’s academic integrity policy” and “I will ensure I have read all the guidance before taking the test” where using adaptive release the real test will not become available until this pre-test has been satisfactorily completed.

6. Use the Blackboard test access log

After a test has been taken you may use the Blackboard test access log to review a detailed list of all the interactions a student had with the test.

You can’t. Students can take a picture on their phone, or write down the questions, or video themselves taking the assessment.

See above.

To our knowledge, online proctoring is not available at UoS.

No

Blackboard: not specifically for tests, but you can read our Blackboard accessibility statement for information about Blackboard accessibility.

Blackboard: Yes, go to https://elearn.southampton.ac.uk/blackboard/deploytest/ and scroll to the “Exceptions” section.

OnDemand: Yes, although all wording and guidance is to not have time limits as the system is designed for invigilators to time students

In uncontrolled environments such as where students are completing computer-based assessments outside of an invigilated University computer based exam location there is no way to assure that problems will not occur.  However, a  number of mitigation strategies are suggested below.

No.

Blackboard: In most circumstances yes, although there are Blackboard Knowledge Base issues that state that sometimes this does not happen. We recommend students click the Save Answer button manually as they answer questions.

OnDemand: Yes.

To our knowledge this is not planned.

In all circumstances it is very beneficial to run a short formative assessment using the same question types and options that would be used in a summative assessment.

This allows students to get used to the interface and process and allows staff to ensure that the results they receive are what they expect.

Further recommendations to make to students are:

• Ensure you are using a computer (not mobile or tablet) on a supported operating system with a supported web browser.
• Ensure your computer is plugged in to a power source.
• Ensure your computer is connected to the internet using a wired (Ethernet) connection.
• Using an incognito window, access Blackboard via https://blackboard.soton.ac.uk direct, not via Sussed.  Use only this one tab to access Blackboard.  Do not open Blackboard in multiple tabs.
• Select the Save Answer boxes, if available, as you progress through the test.
• When using a mouse, do not double click on any buttons or boxes.
• Do not use the browser “back button”, navigate the test using Blackboard’s own navigation interface.
• When the test has been completed, take a full screen screenshot of the page that states your assignment has been submitted, and ensure the data and time is shown.

Identify any students with additional exam requirements (AER) are expected to take the test.  There may be some who require more time than others.

Using the Blackboard groups tool, create a manual enrol group for each AER group.  Name each group so it will be clear which AER group it refers to, e.g. “AER + 30 mins extra”. Set “Group is visible to students” to  No and tick “Create Smart View for this group” add the corresponding students to each group.

Follow the steps in the guide to deploying the test, settings that are particularly relevant for a Short Duration Online Assessment are listed below.

Open test in new window: No – this reduces the possibility of students accidentally logging out of Blackboard in a different window and being locked out of the test

Make available to students: Yes – students will not be able to access the test unless this option is set.

Multiple Attempts: tick and set a least two attempts.  If a student is locked out of their test they can start at least one more attempt before requiring technical support or raising an appeal.

Score attempts using: highest mark.  If more than one attempt is taken, the mark of the highest attempt is used.

Force Completion: Do not tick.  If you tick this box, then once participants have begun the Test, they may not exit and continue working on it at a later date.  This means if students lose their internet connection or have a power outage while taking the test they will not be able to complete it.

Set timer: If you do use a timer be aware that students who lose their connection to the quiz (e.g. network or power issue, closing tab accidentally.

Display after/until: tick the boxes and set the dates and times to correspond with dates and times that the assessment should be accessible to students.

Password: to avoid increased support requests, do not set a password.

Exceptions: add exceptions for AER student groups, you can allow more attempts, extra time if using a timer, and allow for extended display until dates.

Due Date: do not set a due date, use the display until settings to set when the test is no longer accessible.  We recommend this because if you allow exceptions for AER students they will not affect the due date, and the due date is not a setting which can be added to exceptions.

Include this Test in Grade Centre Score Calculations: Do not tick – this will prevent the score of the test affecting the running total in the Grade Centre, potentially giving students an indication of their score even if results are hidden.

Hide results for this test completely from the instructor and the Grade Centre:  Do not tick: ticking this box will mean the results will never be recorded, once students start taking this assessment this option can never be undone.

Show test results and feedback to students: untick everything.  In most circumstances for summative assessment the marks should not be shown to students after they have completed the assessment.  This can be adjusted later to reveal feedback at a set date if desired…

Test presentation: One at a Time – this keeps the screen space small, and keeps a connection with Blackboard when students move to the next question.

Prohibit Backtracking: Do not tick.  If this is ticked students will not be able return to an earlier question.

Randomise Questions: Tick if you wish for the order of questions in the test to be presented randomly.  This affects the question order.  Note that other areas of randomisation are:

• When building a test question you may specify if possible answers (where applicable) should be presented in a random order.
• When building a test you can state that there should X number of questions chosen at random from a question pool. Multiple pools can be used.

Follow the steps in this guide to ensure that the results will not be visible to students.