The new initial and diagnostic assessments from bksb have been created to ensure that each candidate is given a unique set of questions and is identified at an accurate level. During early development, the phrase “random questions” was frequently touted, but it quickly became apparent that this would be a hazardous route to take. That’s because there are two very simple truths which should immediately deter anyone from using an assessment based on “random questions” or a “randomised question bank”, namely:
no two questions are equal in terms of difficulty;
and where multiple-choice and free-text questions are present within the same assessment, candidates answering the former will receive an advantage.
As an example, if the questions 5×6 and 7×8 were given to candidates for whom multiplication was a challenge, they would not have equal pass rates. One would be more difficult than the other. Consequently, the random selection of questions would introduce an unacceptable element of “luck” into any assessment. And whilst the difference in difficulty between one question and another might well be small, when the sum total of those differences was multiplied over 20 questions or more, the overall effect would undoubtedly be significant.
Moreover, where an assessment blends multiple-choice questions which can be guessed with free-text questions which cannot, controls must be introduced to balance the value of all correct answers.
For instance, if candidates were offered one of the three questions below, it is clear that the increased chance of guessing question 2 would give those who received it a distinct advantage over those presented with questions 1 or 3.
Question 1: 8×9=?
Answer: a)72 b)71 c)73 d)67
Question 2: 6×5=?
Answer: a)32 b)31 c)30
Question 3: 7×4=?
Furthermore, even where the mixture of question types within a selection is equal, the presence of such mixes still creates problems when calculating results. Consider, for example, that the previous three questions form a sequence. If a mark is given for each, and 2 marks is considered a pass, then a number of problems will be encountered.
First, even though these questions are clearly unequal in terms of their difficulties, in the final calculation they are nonetheless awarded identical values.This means that there is an 8% chance that a candidate can pass simply by guessing the first two questions. Secondly, the third question is surely a better indicator of ability than the other two. Should we therefore allot it 2 marks and change the pass mark to 3? We may be able to agree that question 3 is more significant, but is it exactly twice as significant as the other two? Moreover, if we accept that question 3 is more significant – due to there being no opportunity to guess the answer – then to a lesser extent question 1 is more significant than question 2, because with more answer options, there’s a reduced chance of someone guessing it. Beware, therefore, any assessment which features “randomised questions”.