Suppose that you think students should be knowledgeable about the rights and responsibilities of information creation. Furthermore, they should be able to recognize social, legal, and economic factors affecting access to information. These two statements form the basis of the Module 4 – The Value of Information – of the Threshold Achievement Test for Information Literacy (TATIL). In this post, I will describe the development of TATIL test knowledge questions. How do we go from a concept to a set of fully formed, sound test questions?
It begins with outcomes and performance indicators written by members of the TATIL advisory board and inspired by the ACRL Framework for Information Literacy. An iterative process of review and revision guided by the TATIL project leader Dr. April Cunningham results in the foundation for writing test questions.
Test question writers are recruited from many different libraries and institutions types, including community colleges to liberal arts schools to research universities. Under April’s guidance, question writers go through the following process:
- Learn the question-writing guidelines
- Become familiar with the question template
- Meet with assigned question-writing partner
- Select a performance indicator
- Generate ideas about a realistic topic or scenario
- Make a provisional choice about question type
- Draft a question
- Write out the stem
- Write correct answer or answers that will demonstrate knowledge related to the performance indicator
- Use knowledge of misconceptions or typical errors to write incorrect answers
- Once satisfied with the question, complete the template document for that question, including:
- Description of what the question will test
- Mental behavior tested: Understanding, problem solving, critical thinking
- Intended level of performance: novice or high level
- Explanation of reasoning behind the response options
- References
- Submit the question for review
- Start on next question.
All drafted question are reviewed by the project leader and two or more librarians who are not writing items with suggested revisions or problems passed back to the item writing team.
If a question makes it out of this process (remember, not all bills become law) then it is grouped with other questions for review by undergraduate students in a procedure called cognitive interviewing. A librarian sits with a student as they go through a set of test questions, listening for problems with comprehension or meaning. Most questions that emerge from the cognitive interview process go through additional revision, usually fairly minor syntax changes to eliminate ambiguity or misleading terminology. Some questions do not survive the cognitive interview process.
The step after cognitive interview is field testing. Here is where we rely on institutional partners willing and able to collaborate on large-scale testing. Each field test manager gives the test to a number of students at their college or university. Data gathered through field testing are sent out for psychometric analysis. This is the stage where we learn how well the questions perform as a group, how difficult the questions are, and whether each question has the ability to discriminate between novice and high-level performers.
Field testing leads to additional revisions of the questions. As with the other evaluative steps, some questions are eliminated after the first round of field testing. Questions and performance indicators may be cut for various reasons. Most often the question cannot be reworded or revised in such a way that deals with the identified problems while continuing to address the performance indicator. Sometimes we realize that the performance indicator is too similar to another indicator.
The final procedure is to put the revised questions through another round of field testing. Results from that round may require some necessitate fine-tuning before the questions are ready to go into production.
The first two modules of TATIL are now in production. Modules 3 and 4 are in their second round of field testing and will be available for use in summer 2018.
Test question development: A real-life example from Module 4, The Value of Information
TATIL Outcome 4.2
Recognize social, legal, and economic factors affecting access to information.
TATIL Performance indicator 4.2.3
Identify reasons that one person’s views are not disseminated to the larger community.
Version one
A mother of a six-year-old child living in a poor community, noticed that her son was suddenly acting differently. Also, many residents had been complaining that the water tasted and smelled strange. They told local government officials, who did nothing with the information for nearly two years. After a professor of engineering did some tests, and a local doctor confirmed that many children had lead poisoning, the city then acted on the information to announce an emergency and advise residents to only use bottled water.
Why did the city officials not react immediately to the complaints of the local residents?
- The evidence was biased and had to be verified before action could be taken.
- The examples and evidence did not come from scientists hired by the city.
- The examples and evidence were provided by marginalized people without influence. [correct]
- There was not yet enough scientifically tested evidence to make such an important decision.
Problems identified through cognitive interviewing: Students were confused why an engineering professor would conduct water tests and were concerned about the use of the word poor.
Solution: Clarify and reword.
Version two
A mother living in an impoverished community observed that her six-year-old son was acting strangely. Around the same time, many of her neighbors noticed that their household water smelled and tasted odd. The residents took their concerns to local government officials, who took no action for two years. After a professor of civil engineering conducted some tests and a local doctor confirmed that many children had lead poisoning, the city acted on the findings to announce an emergency and advise residents to use only bottled water.
Why did the city officials not react immediately and investigate the complaints of the local residents?
- The evidence was biased and had to be verified before action could be taken.
- The examples and evidence did not come from scientists hired by the city.
- The examples and evidence were provided by marginalized people who had no influence with the city. [correct]
- There was not yet enough scientifically tested evidence to make such an important decision.
Problem identified by field testing: A large proportion of high-performing people are drawn to option 4.
Solution: Rewrite option 4.
Version three
A young mother living in an impoverished and under-developed community observed that her young son was behaving strangely. At the same time, she and her neighbors noticed that their household water had a bad smell and taste. Concerned about the quality of the city water, the residents talked about their concerns with local government officials, but the officials took no action.
Two years later, after media reports about questionable water quality, a professor of civil engineering at the local university conducted tests and a local doctor confirmed that many children had lead poisoning. At that point, the city acted on the findings to announce an emergency and advise residents to use only bottled water.
Why did the city officials not react immediately to the residents and investigate their concerns?
- The examples and evidence were biased and had to be verified before action could be taken.
- The examples and evidence did not come from scientists hired by the city.
- The examples and evidence were provided by marginalized people who had little influence with the city. [correct]
- The city officials had not experienced any problems with the water themselves.
This question is now in a second round of field testing along with the other Module 4 questions. In June we will evaluate its performance and decide on its fate: good to go, tweaking needed, or cast out (unlikely at this stage).