What makes the spin-off tests so darn difficult (and frustrating)?

All spin-off tests share common elements since they adhere to the basic test development methods used to construct test items in the international literacy tests. Here are the main reasons that they are difficult and an inappropriate measure  in the context of literacy learning programs:

1. The spin-offs claim to be “authentic” and representative of a task done in everyday life. Test-takers are duped into thinking they can apply commonsense problem-solving, only to find out this won’t work.
2. All test items are no-name items. They are generalized and do not contain recognizable symbols, logos or names, the very elements used by readers to help make important connections to previous knowledge and experiences using similar texts.
3. The tests do not contain language and literacy development indicators that are commonly used in educational and literacy/language development assessment. For example, there are no questions related to word-definitions, the main idea of a sentence, correct grammar use, audience, organization of the text, etc.
4. Questions and responses require high-level scanning skills, which are acquired and put to use after a reader has developed solid comprehension skills and an ability to respond to a wide variety of materials and formats. A test-taker has to know how a text works and what it means before being able to quickly scan for relevant bits of information. To make this even more challenging when completing a spin-off test, a test-taker must use this skill with unfamiliar texts!
5. Test-items incorporate confusing and misleading information called distractors. Not only will a test-taker need solid reading comprehension skills but they will also need enough test-taking knowledge and experience to not get frustrated and misled by distractors.
6. Scores are produced using a predictive algorithm and not whether one is right or wrong on all questions. A score will be assigned before the learner has had a chance to complete the test.
7. Results are only used for accountability and have no connection to curriculum, course completion or transition into other programs or courses.

To better understand what it’s like to take a test, here is an example test item from the ESEE practice test available online here. I attempted to respond to the item using my experience as a reader and repertoire of literacy. In other words, I didn’t assume the perspective of a potential learner. Take a look at the item and note your own responses and processes before reading mine and see if we encounter some of the same challenges.

The first thing I read is the test question. I understand at this point that I have to find the correct dosage amount of some sort of medicine for a child who weighs 32 lbs. I’m familiar with short forms for weights and volume so I begin to scan and navigate the chart noting the bolded titles: LBS, Kg, mL.

I then get tripped up by the title of the chart: Single Oral Dose. What is this? Why is this the title of the chart? Isn’t dosing information only in the last column?

This leads to other questions. What is this medication anyway? Is it cough syrup? Why not state that?  I know the test items don’t include brand names but why not accurately identify the medication? That would help a test-taker figure out the meaning of the word dose and mL.

I get back to making a connection between the test question and the chart information. I’m holding two bits of information in my mind: 1) child and 2) weighs 32 lbs. But I get side-lined again. Since I’m thinking about finding information for a child, I notice the bolded term Under 12. I scan this column assuming it’s a category containing ages. But hold on, what is this column? It’s not children’s ages. I look across the row. Are those ages? What’s Under 5.5? Now I’m confused. What am I looking for again? Ages? Weight? I go back to the question.

Okay then. New tactic. I ignore all information related to age and just look for weight. Where is 32 pounds in the chart? I just need to find the number 32. I find 31-36. I know 32 fits in there. I look back up the columns and realize that these are indeed weights, including the misleading term Under 12. This is likely what is called a distractor, a confusing and misleading bit of information intentionally placed in the test to help determine test item difficulty.

I look at the chart again. I look at the test question and the one sentence preamble. I’m still wondering why I got confused by Under 12. Oh, this test item isn’t really related to children’s medication! It’s about medication for infants and toddlers. Why direct me to think it’s for children, which usually means children and teens? Hence, my focus on the term Under 12. Is this also a distractor or simply a poorly written test item? Ugh. I’m now frustrated because I know my commonsense thought processes are being manipulated.

I continue, feeling frustration mounting. Forget the question and just look at the answers. What are the possible responses? I see each of the answers duplicated in the mL column. The test-taker will need to know that this column contains dosage numbers and mL is a measurement for liquids. All of this must be inferred based on a great deal of background knowledge and familiarity reading charts, understanding weights and measures, the meaning of short forms and experience reading a variety of medicine labels. I focus on the 31-36 cell and follow the row across to determine that 2.5 ml is the likely answer. Hold on, why is the category title mL and the response ml? Argh! I choose 2.5 ml for my answer. I dread going on to the next question.