The ESEE is not so easy: Results of readability analyses

In a previous post I described my attempt to respond to one of the Essential Skills for Employment and Education (ESEE) practice test items, the dosage chart for a liquid medicine. Since then, I have taken a close look at the ESEE test itself.

The test has a major problem if adults with less than a high school level education are the intended test-takers. The test items in the reading section, including the six pre-test items, are written at a  senior high school reading level (Grades 11-12).  

I used two online readability tools to analyse all the reading items. Each readability site uses a number of  analyses to produce an aggregate average grade equivalent. The two sites I used are Readability Score and Analyse My Writing. After reviewing a few sites I chose these two because they produced consistent results with the same text and they produced similar results with the same text when compared to each other. I also ran two texts scored by hand through one of the sites and the results were the same.

The scores in the first two columns represent the average scores produced by each site. Readability Score produced an average of Grade 10.8 and Analyse My Writing stated the average of the six pretest items was Grade 11.3.

The pretest items are supposed to be “mid-Level 1” questions used to determine if a test-taker should take the “easier” Type A test or the more difficult Type B test.

I also looked specifically at one readability tool, the Fry formula. The results of this analysis are in the third column. A Fry analysis was also used on test items developed for use in the international literacy surveys. The average reading level of test items used in assessments like the IALS and PIAAC is Grade 8, with a range from Grade 2 to Grade 15.  Why are ESEE pretest items so much more difficult with no pretest items less than Grade 8?

I was quite taken aback when I saw this and decided to analyse all 23 reading items.  Although there are a total of 45 test questions in what is called Type A and Type B sections of the test, one item may have two to three test questions. Here are the overall results:

• Readability Score average = Grade 11.6
• Analyse my Writing average = Grade 12.6
• Fry average = Grade 10 (only 6 items are Grade 10 or lower and 17 are above Grade 11, including nine items rated at Grade 15 or 15+)

These 23 items represent Levels 1-5 and are basically the same reading level as the six pretest items claimed to be at at mid-Level 1.

Interestingly, no reading items are included in the practice questions. These questions are made available to those who would like to get a better understanding of the ESEE and prepare to take the test. Why are there no reading items in the online practice test available here?

Such difficult test items are a major concern if the ESEE is supposed to produce a measure of program effectiveness. Nearly half the adults enrolled in LBS in 2014-2015 have less than a high school level education. Perhaps this test was designed for the other half who have a high school education or more? But, according to LBS program suitability criteria, learners enrolled in programs should have less than a Grade 12. This too is a factor that is measured and appears in the PMF. What gives? Who is the test for?

Also important to know about the methods used to design the ESEE and other spin-off tests like PDQ, TOWES and Education and Skills Online, all of which have been considered for suitability to produce a learner gains measure, the text to be read isn’t the thing that makes these tests difficult. It’s the test questions that make the tests difficult. When test developers analysed what they refer to as the complexity variables, readability contributes the least to test difficulty and the likelihood that a test-taker can respond to a test item. The difficulty of those Grade 11-12 reading levels is compounded, possibly putting the tests out of reach for anyone with less than a high school level education.

Currently, 33 programs and 45 sites are participating in the Learner Gains Research Project using the ESEE. The aim is to have 1,700 learners take both a pre- and post-test. That means 1,700 adult learners and those who work with them will find out the hard way that this test is not written for anyone with less than a senior high school reading level. Initial conversations with a few programs indicate that both learners and educators are confused, frustrated, upset, and demoralized by the test. I will share their experiences in future posts.

In the meantime, programs may want to modify the criteria they use to exempt learners from taking the test. Here is the criteria (learners are exempt if the response is “no” to any question):

1. Recognizes letters of the alphabet and the sounds associated with them
2. Recognizes sight words such as our, which, eight, something
3. Decodes new and unfamiliar words quickly using phonetic rules
4. Fluently reads a 3 – 4 sentence paragraph, re-states what has
   been read and answers a question about the paragraph

I suggest a fifth criteria be added to the exemption list:

5. Reads a variety of unfamiliar texts at a Grade 11-12 level.