Messages of failure and not being “normal” are built into the ESEE

In a previous post, I wrote about the  difficulty of the Essential Skills for Education and Employment (ESEE), an assessment being piloted as part of the Learner Gains Research Project (LGRP). The research involves 1800 learners who are to take the ESEE at the beginning and end of their time in a program to see if the test could be used in all LBS programs (by some/most/all learners?) to generate accountability measures aligned with international literacy testing scores.

Based on what I’ve learned, things aren’t going so well for many learners participating in the LGRP. I’ve heard from a couple of study participants through the blog who said the test is simply too difficult, takes way too much time (at least twice the recommended time, and sometimes longer), and is simply inappropriate for LBS learners—even those in college programs who tend to have the highest levels of education.

I was also involved in a small study designed to collect some feedback from educators and program coordinators involved in the LGRP. People I spoke too also said the test is too difficult, much too long and completely inappropriate. It isn’t connected to the curricula in use (the K-12 curriculum and predominant approaches to teaching literacy). It isn’t even directly connected to the OALCF, which educators are discovering when they get different results when using the OALCF Milestones and the ESEE.

I have learned that some learners are simply refusing to take the post-test after negative experiences with the pre-test. I’ve also been told that some programs are losing learners. After taking the pre-test, they simply leave. Other learners stay and have shared mostly negative responses with instructors and coordinators, including frustration, incredulity and tears. One instructor said some of the learners she works with were “decimated” after taking the test.

Learners are put in an untenable situation. In addition to the very difficult texts to be read (Grades 11-12 on average), are disturbing messages of failure and inadequacy built into the ESEE.

Students who complete the test see the messages in the test’s summary report (one is available here on page 37) and reproduced below. At the end of the test, a report displays the results and an accompanying explanation. The explanation is needed because the test results aren’t meaningful on their own. Attempts to make the score meaningful are bewildering, demeaning and baseless.

Here is what the learner sees:

You received two Essential Skills scores: 1.     A score in the form of a level, such as Level 1. These scores start at Level 1 and can go as high as Level 4. 2.     A number score found in brackets, such as (250). Number scores normally range between 200 and 300. It varies, but most jobs require reading skills at 250 or higher.

The first part of the summary contains a statement that is simply bewildering: test-takers receive “a score in the form of a level.” So do they receive a score or a level?  The next sentence only adds to the confusion: “These scores start at Level 1 and can go as high as Level 4.” Huh?

The second part explains the meaning of a number score using the example of 250. The sentence reads: “Number scores normally range between 200 and 300.” A couple of things are happening here. First, a 100 point scale is presented along with a reference to a halfway point—250. This is readily interpreted to mean that 200-300 is basically the same as 0-100. Supporting the interpretation is the 250 example. The score is mentioned twice and seems to have significance. It falls precisely at the halfway mark of 200-300, just like 50 falls bewteen 0-100. A score of 250 is perceived to be the same as 50%. Students then conclude that anything less than 250 is a failure.

One instructor I spoke to said she attempted to explain that the scoring system works differently, but students aren’t convinced. After all, they are making reasonable assumptions based on the information provided and their experience with testing.

Back to the part that may have jumped out at you: “Number scores normally range between 200 and 300.” Those who score less than 200 or at the low-end of 200 will not only interpret the score as a dismal failure, but could then perceive themselves to be outside the normal range!  In other words, they could interpret a score of 210 as receiving 10% on a test, and more dismally, as not being normal. It’s not apparent where this statement about normal ranges came from. This is not a norm-referenced test.

The statement is particulary unjustified when one considers test developers’ own findings, based on initial piloting of the ESEE completed last year. The average score of 503 LBS learners who completed the reading section was only 207 (page 15 of this report). They had a pretty good idea that the average LBS learner would not score anywhere near 250.

There is one more part to the statement: “Most jobs require reading skills at 250 or higher.” Who the heck says so? One-third of learners in programs are employed, most participating in the ESEE are (based on some preliminary feedback I received) scoring in the low 200s, and this is exactly on par with previous results. So why the unsubstantaited and baseless 250 cut-off?

Place yourself in the learner’s chair, staring at the computer screen after receiving a score of 207 or so. You’ve just spent two to three hours or more doing a test for reasons that aren’t very clear to you, except you’ve been asked to do so. Then you see a statement that suggests you aren’t “normal” and you may not be capable of working. If this was you, what would you do at this point? Get up and leave, never to return to an LBS program? Cry in frustration?

Yet, there’s more.

If a student is not able to complete any of the six inital locator questions, which are written at a senior high school reading level, he or she will see the following statement:

You did not correctly answer enough questions to proceed to Part 2. Your organization will speak with you about next steps.

Seriously.

Again, put yourself in the place of the learner. You are asked to take the test soon after walking into a program. Most adults who decide to register in a program have already had negative experiences in the school system. They may have thought long and hard about returning to an education program. In addition, they may have recently attempted to register in an adult secondary credit program or a college program, only to be told that their skills aren’t adequate. The individual with past negative experiences, likely with a recent experience of rejection, then sees the above statement.

There are two parts to the statement that can crush people. First, students receive yet another failure message: you didn’t “correctly answer enough questions” so you can’t proceed.

Then they read that someone in authority “will need to speak with you about next steps.”  It sounds like a call to the principal’s office. The statement insinuates that some sort of transgression has occurred. It also suggests the learner could be told that he or she may not be able to register or participate in the LBS program, the program of last resort. Why else would one need to “speak to someone about next steps” if they are already registered?

How would you feel? What would you do? Do you add one more message of inadequacy to the ones already received? Or, hopefully, brush it off and conclude the test is utter nonsense and a waste of your time? But then what?  Will the program make you go through this again? How much of this experience and messaging stays with you?

Program educators and coordinators also receive unsettling messages that could lead them to question the eligibility of adult learners who cannot complete the six-item locator test (even though the texts are written a senior high school level). Perhaps they don’t belong in the program. Perhaps the ministry will not fund the program for registering students who can’t get past the  screening test. One can’t help but wonder if this is the ministry’s ultimate goal.