Over the past several months, a lively discussion has been taking place on the NLA (National Literacy Advocates) listserv regarding recent changes in adult basic education policy and the federal ABE system. Thomas G. Sticht, a nationally-known researcher and consultant in adult literacy education, has contributed to this exchange a number of research and opinion notes which we feel would be of interest to practitioners. So, with his permission and starting with this issue, we will be reprinting some of these pieces here in the newsletter.
Magicians accomplish fantastic feats of deception with misdirection, smoke and mirrors. In the disappearing elephant illusion, the magician places the elephant in a cage made of six-inch-wide bars separated by about five inches between them. To make the elephant disappear, a flash of light goes off to one side of the cage, smoke billows up, and while the audience has its attention misdirected from the elephant in the cage to the light and smoke, five inch mirrors slip between the bars of the cage. The mirrors reflect the walls of the stage, giving the illusion that the elephant has vanished. A fabulous feat of deception using misdirection, smoke and mirrors!
Too often we find that the attempts erstwhile reformers use to make adult literacy education problems disappear turn out to be, no doubt unintended, acts of deception accomplished by misdirecting attention from genuine problems by a flash that produces lots of smoke (but no actual fire) and substitutes the mirrors of illusion for genuine solutions to problems.
Today we find that the most serious, indeed elephantine problems of adult literacy education, that include perennial, obscene under-funding of adult literacy education; marginalization of educational opportunities for adult literacy students through an excessive reliance on charitable activities and part-time, generally overworked, and frequently underprepared teachers; and the failure of corporations, governments, and news media to promote participation in adult education commensurate with their numerous reports and incessant doomsday rhetoric blaming the public school system for its failures and denigrating the literacy skills of the adult workforce (Is anyone aware of the national adult literacy awareness campaign?)&emdash;that all these problems have vanished in the misdirection created by the flash, smoke and mirrors of the call for standards (of what adults should know and be able to do) and accountability enforced by the use of standardized tests or other procedures that produce the illusion of mathematical precision in the very imprecise endeavor of education.
But we shouldn't be deceived, the elephantine problems are still there, behind the smoke and mirrors of standards and accountability. It is time to turn the table on those who practice deception and hold corporations, governments, and news media accountable through the imposition of standards for improving the educational opportunities of under-served adults. We should fight for standards in which we: spend as much to educate an adult as we do to incarcerate one; provide teachers, equipment, materials and facilities for adult literacy students as good as those we provide for university students, managers, and professionals; and give as much attention and publicity to the education of adult literacy students as is given to the education of children in preschools, the K-12 and higher education systems, and corporate training for managers and professionals.
When the flash of light and the smoke of illusory reforms appear, we adult literacy educators need to avoid having our attention misdirected and stay focused on the elephantine problems behind the smoke and mirrors. These problems are still with us, waiting for the magic of belief, commitment, and social responsibility that will make them truly vanish in the new millennium. Abracadabra!
The Workforce Investment Act of 1998, Title II: The Adult Education and Family Literacy Act, requires "core indicators" of performance by federally funded literacy programs. The Act requires that levels of performance for each indicator be established, and the levels "…be expressed in an objective, quantifiable, and measureable form; and…show the progress of the eligible agency toward continuously improving in performance."
The metaphor of "levels" is pervasive in all education, including adult literacy education. We speak of children or adults who read "at the fourth, or fifth, or sixth, etc. grade levels," as though these "levels" are quantities of "reading" that stack up on one another reaching higher and higher plains. Most recently, the National and International Adult Literacy Surveys (NALS/IALS) have promoted the use of five "levels" of "literacy" for describing the literacy abilities of adults in the United States and eleven other nations on Prose, Document and Quantitative scales. In the scheme of the NALS/IALS, adults were assigned to a given level, and it was strongly implied that the person could not perform tasks above the assigned level.
However, the fact was that on the NALS Document scale a person who scored at the average for literacy level 1 could perform almost half the tasks at level 2, a quarter of the tasks at level 3, one out of five at level 4 and even one in six at the highest level, Level 5. Similar findings of adults being able to perform significant percentages of literacy tasks above their assigned level holds for all three literacy scales in the NALS/IALS. For given adults, then, they might not take well to the idea that their literacy was fixed at some static, lower "level." Rather, they might think of themselves as quite capable given that they can sometimes perform very difficult tasks well above their assigned level. Perhaps that is one of the reasons two-thirds of the adults at Level 1 of the NALS said they read well or very well, and why millions and millions have not come running for help to our adult literacy programs.
In trying to use the NALS/IALS levels to evaluate advancement in literacy, a program might actually increase people's skills within a level but not enough to move the person up to the next level (a level has about 50 scale points in it). So considerable horizontal (within a level) growth might not show up as any vertical growth (up to a new level). It is also conceivable that some adults might not improve at all in the numbers and "levels" of tasks that can be performed, but that they might become more efficient in performing the tasks they can already do, but too slowly. Such an increase in efficiency is also likely to escape notice using the "levels" metaphor as a basis for judging advancement.
Research by colleagues and myself some 25 years ago showed that adults with specialized knowledge in the areas of automobile and shop information who read at the "fifth grade level" on a standardized reading test could nonetheless perform on an automobile mechanics job-related reading task test as though they had reading skills one to two grades higher in general literacy (Sticht, T. G. (1975) Reading for working: a functional literacy anthology. Alexandria, VA: Human Resources Research Organization, pp. 43-45). Thus, while their "general" literacy "level" was fifth grade, their "job-related literacy" "level" was sixth or seventh grade.
The military services use for selection and job assignment purposes a ten-part test called the Armed Services Vocational Aptitude Battery (ASVAB) that provides both general verbal and mathematics knowledge measures, and several other knowledge tests such as automotive and shop information, general science, electricity, etc. These tests are used to produce profiles of knowledge and skills such that a person low in verbal ability might still be accepted for service if he or she has higher knowledge in one or more of the special knowledge domains. This is an example of the use of both "levels" and "profiles" in combination to characterize several different aspects of people's literacy (all tests in the ASVAB are written tests).
These sorts of data raise the question of just how literacy ability should be represented. Is it well represented as "levels," like an onion with a core and successive layers of growth out to some current "level?" Or perhaps as "levels" in geological strata? Or would it be more useful to think in terms of networks of specialized domains of knowledge interrelated by the use of common vocabulary words (and, but, the, over, etc.) and a limited set of syntactical rules for selecting and sequencing parts of words into new words (e.g., test, tested) or words into sentences. In this type of representation, growth of any amount in any direction in the knowledge network would count as improvement for accountability purposes. All knowledge that a person possesses or develops could accrue to the person's "accountability account" (perhaps in a portfolio).
Presently, the dominant metaphor seems to encourage the "banking" metaphor criticized by Paulo Freire. The adult's head is considered as a bucket and, using our dipsticks (tests) we determine that the head is filled, say, to the 3rd grade level. The educator's job then is to pour in more cognitive "fluid" to raise the level up to the 9th grade and eventually the GED level as measured by our cognitive dipsticks.
At the very least, it seems to me that some attention should be given to the pervasive use of the "levels" metaphor in attempts to characterize adults' literacy abilities. Much more attention to such fundamental matters should be given before anyone espouses the use of our present stock of "literacy dipsticks" to hold programs and their learners accountable for "continuous improvements" in learning.
...[E]stablishing...performance indicators of adult literacy education is, to say the least, problematic. A case study in the difficulty of developing quantifiable measures of adult literacy ability is given by the National Adult Literacy Survey (NALS) of 1992. The National Center for Education Statistics survey design report stated that the National Adult Literacy Survey would "describe the levels of literacy demonstrated by the total adult population as well as by adults comprising various subgroups, including those targeted as 'at risk.'"
As straightforward as this quote seems, it is actually misleading because, in actuality, there are no "literacy levels" to be "described" in the adult populations of nations. Instead, there are various ways of conceptualizing the nature of literacy and different procedures of measurement that can lead to the construction of alternative representations of adult literacy in society.
When discussed as a form of human cognitive ability, as both the NALS and International Adult Literacy Surveys (IALS) do, literacy is a psychological construct and therefore literacy cannot be directly "described." In fact, in the United States over the last 75 years, different representations of adult literacy have been constructed. For instance, the National Assessment of Educational Progress of 1970-71 assessed adult literacy using the same "academic" tasks that were used with school children, such as knowledge of word meanings (vocabulary), using visual aids, following written directions, using reference materials, locating significant facts, getting the main idea from materials, drawing inferences, and critical reading. At the time, there was no particular difficulty expressed by adult educators or anyone else in using these sorts of "school oriented" items to represent adult literacy.
The Young Adult Literacy Survey (YALS) of 1985, the NALS of 1993 and the IALS of 1995 are the most recent representations of adult literacy at national levels. These surveys represented adult literacy using "tasks that simulate the literacy demands that adults encounter in their daily lives." Similar "real world" tasks were used earlier in the "Survival Literacy" survey by Louis Harris Associates in 1970, the Adult Functional Reading Study of 1973, the Adult Performance Level Study of 1975, and the contemporary Comprehensive Adult Student Assessment System (CASAS).
The fact that assessments of adult literacy over the decades have constructed various representations of adult literacy ("academic," "real world") raises important questions. How should the literacy abilities of adults be represented? Are all representations equally valid? If so, how should we choose the one(s) among the many to use?
A recent U. S. General Accounting Office reported that one expert on adult literacy research stated that functional literacy tests made up of "real world" tasks like those of the CASAS, NALS, and IALS may lack validity because they are not derived from theoretical models of ability but from everyday literacy tasks. Because of the complex nature of such tasks, it is not clear what implications can be drawn from the test performance.
This problem was illustrated in a manual for item writing produced by CASAS. The manual notes that the use of complex, "real world" tasks as items "generally tests the use of two or more skills. Therefore, this context is not appropriate in itself for diagnosing weaknesses in specific skills since it is difficult to determine which skill was performed incorrectly."
In the absence of a clearly specified theory of "literacy" as a psychological construct (as in the NALS/IALS), it is not possible to know how to develop assessments that measure the component knowledge and skills considered to make-up the ability or abilities that constitute "literacy." Without knowing what specific knowledge or skills are being assessed in "real world" tasks, it is not clear to what extent test performance reflects literacy ability or some other abilities, such as problem solving, reasoning, language comprehension, vocabulary knowledge, management of test-taking anxiety, interpersonal skills, or some complex, interactive combination of all these or whatever.
This problem of interpretation of what is being measured may not be so important for those who perform well and can therefore be assumed to possess whatever knowledge and skill is called for in performing the "complex information processing" (literacy?) tasks. The problem becomes critical when the focus of concern is on understanding why it is that those who do not perform well do not perform well. What kinds of services should be provided to help them improve their ability to perform these kinds of tasks?
If government contracting agencies cannot inform adult literacy education programs with some precision about what it is they should be teaching based on the types of measurements the government develops or otherwise promotes to construct particular representations of adults' literacy abilities, should these same government agencies then turn around and use performance on such measures to give or withhold funding for programs that fail to teach and improve whatever it is that the tests measure? It seems to me that a socially oriented litigation organization could find the basis for a solid class-action case of "mal-literacy practice" here!
Reference: Citations to quotes used herein are given in: Sticht, T. (1999, April). Using Telephone and Mail Surveys as a Supplement or Alternative to Door-to-Door Surveys in the Assessment of Adult Literacy. Washington, DC: U. S. Department of Education, National Center for Education Statistics, Education Statistics Service Institute (ESSI).