Adult basic education programs in Massachusetts may be faced with some difficult choices these days as they work to comply with the requirements of the new National Reporting System (NRS) and the state's SMARTT data management system. One of these choices involves the assessment and reporting of individual student progress in literacy and language learning. (Other choices revolve around different issues, such as the reporting of individual student goals, but this article will focus on the reporting of students' academic progress.)
As of July 1, the NRS now requires that each state and hence each federally-funded ABE/ESOL/GED program report the progress of its adult learners in measurable, quantifiable terms, using two "ladders" of six levels each, one built for ESOL and one for literacy/ABE/GED. In Massachusetts, the state Department of Education anticipated this requirement by building into its SMARTT data management system the requirement that programs report student progress for all students. On the ESOL side, this means assessing students in terms of the six Student Performance Level (SPL) levels adopted by the NRS for its own reporting. On the ABE side, DOE is requiring that programs report in terms of grade level equivalents (GLE's) from 1 to 12, which DOE will then translate into the six NRS levels for state reporting purposes. Programs are not required to use standardized tests to arrive at these SPL's or GLE's, but if they elect to use an alternative measure, they must correlate the results of this alternative measure with the SPL or GLE ladder and, eventually, provide proof of the validity and reliability of these correlations.
These student assessment requirements, as mandated by the NRS and implemented by SMARTT, can present programs with some difficult choices in deciding how to conduct their assessment process so as to meet two goals that are at least somewhat in conflict: 1) meeting these new reporting requirements; and 2) providing teachers and students with assessment information that is meaningful, accurate, and useful. This article will review the three basic options that it seems to me adult basic education programs now have regarding assessment.
The first of these options is for a program simply to use standardized tests for virtually all of its student assessment. The basic advantage of this approach, as everyone knows, is that it is rather easy to do--not an insignificant reason, though, I would argue, also not a sufficient one for choosing solely this approach, for it also carries a number of serious disadvantages. The first of these is that standardized tests simply do not appear to be very good ways of assessing the reading, writing, and math abilities of students, and especially of adult learners. The literature on this is vast and I won't go into the specifics here, other than to point to the many articles and books written by Susan Lytle, Marcie Wolfe, Marilyn Gillespie, Elsa Auerbach, Peter Johnston, and many others over the past two decades or more, criticizing these standardized methods of assessing learning and promoting various types of alternative assessment. (Local references would include, for example, the Fall 1988 issue of Focus on Basics, and the numerous issues of Adventures in Assessment published by SABES/World Education. The A.L.R.I. has many resources and lists of resources on alternative assessment, for those who are interested.)
A second disadvantage, which could at least partly derive from the first, is that standardized tests may in fact do a very poor job of capturing and reflecting the learning that does go on in adult basic education classes. For example, in a recent posting to the NLA (National Literacy Advocacy) electronic list, Thomas Sticht discusses a new study by Janet K. Sheehan-Holt and M. Cecil Smith, which finds little improvement in scores on the NALS (National Adult Literacy Survey) test on the part of adults participating in ABE classes. It may thus prove to be a major risk for adult basic education programs across the country and for the system as a whole to be judged largely on the basis of students' improvement in scores on tests that may be inherently incapable of capturing much of the learning that is taking place for these students at these programs.
The third disadvantage is that, despite the literally hundreds of tests that have been produced in this country, very few of these are developed specifically for use with adult learners, and there may be certain portions of our adult learner population for whom no test is even intended to be appropriate. For example, ESOL teachers have indicated that the BEST test, which is the test used almost universally for determining SPL levels with non-native-English speakers, was originally developed for use with certain refugee populations and is not necessarily appropriate for some other ESOL populations, especially students at higher levels.
A fourth disadvantage is that all assessments must be rendered in terms of either SPL's (for ESOL) or GLE's (for ABE). I can't really speak to the difficulty of doing this in ESOL (i.e., how well the SPL ladder works to reflect students' English language achievement). However, the use of GLE's to report ABE progress is certainly problematic (though it may be mechanically easy enough to do, if the test itself provides these supposed correlations). Quoting briefly from a few sources: • "Problems with grade level completion criteria for literacy statistics are well documented (e.g. Coles, 1976)." (Hannah Arlene Fingeret, Adult Literacy Education: Current and Future Directions, ERIC, 1984, p.8) • "Although the problems with grade levels as indicators of adult performance and progress are well-established, their use in the field of adult literacy is surprisingly pervasive." (Susan Lytle, Thomas Marmor, and Faith Penner, paper presented in 1986) • "Critics of the use of grade levels, however, point out that there is no valid translation indicating what real world literacy skills correspond to completion of a certain number of years in school." (Carolyn Chase Ehringhaus, in the Adult Education Quarterly, 1990, Vol. 40, No. 4, p. 189) • "Test results that give grade level scores or indicate that learners can identify specific skills on paper-and-pencil tasks yield very limited information. Despite the fact that our society in general seems quite impressed with measurable results that can be reported numerically, such data fail to match the overall goals. The assumption that numerical scores give evidence of confidence and competence is highly questionable." (Rena Soifer, et al, The Complete Theory to Practice Handbook of Adult Literacy, Teachers College Press, 1990, p. 171)
So, while using standardized tests as the sole means of assessment may be relatively easy, there are numerous other difficulties and risks associated with that route. A second possible assessment option for programs is to elect to use various means of alternative assessment and to translate the results of these assessments into GLE's and SPL's. The major advantage to this approach is a very important one: it would provide assessment information that creates a much fuller picture of a student's literacy abilities and that is likely to be more meaningful and much more useful to teachers and students alike. There are again, however, several likely disadvantages as well. The first is the time and energy it would take to create or adapt these methods of alternative assessment for use at a particular program with a particular population of students. It should be noted, though, that a great deal of work has already been done in this area (see, for example, the various Adventures in Assessment volumes) and more could be supported by targetted funding from the state Department of Education. Secondly, there will be the difficulty of proving to a sufficient degree the validity and reliability of these measures, though obviously the criteria set for achieving this level of proof will in large part determine how difficult this task will be for individual programs. Again, this difficulty could be mitigated through collaboration on the part of various programs and the support of DOE funding.
A third disadvantage is found in the requirement that these alternative assessments must be translated into SPL's or GLE's. Alternative assessment is not merely an alternative way of getting to the same place; it is also to some degree a different destination. Alternative assessment is based on a view of literacy and learning that doesn't see learning to read and write and do math as activities that can be laid out in a neat, sequential series of skills through which all learners progress from bottom to top. Alternative assessment approaches attempt to create a picture of a learning process that is by its very nature non-linear and that can vary tremendously from person to person. Having to translate, at least on the ABE side, alternative measures of assessment into GLE's certainly acts to negate the original intent and meaning and value of the alternative assessment process.
A third option which programs have is to combine elements of the first two (including their advantages and disadvantages) by using both standardized tests and alternative assessments. This hybrid option would combine the main advantages of the other two by using standardized test results to meet the requirements of the new reporting system in a relatively easy way, while using an alternative assessment approach to provide meaningful and useful information to teachers and students. The primary disadvantages this option would bring are that it would still require the time and effort on the part of programs to develop their alternative assessments and would still run the risk of not capturing for reporting purposes the actual learning that is going on in classes, as discussed above under option one. Nevertheless, this option may be the best of those available, even though it will require more work.
So, for Massachusetts adult basic education programs, these are the current assessment options as I see them. I urge practitioners and others to write to us and give us your thoughts on these issues. Tell us if you see other options available regarding assessment, let us know what choices your programs are making and how you will be implementing them, and tell us what SABES can do at this point to help practitioners and programs with regard to assessment and accountability. We will publish any responses we receive in an upcoming newsletter. And in the long run, we all as a field will need to "assess" how well the new approaches to assessment and accountability--the NRS and SMARTT systems--are capturing and reflecting the learning that students achieve as they attend our classes.
Steve Reuys is Staff Development Coordinator at the A.L.R.I./SABES Greater Boston Regional Support Center.