In February, 2001, a task force--formed to address performance and accountability in Adult Basic Education in Massachusetts--met for the first time. This task force, called the Performance Accountability Work Group (or PAWG), was convened by the Massachusetts DOE/ACLS in response to the federal Workforce Investment Act (WIA) requirement that all states develop an "ABE performance accountability system." (A list of the members of the PAWG can be found at the end of this article, along with their e-mail addresses, so you can contact them with your thoughts and suggestions.)
In order to meet the federal requirements, other states have adopted standardized tests such as the TABE, the BEST, or CASAS, or they are attempting to develop and implement a more inclusive approach to learner assessment, one that reflects more than the results of a paper and pencil test. Examples of these efforts include states such as Ohio that are in the process of standardizing a portfolio approach to assessment, or states that are adopting Equipped for the Future (EFF) as their ABE approach, such as Maine. In Massachusetts ACLS successfully lobbied the USDOE for more time to develop an accountability system that more accurately reflects the successes of our adult learners and that supports the continuous improvement of its programs.
For nearly a decade, practitioners around the state have contributed to the development of the Massachusetts ABE Curriculum Frameworks and to the development of assessment and evaluation procedures that attempt to align with the Frameworks. These and other "home grown" or program-developed assessment materials, as well as commercially-available tests selected by programs, form the basis for our accountability system at the present time. Scores from pre-, mid-, and post-tests are entered into the state's data collection mechanism, SMARTT. For programs using assessment materials that don't result in numeric scores, they must correlate what they are doing to assess learners with requirements outlined through the SMARTT system--the "crosswalks" that programs were required to submit to ACLS this year.
One of the first priorities of the work group was to organize for the work ahead. The group agreed to have rotating facilitation responsibilities for each of the eighteen meetings planned. The self-selected facilitators form a planning committee to develop the agenda and activities for the following meeting. Smaller learning teams are then often formed to tackle the work. Also as a first step, we adopted a roadmap as a guide for our work through June, 2002. The first six months of the roadmap help us identify the range of underlying questions and frame the issues that we face in developing Massachusetts' ABE performance accountability system. Our roadmap looks like this: February 2001: Launch task force. Adopt roadmap. Launch learning groups. March thru May: Learning group research and reporting. June: What is a continuous improvement system? July: In-depth review of possible assessment measures and tools. August: Review GED/ADP/EDP measures under education reform. September thru December: Review of measures/tools continues. Closure of review, move toward selecting options. January thru June 2002: Determine levels within our accountability system. Develop assessment/evaluation procedures for system improvement.
Four learning groups were formed to study areas identified as fundamental in providing a solid base from which to start. The areas are: ESOL Assessment; ABE Assessment; Continuous Improvement Systems; and Countable Outcomes. In ESOL and ABE Assessment, the groups are identifying the range of options available for assessing educational gains by adult students. So, for example, these two groups will be reviewing both commercial and alternative forms of learner assessment presently in use in adult education. The Continuous Improvement group is examining what it means to become such a system. Continuous improvement systems are akin to "learning organizations" in the business world. As defined in the July/August 1993 issue of the Harvard Business Review, "A learning organization is an organization skilled at creating, acquiring, and transferring knowledge (i.e., skills and attitudes) to reflect new knowledge and insights." One member of this group says that continuous improvement means using an inclusive process to plan, implement, evaluate, and modify systemic or programmatic practices or structures in an environment that embraces risk taking and the identification of problems as critical to organizational growth. The Countable Outcomes group is defining what "countable outcomes" are (meaning primarily outcomes that are not educational gains, such as getting a job or becoming a registered voter), and looking at other systems and structures that also identify countable outcomes. The learning groups have begun their research and, as noted in the roadmap, will be researching their subjects though May, with progress reports given periodically.
The PAWG's work has started to unfold as a series of questions that we pose to ourselves and try to answer. With the first question, "To whom are we accountable?", we grappled with identifying the many stakeholders in ABE and the different ways to prioritize these stakeholders. Our task in small groups was to develop a hierarchy of customers so that we can eventually find the answer to the central question, "What are we responsible for?"
But before that, our next task was to try and answer the question, "What's good about what we do?" We brainstormed an extensive list that included items such as: assist learners in job improvement; help students believe they can accomplish things; help adults to understand their learning (metacognition); help adults reduce their dependence on others; and help learners increase literacy skills, just to name a few of the items on the list. In small groups, we then had to determine which of the items a program can cause versus which items are simply associated with programming, along with which items can be counted versus which items can be assessed. Again, all of this preparation allows us to narrow our focus and identify the specific elements that help us to answer the central question, "What are we responsible for as a system?"
With that in mind, we moved to answer the question, "What can be measured at the level of the program (P), what can/should be measured externally (E), what cannot be measured at all (I for impossible)?" and we assigned each item a P, E, or I. At this point in the work, we have come to the question, "What should (ideally) we be responsible for as a system?"
In my work as an assessment specialist, I note that programs around the state are asking themselves these same questions in their efforts to build systems that are useful and true both for themselves and for their adult students. It appears that the natural tendency, for individual programs as well as for the PAWG, is to start by asking the kinds of questions that uncover the values and assumptions that shape the focus and quality of a particular program or system.
Early on, the group agreed that it is necessary to reach agreement on the goals and inherent values embedded in the development of an accountability system, hence our "answer-some-fundamental-questions-first" approach. But time is of the essence, and the group also feels the urgency to address the specific technical questions programs are asking in light of the federal and state demands placed upon them to assess learners. This tension is acknowledged and revisited often. For now, the group is doing its best to work on both of these crucial pieces in tandem, with the more philosophical, values-laden questions being addressed by the whole group and the more technical, specific measures/tools questions being addressed by the small learning groups.
In addition, questions are coming from the field at large in regard to the work of the PAWG and what program responses should be vis-a-vis this work. For example, some programs want to know whether they should continue developing assessment materials as they presently are doing, whether they should stop and wait for more direction from PAWG, or if they should do something in-between. Given the fact that our work is being defined as we go and is in the developmental stages, it is challenging to address these questions from the field. The PAWG is in an interim period and does not have concrete answers yet itself; it would therefore be nearly impossible for the PAWG to advise the field at this point. Once more progress has been made, forums for discussing procedures and directly entertaining questions will be put in place. For now, one thing we should keep in mind is that our assessment work needs to align with the Curriculum Frameworks.
From my personal experience as a participant in this process, I would say that if a program has not begun to develop its own materials to meet accountability requirements, then they should not start at this time. These programs should continue as they are now, using the tools they have previously selected, and writing up and submitting their crosswalks. For programs that are already developing materials, I would say they should continue to a place in the work where they feel they can pause, until the PAWG makes progress and can give more direction to the field. Note that these bits of advice are my personal beliefs, not advice that comes from the PAWG or any of the other participants in that group. But as a former program director, and now a roaming assessment specialist, I would particularly encourage program development work to continue on some level because I see the enormous benefits practitioners are enjoying by participating in these work groups all over the state. I see this effort as only a positive thing, regardless of the end-product for our accountability system. Practitioners are engaging in research, dialogue, and strategic planning with one another. Programs are asking themselves the hardest questions and making every effort to uncover answers that will make their work more productive. Staff and teachers are their own worst critics (a good and a bad thing!!) which compels them to examine their practices and break out into new realms or confirm exemplary practice. I have been overwhelmed by the impressive dedication and willingness on the part of practitioners all over the state to engage in these mini-learning organizations put together by their programs to meet the WIA challenge. In my opinion, if Massachusetts ABE practitioners pursue their research and learning in this way, we will all be better prepared for what's to come in July, 2002.
Members:
Shelley Bourgeois, Director of Education, Jackson Mann Community Center, Boston, <shelbourgeois@yahoo.com>
Stacy Evans, Coordinator, Project Link, Berkshire Community College, <sevans@cc.berkshire.org>
Marie Hassett, Consultant, <bricolage1@earthlink.net>
Chris Hebert, Director/Counselor, Quinsigamond Community College, <Chrish@qcc.mass.edu>
Marcia Hohn, Director, Northeast SABES, <mdrewhohn@aol.com>
Andy Nash, Staff Development Specialist, NELRC, <anash@worlded.org>
Mina Reddy, Director, Community Learning Center, Cambridge, <mreddy@ci.cambridge.ma.us>
Tim Sappington, Director, N. Central Mass Regional Employment Board, <tsappington@net1plus.com>
June Sekera, Agency VP, Commonwealth Corporation, <jsekera@commcorp.org>
Anne Serino, Director, Operation Bootstrap, Lynn, <aserino@mindspring.com>
Judy Titzel, Program Developer, World Education, jtitzel@worlded.org>
Bill Toller, Assistant Superintendent, Hampden County Sheriff's Dept., <BILL.TOLLER@GTE.NET>
Staff:
Bob Bickerton, Director, ACLS, <rbickerton@doe.mass.edu>
Marie Cora, Staff Development Specialist, SABES, <mcora@worlded.org>
Robert Foreman, Program Coordinator, ACLS, <rforeman@doe.mass.edu>
Judi Pregot, Team Leader, ACLS, <jpregot@doe.mass.edu>
The Four Learning Groups:
ESOL Assessment: Mina Reddy, Chris Hebert, Andy Nash, Robert Foreman
ABE Assessment: Shelley Bourgeois, Stacy Evans, Judy Titzel
Continuous Improvement Systems: Anne Serino, Judi Pregot, Marie Hassett, Tim Sappington, Bill Toller
Countable Outcomes: Marcia Hohn, June Sekera, Marie Cora, Bob Bickerton
Marie Cora is a staff development specialist for SABES/World Education and editor of Adventures in Assessment.