Synthesis of Discussion on Evidence-based Adult Education

This document is available as a pdf at http://www.alri.org/Rosen/ebae.pdf

Over several weeks, from late October through early December, 2003, an online discussion took place in a work group at http://communityzero.com/ebae . More than 200 adult literacy educators from across North America participated. The purpose of the work group was “to have a thoughtful conversation about establishing an Evidence-based Adult Education System”  in the United States. 94 messages, mostly from practitioners and researchers, were posted in response to three sets of questions. The entire transcript will be found, in three parts,  at http://www.alri.org/Rosen/ebae1.htm , http://www.alri.org/Rosen/ebae2.htm and http://www.alri.org/Rosen/ebae3.htm , where each post is identified by e-mail address and some posts are also identified by name. 

In the synthesis below, some posts are quoted (although not attributed) and some are paraphrased or summarized. All are identified by posting number. Not all posts were included in the synthesis.  Since the discussion was designed to be in response to a paper, Establishing an Evidence-based Adult Education System, by John Comings, I have indicated when John has stepped in to the discussion to add, clarify or respond.  I have tried to organize the responses by categories to help the reader.  It is possible, of course, that I have misunderstood some comments or placed them in categories with which the writers might not concur.  If so, I would welcome writers’ comments at djrosen@comcast.net

David J. Rosen

January 23, 2004

Part One Questions

How could the process for developing the system, set out on pages 10-15 (Building an Evidence-based Adult Education System, Baseline Program Models for Adult Education, Defining, Testing, and Using the Baseline Program Models, and Establishing an Evidence-based Education System), be improved?

How could the initial steps, set out on pages 17-18 (Next Steps), be improved?

You may wish to ask questions about, or refine, the steps suggested, or you may wish to suggest additional steps.

Part Two Questions

The second part of our discussion focused on:

The two questions were:

And these were further elaborated by me as follows:

“Perhaps we could look at it this way: what groups/kinds/types of students do you think we should study (first), groups with what kinds of instructional needs and goals? For example, should we study ESOL SPL level 1 and 2 students whose goal is "learn English"? Or whose goal is "get U.S. citizenship," or whose goals include those and possibly others such as "talk with a doctor or nurse on the telephone," or "get a driver's license" or.....? What are the pros and cons of studying groups of beginning level ESOL students? Suppose you had $10M for adult education research over 5 years, and it were up to you (anyone reading this) how to spend this? What group(s) with what needs and goals would you choose and why?” [Message 7, Part Two]

“What reasons/purposes/goals that students have for coming to programs would you like to see program models address? Which adult learner purposes would you like to see as starting points for program model research? For example, any of these below?

Program models designed to help:

What's missing from this list? With which student purposes/goals/reasons should we start first as we think about program models to study?” [Message 11, Part Two]

Part Three Questions

In this part of the discussion we hope to focus on how we would know if a model were effective, how to test its effectiveness.

Overview

“I must start by saying that I have serious concerns about the possibility and advisability of establishing an evidence-based adult education system. Yet, I understand the political necessity of doing so, and also want to think optimistically that doing so (well) actually would improve outcomes for learners. At this moment, in the current climate, we really can’t afford to say this is philosophically a bad idea and/or it can’t be done. While we may continue to advocate for changing the way things are, we better jump in and make things as good as we can given the current reality. If we don’t, someone else will and s/he won’t have a background in adult literacy education!” [Message 56, Part One]

Improving the Paper

Medical Model

Assessment Instruments

Need for Consensus on a Definition of Student Achievement

The paper’s careful, thoughtful process for building an evidence-based system, especially, “the call, repeated in several ways, for constructive inquiry, dialogue and consensus-building among the multiple stakeholders in such a system” is good, but needs to include building consensus around a working definition of adult achievement.  It especially should include discussion and critique of definitions of achievement based primarily on scores from off-the-shelf standardized tests and which are framed in terms of K-12 grade levels.  “Broadly-agreed-upon definition(s) of achievement would have significant impact on the ‘Next Steps’ put forth in the paper. For instance,

1) If we intend to purposely define groups of students according to a narrow range of needs/goals, are we content to limit our understanding of their needs/goals to test scores and grade levels?

2) Our definition(s) of achievement will, and must, have an impact on our choice of tools to validly/reliably measure that achievement. So at what point in this process do we get to have what would promise to be an edgy and exciting and very important conversation about appropriately defining and adequately measuring adult achievement? ” [Message 11, Part One]

Interface with the Equipped For the Future Initiative

“How do you see this system interfacing with what is being done by Equipped For the Future?” [Message 8, Part One]

Need for More Exploratory Research

We need more exploratory research and discussion up front to determine the appropriate outcomes for further investigation. [Message 12, Part One]

From John Comings:

            “This paper is a starting point, not a final plan, and so when we identified the groups, we constructed categories that were big and easy to describe. Once a system like this started, the number of groups could grow. Still, a lot could be learned within the large groups that would be applicable to most if not all, subsequent student groups.

This research would make the case that instructional groups in programs should be made up of students who fall into defined categories, rather than multi-level groups. This would require greater funding.

If members of the list think this approach will not work, can they suggest an alternative?

Could anyone suggest a different set of groups?

I think groups 1 (beginning ESOL students literate in their native language) and 3 (ASE students with higher level literacy and math skills) are good groups for the start of this line of research. These students come to our programs in significant numbers, we know a lot about how to serve them, and we have good outcome measures for them.

Which groups do members of the list feel might be best for the beginning of this line of research?”  [Message 13, Part One]

Defining Learning Outcomes

• “clearly defining the outcomes is a monumental task, even with the existing standards and principles noted in the paper.” The paper doesn’t include enough discussion around this issue. It is inappropriate to define groups of students to focus on using Student Performance Levels (SPL’s) and Grade Level Equivalents (GLE’s) to measure advances in student gains, but we also need to describe learning gains so that are understandable to the general public (presumably this is why we use K-12 definitions and try hard to align the content of what adults know/are learning at particular levels). [Message 14, Part One]

•  “It appears that the baseline model will be tested on the agreed upon outcomes. The example on page 6 talks about the impact of managed versus open enrollment (the baseline model) on learning gains (the outcome). Should the steps on Page 15 make specific mention about the process of developing those clearly defined outcomes? I see that the Next Steps on page 17 does make reference to agreement on the outcomes.” [Message 18, Part One]

Defining the Baseline Program Models

Defining the baseline program models will be the most difficult piece of the process, as proposed in the NCSALL paper. The concept of baseline models is still too fuzzy but is a necessary starting point for the entire process. [Message 15, Part One]

Selection of Research sites and Control of Variables

 • “…time of day, experience of teachers, curricula being used, number of classroom hours are all separate variables. Are there enough established research sites available to set up classrooms where the number of variables can be limited? Might it be necessary to recruit additional sites so that there are enough classrooms to provide the needed critical mass of data? If so, what might some criteria be to qualify?” [Message 17, Part One]

 • “In setting up the research sites, how will the number of variables be controlled? How many models can one study have?” [Message 17, Part One]

• “In the next steps the term "practitioners" is used. Does this term refer to practitioners from the research sites or practitioners in the field? We are concerned that research site practitioners may not reflect real world experiences. Often times their education exceeds that of average adult education instructors and their experience base may differ from the average instructor (e.g., better resources and a support system).” [Message 17, Part One]

• There is a potential disconnect between research sites and the ‘real world’. “For example, an adult education program generally operates within the policies established by the local, state, and federal government. Would being a research site change any of that i.e. ‘exempt’ them from such policies? ” [Message 18, Part One]

• “Can NCSALL provide some example criteria on how research sites could be chosen and how many would be needed?” [Message 18, Part One]

• “My primary concerns come from much observation of the inability of educators to control so many, if not most, of the variables that impact on the effectiveness of our practice, especially in adult programs conducted in a democratic, and competitive, society.” [Message 19, Part One]

From John Comings:

• “I also agree with Debbie that the constraints on research in education in general and in our field specifically are daunting, but that doesn't mean this research is impossible. We would probably need a minimum of 8 research sites. I'm not sure a random assignment experiment would be the best approach to evaluating a baseline model, but the methodology would have to be rigorous and the research sites would have to be replicable within reasonable budget levels.” [Message 23, Part One]

• There is a potential disconnect between researchers and practitioners.  “From a practitioner perspective, there is only one set of rules – the rules that allow us to receive continued funding for our learners. If you’re going to set up lab sites that live in some other reality, their findings become meaningless to us. To me, research to practice means we all get our hands dirty trying to figure out the really hard stuff….. it’s important not to be too ambitious with the scope of the initiative. However, let’s not take the easy way out, either.” [Message 53, Part One]

Dissemination of information so that Effective Models are Used in the Field

“I envision a web-based data system with the targeted program areas as the organizing principle. Based on the concepts in the occasional paper, the major content in each area would be program model descriptions (including video samples where illustrative) linked to the research reports on which they are based. The continuous improvement cycle part of the system would feature evidence from practitioners about using (evaluating) program models, discussion of the evidence, and modification to the models. I don't envision as much of a separation between ‘program model evaluation/testing’ and ‘practitioner knowledge to improve implementation’ as the diagram on p. 3 implies.” [Message 20, Part One]

Researcher/practitioner Paradigm

•“I share the concerns of others about the disconnect between researchers and practitioners and the suspicion that the separation of the two roles is a faulty paradigm. Ideally the researcher should be a classroom teacher, but there is an obvious need for a support system for the teacher-researcher.

In California there are federally funded statewide projects (CASAS and OTAN) that have some experience with action research in the areas of instructional technology and learner retention. Connecting with a national system that includes identifying research goals, selecting appropriate methodology, collecting and analyzing data, and developing and sharing reports would be welcome.” [Message 20, Part One]

•“….practitioners [in this model] …essentially are posed as receivers and then revisors of new knowledge generated through basic research. I would like to nominate teachers as posers of questions and initiators of research as well (not just reactors). Their findings could be ‘tested’ by other practitioners and researchers. We have to keep in mind that teachers often pose different kinds of questions from those that researchers might be interested in, and go about answering them in very different ways. They have access to information in very different ways than traditional researchers. I think the arrows on the diagram need to move back and forth!” [Message 56, Part One]

Alignment of What gets Taught, Learned, and Assessed

• “…an effective, evidence-based adult education system will result only from collaborative, good-faith efforts to align all components of the system -- what gets taught, what gets learned, and what gets assessed, along with all the ways we need to support quality instruction and assessment -- with what all the stakeholders agree are the most important goals for that system….how we define ‘achievement’ is at the very heart of the matter.”  [Message 22, Part One]

• “….What is important for adults to achieve through their participation in our programs? What do adults need/want to know and be able to do in order to meet their goals in their important adult roles, goals for ‘right-now’ as well as for lifelong learning?

In a well-aligned, evidence-based system, I think this ‘content’ question will lead naturally to the ‘assessment’ question: Once we agree on what is important to teach and learn, how will we appropriately measure learning of important content? (And, by the way, how will we support teachers in improving delivery of instruction focused on this important content?). Then we can look at currently available standardized tests, identify what content they do in fact measure, and decide how well that matches up with what content we think is important to measure. At that point we may be in a position to adopt currently available tests, and/or change them, and/or develop new ones, so that we end up with an assessment system that actually tells us what we want to know about adult achievement.

I don't believe our current system works this way. I don't think it can -- because we are not working with a consensus about what is important. And since we don't have that consensus, we can't really address the question of whether currently ‘approved’ standardized tests adequately measure what we think is important. We think that each of these tests may give us valid and important information about some aspects of adult learning – ‘parts’ of what is important, but we know that the tests are based on different sets of assumptions about what is important, as opposed to broad agreement on what's important, so they do not give us the same information. So my concern about defining adult achievement -- and groups of adults, for that matter -- based on the scores from currently available standardized assessments is a concern about misalignment between what we believe is important and what we measure. Given this misalignment, how can we possibly hold learners, teachers and programs ‘accountable’ for the results?” [Message 22, Part One]

Selection of Groups to be Studied

• “In one classroom, the range of student needs, academic levels, etc. is often quite broad, resulting in a less than ideal instructional delivery model. How would this impact the selection of 'groups'.” [Message 8, Part One]

From John Comings:

• “The EBAE paper suggests a way to move forward before that consensus is reached and the instruments are developed. That is, start with one or two populations that have goals and needs that are well defined and outcome measures that may not be perfect for accountability but are acceptable for research purposes….” [Message 23, Part One]

From John Comings:

• “So, are there subgroups of students for whom we already have a consensus on outcomes and good measures of those outcomes?”

NCSALL research indicates that a significant number of our students (maybe up to 30%) come to our programs seeking a GED or its equivalent and have reading skills sufficient to pass the test or meet the requirements of their ASE program. These students need help preparing for the GED or meeting the requirements of their ASE program, help making the transition to postsecondary education or training, and instruction that builds their "academic vocabulary" (the reading skills that are particular to the academic environment).

I believe we have the tools needed to identify these students, know how to serve them well, and have good outcome measures to judge program success. Why not start our research with this group? Are there other groups that are ready for model evaluation?” [Message 23, Part One]

• Include some ESL, the largest group of enrolled students (PY 01 data: ABE – 37%, Secondary – 21%, English Literacy – 42%)

• “focusing on adult high school diploma (groups) does complicate matters because of varying state and local standards for diploma acquisition.”  [Message 33, Part One]

•  “If we choose this group [GED students] to begin with, how sure can we be that what we learn about program development and evaluation design will be useful in future development of program models/evaluations where the student group is not so neatly defined by a credential? How well will our learning "transfer" as we begin to address the rich variety of learner goals and needs in our definitions of adult achievement? Or is it even important for what we learn from one model to inform development of others? Can we say the process "works" or "doesn't work", independent of/distinct from any particular student group, defined by any set of needs and goals, that we choose as its focus?” [Message 35, Part One]

• “If we go looking for another group, I'd suggest we have a lot of good work waiting for us for those adults who score below RGE 3. While the group is fairly definable by extreme low scores on existing standardized tests, several good research opportunities await (in no particular order):

1) instructional models to support emerging literacy

2) assessment measures to capture literacy growth (nothing we are using now is sensitive in this range)

3) professional development needs for instructors to teach the models

4) personal development/transformational goals and experiences of adults learning literacy

5) some psychoeducational testing for all to get a sense of the cognitive profile, how many of these "reading disabled" students are truly learning disabled and how their learning disablities profile/s are similar or different from what we know in the adults with LD research (I proposed this part as a grant to NICHD but was turned down, darn it, still seems like a good idea!)” [Message 39, Part One]

• In choosing groups, for example ESL groups, take into account: cultural, employment and geographical (urban, suburban, rural) demographics as well as performance on a test.  Regarding culture, learner attitudes toward education vary across cultures and must be taken into account. [Message 41, Part One]

• “I worry about starting with high functioning GED students for a couple of reasons. First and foremost, the question that begs to be asked of our learners is GED for what? At the local level, we spend a lot of time talking with learners about the GED not being the silver bullet of future success. Starting with GED students reinforces the GED as destination approach to our work. Secondly, as a practitioner, I would say that high-functioning GED students are the population I need the least help in serving. I’m certainly not saying there isn’t a great deal to be learned, but I am suggesting my need is much greater in other areas (such as ESL for the workplace). If research to practice is going to work, it has to be research practitioners want and need. Finally, I would like the research coming out of these lab sites to strengthen our position in the workforce investment system. Yes, ESL students seeking family-sustaining employment is a much harder group to work with than good readers who want a GED, but it makes much more sense, from my perspective.”  [Message 53, Part One]

• “Ideally, I'd start by identifying broad learner subgroups that practitioners deal with every day such as job seekers (at a local one-stop) or English language learners (at a cultural or community center) or 1st level learners (at a literacy council) or GED students (in a high school program). In my experience, there is often a correlation between setting and subgroup (which in and of itself suggests some interesting areas for investigation). This is the raw customer base that practitioners have to work with on a daily basis so it only makes sense to me that this is where implementation (and investigation) begins.

If, for example, you start with job seekers at a local one-stop center, it would be very interesting to look at learning gains, goal attainment, and customer satisfaction with learners grouped according to career clusters versus learners grouped according to academic levels. Or, if you start with a school-based GED program, it might be interesting to look at learning gains, goal attainment, and customer satisfaction with learners grouped according to variables such as academic level (the old standard); age (our 16-21 year old students talk about this often); comfort with, and access to, technology in the home that can be used for extended learning; interest in post-secondary education.”  [Message 6, Part Two]

• Start a group with goals in this order of priority:

learning English,

filling out government forms, applications to vote, bank statements, resumes etc.

work related skills, math skills etc.

participation in cultural or community events [Message 8, Part Two]

• “I think the better way to think about group type is not about the student per se, but about WHY the student comes to a program. It's the conversation with the student about what they need from us that helps me think about a structure ('model?') for them. It's not enough to know that a person "wants to improve their English" - it's not helpful, not specific enough, too broad” [Message 9, part Two]

• “I understand that we need some parameters with which to start collecting data. However, I would be hesitant to use the goals and/or needs that learners state as they come into programs as a parameter. Education is largely about learning what there is to learn and finding out what you are interested in and eligible for. A very big part of the learning that happens in ABLE is that people grow in their goals, their understanding of needs and rights, and their worldview. We talk about this type of learning regularly, as Andres did in his post, but it is very difficult to capture in quantitative research. I'd much rather see baseline data reflect things like:

a) literacy skills and habits (use V. Purcell-Gates's NCSALL work)

b) self perception/self efficacy

c) knowledge of and use of community resources for which a person is eligible (we could at least start with health resources, there is plenty of literacy/health research to draw from)

d) ways of knowing/ways of learning (Robert Kegan's NCSALL work)

These parameters obviously will take more time to elicit and establish than a checklist of goals, but those incoming goals are often way off base to what the true growth is in the educational endeavor. “ [Message 12, Part Two]

• “…we need to ask, ‘What do we know about how we are serving certain sub-groups of students?’ but we may not be able to get this information from NRS data.”…. states find it difficult, if not impossible, to disaggregate the math scores from the reading scores for each Educational Functioning Level. *And*, many programs don't even report reading scores for students--they either enter the lowest score only, usually math, or the score that will show growth the most easily (also math)….. Once we know which groups of students we are serving well/not so well, I would want to consider, ‘Do we want to focus on our strengths or our gaps, as a system…. a powerful alternative to the paper’s plan to develop baseline models would be to somehow identify programs/states that have consistently had success with a particular sub-group and do a grounded study of what aspects of these programs contribute to the high achievement. A baseline model would then be built around the findings.” [Message 13, Part Two]

• “We are working on a design for a literacy model for the Bureau of Prisons that will satisfy Congress’ mandate to provide inmates with literacy instruction leading to a GED and to help them prepare for re-entering society. I do not see these two fundamental purposes as comprehensive, and I see the need to accommodate learner-centered purposes as critical. Sara’s comment “after the goals of the sponsor are met, the specific needs of the population are integrated” was helpful.

We make grouping decisions about literacy learners in our system based on three broad criteria:

(1) Does the learner speak sufficient English to function in an English-based literacy class, or does he/she need to be placed in ESOL or the Spanish literacy program?

(2) How does he/she perform on component tests of literacy and language? (We plan on applying Strucker & Davidson’s Adult Reading Components findings to this project in the near future.)

(3) How much time does the inmate have left to serve, and which re-entry needs (e.g., housing, health literacy, finances, employment, transportation, family-related concerns, parenting support, survival reading, drug treatment...) are most pressing.

We probably will not group inmates based on their learner-centered goals. Rather, I see these deeply personal (and powerful) goals integrated into the program they way Sara described earlier.” [Message 16, Part Two]

•  “…isn't it part of Comings et al finding that program persistence is related to students' ability to work on their own goals? And Purcell-Gates' finding that increase in literacy practices is related to students' use of materials contextualized from their own lives? This points to clustering student sub-groups by goals they wish to achieve., with a greater likelihood that they will achieve these goals through persistence.” [Message 18, Part Two]

• “ Students By Grouping Within My Lab [all of whom come seeking a GED]:

Advanced Short Timers (Those student soon to get their GED) (and no, I do not run a prison based system although some of the teenage students might disagree)

Elderly

Post ESOL Students

Working students whose job mandates a GED

Learning Impaired Students

Teenagers

Court and Welfare Mandated Students

Pre-literate Students

The above groups are listed by most able to sustain attendance, as well as most able to participate with an active mind” [Message 10, Part Three]

Instructional Systems Design Approach to Designing Models

“….A way to proceed in the process of developing evidence-based models is, as Comings noted, to draw from other disciplines. One such discipline, concerned with and well grounded in research is the field of instructional design. Using traditional instructional systems design, learners are analyzed to determine characteristics such as prior knowledge and entry-level skills. Next, goals and objectives of instruction are specified. Subsequently, content and instructional strategies are developed to meet the specified goals. Finally, assessments are conducted to determine the level of performance achieved and the overall success of the instructional design process. Regardless of the pedagogical approach, learners must be informed of the content to be assessed. The cyclical nature of instructional systems design ensures that performance assessment is focused on educational goals, content, and teaching strategies in a continuous feedback loop. For this reason, content standards are an essential precursor to performance standards. How can we specify a level of performance if we haven’t specified what to measure? The principles of instructional systems design are violated when we start with performance measures and move backward. Once content is identified, appropriate performance measures can be developed. And many performance measures will need to be developed for the adult educational system. No "one size fits all" approach will suffice.” [Message 24, Part One]

Legitimacy of Evidence

“The underlying issue many seem to be raising is not so much the difficulties in establishing evidence-based criteria that provide good models for program improvement, but the legitimacy of the evidence drawn upon.” [Message 29, Part One]

Need for Systematic and Synthetic Literature Review

We need a review of the literature pointing to program quality. “A systematic and synthetic review of that body of work would represent one important baseline in identifying the core issues the field is grappling with, which then can be subject to further refinement. In addition, as practitioner-researchers, what would also be valuable is a field wide query through the lists and other venues on the three queries posed by page 12. What concerns me in the current document is the privileging of the researcher…

To cut to the chase:

[Message 29, part One]

Practitioners’ Limitations

“Any participatory research study should acknowledge the stake and expertise of practitioners, but also the limits of those practitioners' capacity to take on a big project.

Setting up the research partnership so it works on the operational/partner level is crucial. A couple of priorities I see are:

1. Finding funding to provide "release time" to participating practitioners to ensure they can be fully involved.

2. Taking an asset-based approach that builds on best practices already in place (not just those proposed in theory). Thus the purpose of the literature review would be to create benchmarks for literacy outcomes or correlates of those outcomes (average number of hours of instruction logged/student; average standardized test gains; certifications gained (GED, vocational test, driver's license, citizenship, etc...); self-reported learner/teacher satisfaction levels.

3. Inviting programs to self-nominate based on good outcome data or an investment in improving outcomes.

4. Involving researchers as outside consultants who

help practitioners structure research questions, gather and analyze data, and then provide the umbrella function of seeing and communicating the common themes emerging from the well-grounded, practitioner led study teams.

5. Envisioning the development of assessment tools that complement (rather than supplant) standardized assessments, . If we want federal dollars, we can't whine about all standardized testing being unfair - or expect that a system for evidence gathering become too idiosyncratic or time-consuming. A good example of such an assessment tool is the competency checklist. REEP in Arlington, Virginia conducted a small study in which they documented that the evaluations of teachers using checklists produced results confirmed by standardized tests. In so doing, they made a case for using a more nuanced form of student assessment that offered more to their students and teachers, while also passing muster in terms of reliability and validity.

I would also caution against too ambitious a scope for the initiative. With limited budgets, the return on investment when it comes to following up with students who have left the program seems relatively small.” [Message 30, Part One]

Need for Longitudinal Study

“I would love to have some researcher do a longitudinal study of families to determine what the value added for children is of a mother's participation in an adult ed program (provided that participation is of sufficient intensity/duration to be expected to matter).” [Message 30, Part One]

Limits of Evidence-based Research

“What I reacted to most was the (almost in passing) remark that the evidence-based model is "normal science" as defined by Kuhn, which, obviously, doesn't allow for different fundamental assumptions, "paradigm shifts", or for questions about whether fundamental principles of another area of educational research (K-12) would apply, say, to ESOL, or that principles of ESOL would apply to ABE. This is just a shot in the dark, but might address some of the questions of implementation that have been raised. “  [Message 32, Part One]

From John Comings:

RE: Discussion Part One: How to Improve the Process for Developing the System        

“I'm not sure that what we would learn from testing a model for ASE would inform models for other groups, but I suspect that it would. Even if it did not, we do need a model for ASE that not only helps people acquire a credential but also helps them build basic skills and knowledge and transition to postsecondary education and training.

This would be a research project, and so we would be able to use measurement tools that might not be appropriate or too costly for use in an accountability system. Multiple measures of outcomes and longitudinal studies of impact are possible. The model for orientation, instruction, support services, and transition would, however, have to work within the constraints of a reasonable per-student cost and of our student's lives.

One problem with discussions in our field is that we serve so many different kinds of students. Thinking about this specific group, ASE students whose reading test scores are at or above grade-equivalent 8, might make our discussion about the approach to building an evidenced based system easier. Later in the discussion we could talk about what that model might look like and how we would judge its effectiveness.

With this specific group in mind, do you have advice on how to improve the approach and the next steps described in the document.”  [Message 36, Part One]

Research Design

• How is it possible to apply a standardized method of assessment for research purposes without standardizing the student group in the process?  “The devil is in the details.” Amplification of this question is included in [Message 37, Part One]

• “Is there a web site that lists research studies that meet the criteria for evidence-based? Perhaps experimental and quasi-experimental models in adult literacy? I realize this could mean two things: 1) studies outside program models, and 2) studies of program models themselves.” [Message 58, Part One]

• Another Colleague replied:” There is a study, "Linking Research and Evaluation to Policy and Practice in Adult Education: Final Report done by ABT Associates Inc.”  [Message 59, Part One] [The reference for this study is: http://www.abtassoc.com/Page.cfm?PageID=7800&RS=1 ]

Importance of Student Follow-up

We need to follow up with students who have left (not necessarily those who have succeeded) to learn from them, to improve program design. [Message 40, Part One]

Evidence that Evidence-based approaches will be Useful, will Improve Programs, Positively Affect Learner Outcomes?

• “The basic assumption I would challenge is the notion that standardization is actually desirable (in the sense of baseline programs, for example). Why? Ironically, evidence would seem to suggest that evidence based instruction doesn't actually make that much difference to outcomes-- if anybody has evidence that it does, I'd love to see it.” [Message 42, Part One]

• Reply to [Message 42, Part One]

“I’d very much like to know what evidence is being cited here. Though I’m not a researcher, this perspective isn’t really consistent with my experience as a program director and practitioner. For the past seven years, 15 members of my staff and I have met monthly to look at research and program data and use it as a guide for improving our agency performance (enrollment, retention, pre/post rate, learning gains, GED attainment). It’s been very successful and our performance has improved dramatically in these areas. The bigger question we grapple with is….what do these outcomes really tell us? If 67% of our GED students now pass the GED test each year, can we really assume that these adults now have the knowledge and skills they need to succeed in their adult roles? As a practitioner I have no doubt whatsoever that an evidence-based system would improve outcomes. If there is agreement on the outcome, and practitioners implement research-based practices and watch their performance data, those outcomes will improve. The thing I worry about is who’s determining which outcomes and the impact that has on the evolution of practice. “ [Message 53, Part One]

Possible Principles for Baseline Models

“1) a program approach/philosophy which is articulated, understood and widely agreed upon within the program (not necessarily one that I would embrace, but one that is clear and that the program staff and students embrace;)

2) program stability (low or reasonable staff turnover; reasonably stable funding; adequate instructional materials) and

3) sufficient intensity of instruction such that a reasonable percent of students meet their goals and/or make learning gains.)” [Message 43, Part One]

Need to look to New Models, Models in Other Countries

“….if we are only studying what we are already doing, we will miss a whole spectrum of possibilities that we have not conceived of due to our constraints and assumptions. Part of our exploration of what a great program can be and do should be to look outside of our own federally and state funded system. What's going on in Canada? Australia? The UK? Other non-English speaking countries?” [Message 45, Part One]

Need a Different Sequence of Steps from those defined on pages 16-17

“I think there is some danger in starting on a course of research without, at least, having a more comprehensive research agenda in place first. I am concerned about the way preliminary research could be used, in this policy climate, to make quick decisions based on efficiency and narrowly-defined outcomes, without waiting for all the supplementary questions to be answered.

I believe this research agenda needs to be built on an agreed upon set of purposes for our system that go beyond individual achievement and beyond the social goals we've been handed by WIA, getting to some of the outcomes about self-efficacy and civic engagement that Andres and others have raised. Our research questions could then reflect the full range of purposes we're trying to achieve as a system. In my mind, this does not lead to the current sequence of steps as outlined on pages 16-17, but rather starts by identifying that range of purposes (increased health awareness, increased understanding of and voice in children's education, increased access to living wage employment, etc.) and then researching the program factors that effectively support this. And this might mean examining non-traditional models (such as programming organized less by "level" and more by goal areas) right from the start. “ [Message 46, Part One]

Need to Recognize Diversity of Field’s Approaches

We need to “recognize the importance and the limits of the numerous contextualized and decontextualized approaches to instruction and learning” in the field. [Message 49, Part One]

Program Models

• Examples of program models designed to address goals like increased health awareness, increased understanding of and a voice in children's education, and increased access to living wage employment.

• Madeline Hunter (lesson planning) model [Message 50, Part One]

• “….what would distinguish a program with a 'model' from one that does not have a 'model' is important.”  Although it isn’t clear what a model is/isn’t, some characteristics of a model are: specific content designed to meet students’ needs; specific structure of teaching/learning to reach specific goals; can be articulated in a package to those who inquire about it. [Message 60, Part One]

• We need baseline components before baseline models. [Message 60, Part One]

• Reply to [Message 60, part One]:

“Am I correct in understanding "baseline components" as a foundation of program features, for example: sufficient intensity of instruction, reasonable staffing stability, basic support services such as counseling and access to computers, and perhaps others?

And are you agreeing that before serious and costly research is undertaken, the program(s) where the research is done must have such foundation program features in place?

Are you then saying, in addition, that a program to be studied should have a clearly defined model (not yet defined) which shows promise, and is likely to be worth the effort to study it? [Message 61, Part One]

• Reply to [Message 61, Part One]:

“yes, what you describe is what I mean: appropriate staff, levels of instruction, appropriate materials, support services. And a further yes: I do not think it's wise to study just anything, I feel like we should identify first WHAT is worth studying. I think we should identify what the basic components of any successful program must have before we then devote precious time, energy, resources to examining it in-depth. Again, the issue of defining "baseline components" rears its head, but I feel that this area is much more manageable to develop than what program model might be worth studying. I think this is already happening on the list with folks discussing content-oriented programming - what are the pieces of the content-based programs that are common? That seem to be successful? Let's identify those pieces.” [Message 62, Part One]

• “In a posting on October 28th to the NIFL-AALPD electronic list, Heide Wrigley wrote ‘It would be great to find models where different service provider[s] in a community work together to lay out [a] service map of sorts and then offer some joint orientations that let students see where the best place for them might be. ‘

The adult learning context for Heide's comment was ESL/ESOL learning. For example, she refers to: "community colleges [which] offer low level ESL/ABE classes and then transition students to credit classes," "basic literacy," and "accelerated learning models that focus English for Special Purposes (certification)"

For a large urban area, where many different models can exist, an adult education "service map" (perhaps with counseling in the native language) might be a good step to help programs as well as potential students get clear about how the program models differ from each other with regard to: kinds of students served, levels, students' goals and purposes, intended program outcomes, and program philosophy or approach. Several models within this system of services could be studied. There could be parallel growth between a system of comprehensive services and a system of evidence-based adult education research, with the goal of improving the system of services as well as the models within the system.” [Message 14, Part Two]

• “Heide, the farm worker retraining program is designed with the sponsoring organization's goal in mind, which is to provide the students with the necessary English and construction skills to be able to obtain an entry level job in the trade. Students attend classes eight hours a day, five days per week. Again, the design is centered on the requirements of the sponsor so that the students are able to receive minimum wage stipends for eight hours while attending school. The average farm worker in our classes earns $3,500 per year prior to the retraining and after completion of the 20 week training they are employed earning anywhere from $6.50 to $9.75 per hour. After the goals of the sponsor are met in the design, the specific needs of the population are integrated. The program is composed of three phases: Phase I) instruction is conducted in Spanish and is composed of study skills with some leadership modules, GED, and technology literacy; Phase II) instruction is conducted bilingually and is composed of 4-hours of construction and 4-hours of VESL, math (which includes document reading such as interpreting blue prints, measurements, and other job specific tasks), and technology literacy; Phase III) three months of on-the-job training with a construction company. The components of Phase I and III are modified according to the needs of the students and funding available.

Key Elements: 1) Committed partnerships with sponsoring organizations

2)         A thorough understanding of the goal of the program

by all of the stakeholders

3)         Coordination of very frequent meetings by the instructors in the construction and the VESL, Math and computers.

4)         The components of the program provide a combination of classroom instruction and hands-on training.

5)         The outcomes are a finished home in 20 weeks with the majority of the students employed.

What I have learned from the farm workers and workplace literacy programs is that professional development is essential. We pay instructors to attend PD and for developing materials. Instructional strategies can be learned to address a diverse population or a bilingual one and good follow-up on the implementation of the strategies in the classroom is another key element. The instructors in our area are "ambassadors of tact and knowledge" in the classroom and at the worksites. These ambassadors make or break an entire program. The bread and butter of customized programs are repeat business. Based on intuition and experience, instructors are the number one motivators and creators or destroyers of dreams for a fragile population. We also use consultants who are experts in leadership development. By raising the self-esteem of our students, the use of the language outside the classroom is more evident (this was reported by employers). I am thoroughly enjoying the PD discussions on-line, and I am just thrilled to have discovered this new world of knowledge. Thank you all for such a candid and informal approach to sharing.” [Message 15, Part Two]

• “In Rhode Island during the mid-90s, a state initiative called "Child Opportunity Zones" were created. Some referred to this initiative as the COZy system, others decided to refer to it as the "Community Opportunity Zone". At any rate, start-up grants were funded by the state - and there were a handful at first. From there, each COZy had to pursue its own funding streams. By the time I left in 2000, there were something like 24 COZys around the state. What were they? A "one-stop service center" for the neighborhood. The state's parent involvement program (don't recall the name) was very involved, and many neighborhood parents were the COZ Coordinators. They did needs assessments in the neighborhood, and then went out to find those services - all free pretty much. So my programs (volunteer ABE/ESOL programs) were very much a part of several COZys in the Providence area. Other providers typically included the local police and fire depts, VNA, local schools or colleges, community centers, sites that provided technology. So all sorts of services were gathered and provided around that neighborhood COZy. They met with varying degrees of success. But my experience working in that type of set-up was very productive and interesting - people were able to stay in their programs (no matter which program: ABE, health, tech, something else) longer since there were some set of comprehensive services provided for participants.”

World Education’s College Transitions program is a model worth looking at:   www.collegetransition.org. [Message 19, Part Two]

Should we wait for Consensus in the Field around Goals and Models before Moving Forward?

• Yes, in Juliet Merrifield’s Contested Ground, she argued that we must envision what we want to get out of literacy, and build a consensus.  We need this consensus before we can move forward. [Message 52, part One]

• No, “it doesn’t make sense to wait until we have consensus on outcomes and measures before moving forward toward an evidence-based system.”

[Message 53, Part One]

• Yes and no.

“Regarding the matter of consensus, like many others, I feel leery of embarking on this process without a consensus of purpose and without good measures of achievement once we agree on what we are trying to achieve. However, when I thought about it a little more, I realized that the whole notion of consensus is pretty problematic. How in the world could consensus be reached? I think that this should not be the goal. Any kind of a system that requires consensus on such a huge and socially constructed notion as the purposes of literacy education is fundamentally flawed. And yet, I wouldn’t want the system to move forward without defining some purposes…..EFF standards are a good stand in for now.” [Message 56, Part One]

Baseline Models Concerns and Questions

• “I have great difficulty with the whole idea of baseline program models. In its most simplistic definition, I get the concept, but when I start to think about the details it becomes a problematic notion. Given the vast range of programs, how many models should there be? If the models are made too general to decrease the number, they aren’t very useful. If they are tailored to the multitude of program contexts, formats, purposes and goals they probably become too numerous to be useful. Furthermore, it seems that by the time baseline program models have been established, using the process outlined in the white paper, we would be well down the line toward identifying evidenced based practice. This isn’t a bad thing, but it makes me wonder if the process outlined isn’t too redundant and overlapping. Is there a way simultaneously to begin establishing an evidence-based system while identifying program models? Can we be maximally efficient here? Just to complicate things further, I feel that baseline program elements number 2 is hugely broad and would have to be broken up into smaller elements to be useful.” [Message 56, Part One]

• “…I would not use the goals learners state as parameters for determining baseline components or program models either. All I'm saying is that I think it's not the person we need to look at, but why they come…” [Message 17, Part Two]

How to Know if a Model is Effective

• “We look at three areas [in our [program]; number of GED passed or in progress; number of accommodations for folks with disabilities and retention. Since we have been doing this, our program has become more focused and more people are graduating overall.

Our ultimate measure is whether people pass the GED after being in our program. Since we really began monitoring our GEDs, our numbers have risen tremendously. The rise in GEDs is our measure, but you have to be careful. Adult education is not like K-12 where we have everyone start at 5 years old and end at 18. I think you need to look at what we can do to increase the number over the long haul.

Second, since we work with learning disabilities here in San Diego, we have been maintaining about 15% of our GEDs taken with accommodations. This is measuring that we are serving those in our population who need these services.

Third, retention. Students come if the program is working. They vote with their feet. “ [Message 2, Part Three]

From John Comings:

• “A specific example might help focus the discussion. One subpopulation might be ASE/GED students, whose reading skills are sufficient to pass the requirements of their ASE program or the GED and whose goal is a job with good pay and benefits.

Research suggests this goal can only be met if students are encouraged to build their skills and knowledge so that they can attain a higher score (GED score predicts income) and helped to transition into postsecondary education or training programs (adults with two years of postsecondary education or training are more likely to attain good pay and benefits).

A program model that would help students in this subgroup reach their goals might:

Intake and Orientation: help students understand the steps necessary to reach their goal and work with them to develop a learning plan that fits with the demands of their lives. That learning plan would outline the instructional and support services they need to be successful.

Instruction: Most ASE/GED classes are a combination of individual study, small group work, and whole class instruction. This approach developed as a response to mixed enrollment and open entry/open exit classes. With well-defined curriculum goals (the ASE requirements or the GED test) and an acceptance of the needs of this subpopulation that lead to mixed enrollment and open entry/open exit classes, development of an effective model of instruction that draws on the experience of good teachers should not be difficult. Adding a self-study component that utilizes the recent development of technology applications should not be difficult either.

Since these students will have post-secondary education as part of their goal, this curriculum would have to be augmented with content (such as academic vocabulary) that would provide the skills and knowledge needed to be successful in postsecondary education and training.

Support Services: ASE/GED programs usually cannot provide direct support services (such as day care) but they can provide counseling and assistance to students so that they can arrange those services for themselves or find ways around these life barriers to participation. Programs can help students develop the emotional support they need by establishing a program climate and procedures that build support among students and between staff and students.

Transition: Programs that help ASE/GED students make the transition to postsecondary education do focus on skills and knowledge, but they also focus on establishing a support structure for students once they enter postsecondary education. The building of this support structure begins in the ASE/GED program but should continue after the transition.

Cost: A program like this might cost $1500 to $2500 per student.

Evaluation: An evaluation of a program like this would have to include a longitudinal component that followed students for 3-5 years. A random assignment experiment or a treatment and comparison group study could look at this model vs the existing model that costs quite a bit less. I believe the evaluation would show that this more expensive model is cost-effective.

With this data in hand, our field could argue for policies that support this kind of program for this population. If someone comes up with a good idea that might improve this model (a new math curriculum for example) or make it less expensive (a new technology tool for example), it could be tested against this accepted model.

Other subpopulations (adults with very low literacy skills or those seeking education to help their children in school for example) may be more difficult to fit into this approach, but starting with the easiest subpopulations might make model evaluation for these subpopulations easier.

Do you think this would work? How might you do it differently? [Message 6, Part Three]

• [Reply to Message 6: Part Three]

“….all you propose here makes sense, but the funds aren't in place to implement and evaluate programs in the way you suggest, or the support for professional development and the respect for the fact that teachers do learn and want to learn is absent in many adult learning programs….. yes, you've proposed a fine model; yes, programs likely incorporate many of the elements you describe - if not all of the pieces you propose, but no, we can't seem to bridge the gap between the kinds of evaluation and analysis that Big Research seems to demand of us, while ongoing small and participatory measures tell us that people are learning.”

[Message 7, Part Three]

From John Comings:

• [Reply to Message 7: Part Three]

“Thanks. Yes this is what good programs look like and I'm suggesting that we do the "big evaluation." Once that evaluation is done and identifies impact, which I'm confident it would, then the field would have the evidence it needs to justify funding for this subgroup and accountablity could move from measuring outcomes to measuring whether or not programs conform to this model of good practice. Future research for this subgroup would take place within programs that conform to this model.” [Message 8, Part Three]

Cautions in Measuring Effectiveness

A danger to avoid in testing effectiveness of models: “as mandated programs need to increase positive outcomes they would tend to focus on those students whose success is most likely”  [Message 9, Part Three]

Other Good Features of Program Models

standardized questionnaires/checklists to determine not just goals, but learning needs (e.g. EFF)

Intake that incorporates interviews

emergent curriculum and modification of existing curriculum based on these findings

Implement the Teaching/Learning cycle keeping in mind that skills content is imperative

Documenting success involves:

matching student goals with outcomes

documenting which level learners have the most increase in both reading and math;

measuring attendance:

measuring frequency and total hours, and

measuring attrition [Message 9, Part Three]