Home Page of the Digital Age 2 Course

 

Course Outline

Bibliography

Reading Assignments

Homework Assignments

Assessment

Announcements & Updates

Digital Age 1 Course

Dave Moursund's Websites

Oregon Technology in Education Council

School Reform

The Northwest Regional Educational Laboratory (located in Portland, Oregon) has a nice Website of information about school reform. Northwest Regional Educational Laboratory 101 SW Main, Suite 500, Portland, OR 97204 Telephone (503) 275-9500 .

Information given on this page is quoted from the Website with URL: http://www.nwrel.org/scpd/natspec/catalog/index.html. The entire document is more than 200 pages in length.

Catalog of School Reform Models

Appendix C: Continuum of Evidence of Effectiveness

Using the Evidence of Effectiveness Table
Example 1

Example 2

Example 3

Top of Page

Catalog of School Reform Models

The Catalog of School Reform Models: First Edition was developed to support schools, school districts, states, and others as they proceed with their work under the Obey-Porter Comprehensive School Reform Demonstration program (CSRD) passed by the U.S. Congress in 1997. It provides information on 26 entire-school, or whole-school, reform models and 18 skill- and content-based models. An Addendum comprising an additional 4 models (1 whole school model and 3 skill- and content-based models) also has been published.

The catalog was developed by the Northwest Regional Educational Laboratory (NWREL) with assistance from the Education Commission of the States. It presents information as prescribed by the U. S. Department of Education, Office of Educational Research and Improvement (OERI), the funding agency. The addendum was developed by NWREL with assistance from representatives of the other nine regional labs, who helped (a) structure the review process for models, (b) develop criteria for inclusion, (c) conduct reviews, and (d) write descriptions of included models.

 

Table of Contents
Catalog of School Reform Models
First Edition

Acknowledgments

Preface

GENERAL INTRODUCTION

ENTIRE-SCHOOL REFORM MODELS

Introduction

Models

  • Accelerated Schools Project (K-8)
  • America's Choice School Design (K-12)
  • ATLAS Communities (preK-12)
  • Audrey Cohen College: Purpose Centered Education (K-12)
  • Child Development Project (K-6)
  • Coalition of Essential Schools (formerly 9-12, now K-12)
  • Community for Learning (K-12)
  • Community Learning Centers (PreK-Adult)
  • Co-NECT Schools (K-12)
  • Core Knowledge (K-8)
  • Different Ways of Knowing (K-7)
  • Direct Instruction (K-6)
  • Edison Project (K-12)
  • Expeditionary Learning Outward Bound (K-12)
  • Foxfire Fund (K-12)
  • High Schools That Work (9-12)
  • High/Scope Primary Grades Approach to Education (K-3)
  • League of Professional Schools (K-12)
  • MicroSociety® (K-8)
  • Modern Red Schoolhouse (K-12)
  • Montessori (PreK-8)
  • Onward to Excellence (K-12)
  • Paideia (K-12)
  • Roots & Wings (PreK-6)
  • School Development Program (K-12)
  • Success for All (PreK-6)
  • Talent Development High School with Career Academies (9-12)
  • Urban Learning Centers (PreK-12)
  • Matrix of Components

SKILL- AND CONTENT-BASED REFORM MODELS

Introduction

Models

  • Reading/Language Arts Models
  • Breakthrough to Literacy (K-2)
  • Carbo Reading Styles Program (K-8)
  • Cooperative Integrated Reading and Composition (2-8)
  • First Steps (K-10)
  • Junior Great Books (K-12)
  • National Writing Project (K-16)
  • Reading Recovery (1)
  • Strategic Teaching and Reading Project (K-12)

Mathematics Models

  • Comprehensive School Mathematics Program (K-6)
  • Connected Mathematics Project (6-8)
  • Core Plus Mathematics Project/Contemporary Mathematics in Context (9-12)
  • Interactive Mathematics Program (9-12)
  • MATH Connections (9-12)
  • University of Chicago School Mathematics Project (K-12)

Science Models

  • Developmental Approaches in Science, Health and Technology (K-6)
  • Foundational Approaches in Science Teaching (Middle School)
  • GALAXY Classroom Science (K-5)

Other Models

  • Basic Skill Builders (K-6)
  • HOSTS (Help One Student To Succeed) (K&endash;12)
  • HOTS (Higher Order Thinking Skills) (4-8)
  • Lightspan (K-6)
  • The Responsive Classroom® (K-8)

APPENDICES

  1. List of References
  2. Components of Comprehensive School Reform Programs
  3. Continuum of Evidence of Effectiveness
  4. Regional Educational Laboratories
  5. Comprehensive Regional Assistance Centers
  6. Comments Form

    ===================================================

Top of Page

 

 

Appendix C: Continuum of Evidence of Effectiveness

 

Most Rigorous
Somewhat Rigorous
Marginal

Theory/Research Foundation

Does the model explain the theory behind its design, including references to the scientific literature, that elucidate why the model improves student achievement?

Does the model state the theory behind its design explaining how the model's components reinforce one another to improve student achievement?

Does the model explain the theory behind its design?

Evaluation-based Evidence of Effectiveness

Have student achievement gains been shown using experimental and control groups created through large-scale random assignment or carefully matched comparison groups?

Have student achievement gains been shown using between or within-school comparisons?

Have student achievement gains been shown for a single school?

 

Has the model produced educationally significant pre and post intervention student achievement gains as reliably measured using appropriate assessments?

Has the model produced student achievement gains relative to district means or other comparison groups using appropriate assessment instruments?

Has the model produced improvements on other indicators of student performance , e.g. student attendance, graduation rates, or student engagement?

 

Have the student achievement gains been sustained for three or more years?

Have the student achievement gains been sustained for one or two years?

Have other indicators of improved student performance been sustained for one or two years?

 

Have the student achievement gains been confirmed through independent, third-party evaluation?

Has the model been evaluated by a state, district, or school evaluation team?

Has the model been evaluated by its developers?

Implementation

Has the model been fully implemented in multiple sites for more than 3 years?

Has the model been fully implemented in the original site(s) for more than three years?

Has the model been fully implemented in the original pilot site(s) for a minimum of one school year?

 

Is documentation available that clearly specifies the model's implementation requirements and procedures, including staff development, curriculum, instructional methods, materials, assessments, and costs?

Is documentation available that attempts to describe the implementation requirements of the model including staff development, curriculum, instruction methods, materials, and assessments?

Is documentation available that provides a general description of the program's requirements?

 

Are the costs of full implementation clearly specified, including whether or not the costs of materials, staff development, additional personnel etc. are included in the program's purchase price?

Have the costs of full implementation been estimated, including whether or not the costs of materials, staff development, additional personnel, etc. are included in the program's purchase price?

Is documentation available that provides general information about the program's costs?

 

Has the model been implemented in schools with characteristics similar to the target school: same grade levels, similar size, similar poverty levels, similar student demographics such as racial, ethnic, and language minority composition?

Has the model been successfully implemented in at least one school with characteristics similar to the target school?

Is information on grade level, size, student demographics, poverty level, and racial, ethnic and language minority concentration available for the schools where the model has been implemented?

Replicability

Has the model been replicated successfully in a wide range of schools and districts, e.g. urban, rural, suburban?

Has the model been replicated in a number of schools or districts representing diverse settings?

Is full replication of the model being initiated in several schools?

 

Have the replication sites have been evaluated, demonstrating significant student achievement gains comparable to those achieved in the pilot site(s)?

Have some replication sites been evaluated, demonstrating positive gains in student achievement?

Are promising initial results available from the replication sites?

Top of Page

 

The following examples show how the evidence of effectiveness table might be used:

Example 1

A school is considering a model whose stated purpose is to facilitate the school's development of a common set of goals for the school. The model provides five teachers and the principal with coaching in the principles of whole school transformation. Each school using the model is put in touch with other schools using the model. To-date the summary of the research-base for the model suggests that a single school which has used the model for the past two years has shown improvement in math scores over the last year. There is, however, no systematic evaluation of the model currently underway or planned. The costs for the model are approximately $3,000 per participant, approximately $20,000 per school.

Using the table as a guide, based on the description provided, a State, LEA, or school would probably conclude that the evidence of effectiveness for the model is unacceptably weak and, therefore, not accept this model. No research basis or other justification is provided for the theory behind the model, only a very vague statement that school staff should work together to be effective. The evidence for the effectiveness of the implementation of the model is extremely sketchy. The description includes a statement that the model has been implemented in a number of schools but there is no analysis of what it would take to implement the model. Given that only a few teachers and the principal would be involved and the estimated costs, the model probably provides only a low level of involvement. The model provides no evidence that this level of implementation is sufficient to produce results. The only student achievement results presented are for a single school for a short period of time in one subject. There is no information on how achievement was measured nor is any evaluation planned. Given this level of evidence, the model would likely fall below the marginal standards of rigor that States, LEAs, and schools would want to consider for a research-based comprehensive model of school reform. Apart from the marginal evidence of effectiveness, the model also does not address all nine components listed in Question B-1.

Top of Page

 

Example 2

A school is considering a model that emphasizes a curriculum in reading and mathematics using specific instructional techniques to guide classroom teaching and learning activities. The model provides teachers with intensive, on-going staff development using professional facilitators trained by the model developer. In addition to providing staff development, the facilitators remain on site as the model is implemented to ensure that all components of the model are working together. The program has been fully implemented in approximately 300 schools in 37 districts in 9 states around the country. Student achievement is measured not only by commercial standardized tests but also by state assessment systems where appropriate. Local adaptations of the model are available for schools serving a predominately Spanish-speaking community. When compared to schools matched on socio-economic characteristics, schools using this model show reading and math scores approximately three-quarters of a standard deviation higher. These results are similar for both African-American and white students. The program has been evaluated by its developer in approximately 12 sites over two years.

The evidence for this model is much stronger than for Example 1. While this model provides some details along each of the four dimensions in the chart, the implementation evidence is quite general. Furthermore, the school proposing to implement this model would need to address, in a coherent manner, all nine components listed in Question B-1.

There are some additional questions that States and school might ask about this model: Could the developer describe what was provided in the way of instructional materials? How will teachers learn the principles of instruction? For which grades and which types of schools are the achievement gains demonstrated? Because the model has only been evaluated by the developer, States or school could ask if there are any plans for an independent, third-party evaluation. While it is likely that the developer could provide satisfactory answers to most of the questions, the process would help reveal the relative strengths and weaknesses of this particular model.

Top of Page

 

Example 3

An elementary school in need of improvement has been studying how it could improve the very low scores on State assessments scores of its students in reading, math and other core content areas. The school leadership, in consultation with staff, parents, local university representatives, and community groups, has carefully reviewed school performance data and assessed what needs to be improved across the entire school program, concluding that the school needs a comprehensive approach to reform all aspects of its operations and instructional program.

As part of this process, participants reviewed both individual academic curricular programs as well as comprehensive reform models that include both teaching and learning materials and guidance for school organization and management. The school and its partners looked specifically at evidence of effectiveness for both the individual programs and the comprehensive models. In addition, they studied the match between the programs and the State's rigorous content standards of excellence for academic subject matter, and sought out information on how well the programs had been implemented in similar districts. After considerable discussion among school administrators, teachers, parents, community members and outside experts, the school decided to develop its own comprehensive school reform model, which would include upgrading curriculum and instruction, teacher professional development, school organization, parental involvement activities, and testing.

The school's proposed model is based on the careful integration of distinct, research&endash;based curricular programs with strong track records of effectiveness. The goal of the integration is a coherent instructional package that would address State content and performance standards; be aligned with district and State assessment systems; include professional development that helps teachers master the curricular programs as well as integrate the parts into a unified instructional approach; and include an evaluation strategy so the school can learn what is working and change what is not. As a result of this process, the school has decided to work under a Title I schoolwide approach so that it can bring together a variety of Federal, State, and other resources to adopt the curricular programs necessary to reform its instructional program and then move into a long term implementation effort.

Using the table as a guide, the example makes clear that the school has looked at the evidence of effectiveness that supports its choice of discrete curricular programs (in that regard addressing issues in row two). However, row one of the table suggests that, in its application, the school should explain the theoretical or research foundation for the model it proposes. At this point the school has not made clear why it expects its comprehensive model, which combines multiple discrete curricular elements, to function effectively as a whole. Concerning implementation, it is unclear how the school has assessed what will be required to make the program work at the classroom level. Thus, answers to the questions in row three of the table would be useful in the school's CSRD application. Finally, the example indicates that the school sought information on the uses in other settings of the selected programs. This shows a sensitivity to the questions raised in row four of the table. In a CSRD application, the school should discuss these issues more fully. The school should also discuss more thoroughly how its model incorporates each of the nine components listed in Question B-1.

Top of Page