|USFDC Home||| RSS|
This item is only available as the following downloads:
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20009999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00179
Educational policy analysis archives.
n Vol. 8, no. 35 (July 26, 2000).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c July 26, 2000
State standards, socio-fiscal context and opportunity to learn in New Jersey / William A. Firestone, Gregory Camilli, Michele Yurecko, Lora Monfils, [and] David Mayrowetz.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 8issue 35series Year mods:caption 20002000Month July7Day 2626mods:originInfo mods:dateIssued iso8601 2000-07-26
1 of 25 Education Policy Analysis Archives Volume 8 Number 35July 26, 2000ISSN 1068-2341 A peer-reviewed scholarly electronic journal Editor: Gene V Glass, College of Education Arizona State University Copyright 2000, the EDUCATION POLICY ANALYSIS ARCHIVES. Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education State Standards, Socio-fiscal Context and Opportunity to Learn in New Jersey William A. Firestone Gregory Camilli Michele Yurecko Lora Monfils David Mayrowetz Rutgers UniversityAbstractA survey of 245 New Jersey teachers provides a base line for examining how the introduction of state standards and assessm ents affects the teaching of math and science in the 4th grade. Thes e policies are promoting teaching of additional topics in both are as. The changes in the delivery of professional development have not yet b een sufficient to lead to substantial changes in instructional practice. W hile inequities in access to material that characterized the state in the early 1990s have diminished, we find a pattern of inquiry-oriented s cience teaching more prevalent in wealthy districts and teaching to the test more prevalent in poorer ones. We also note some areas where middle-i ncome districts appear disadvantaged.
2 of 25 A central goal of the standards movement ha s been to help all children learn challenging content (Smith & O'Day, 1991). Forty-fo ur states have now adopted standards for student proficiency in the core acade mic areas, 41 states have aligned assessment with their math standards, and 25 have a ligned assessment with their science standards (Quality Counts, 2000). While great atten tion is being paid to what students are learning, less scrutiny has been given to what they are taught. Yet, the former depends at least in part on the latter (Wiley & Yoo n, 1995). For that reason, state standards are intended to provide guidance on what should be taught, as well as what students should learn (Smith, Fuhrman & O'Day, 1994 ). The adoption of standards and assessments d oes not guarantee students access to instruction, especially for poor students. For that reason, people have begun to worry more about "opportunity to learn" (OTL) or "whether or not students have had an opportunity to study a particular topic or learn ho w to solve a particular type of problem presented by a test" (Husen as cited in McDonnell, 1995, p. 306). Advocates for minorities have seen the reporting of OTL standards as a way of ensuring that poor and minority students are not disadvantaged inappropria tely when standards are raised. As one observer noted, without OTL standards, "you don 't know if the school if failing, or if students are failing" when test scores are low (Rot hman, 1993, p. 21). Both the federal and state governments have been much more willing to adopt student performance standards than OTL standards si nce the latter specify the government's obligation to deliver services to stud ents (McDonnell, 1995). Moreover, the legal mandate for guaranteeing that OTL be prov ided is ambiguous, even though the issue arose in the early years of state testing. Ac cording to Millman and Green (1989, p. 356): The court decision in the Debra P. vs. Turlington (1981) case seems to have established the necessity that, at least for certif ication tests for high school graduation, the tested material must consist of con tent that is currently taught, that is, the student must have been provide d adequate preparation and, thus, had a fair opportunity to learn the mate rial. Precise requirements of a fair opportunity to learn remain ambiguous. Several decades of research have indicated how difficult it is to change teaching practice (McLaughlin, 1990; Cuban, 1993). Simply im posing standards by decree is not likely to modify teaching practice if teachers do n ot understand what is expected of them or have the resources to carry out a standards-base d program of instruction. The situation can be especially challenging in mathematics and th e sciences where elementary education teachers may lack the background knowledg e to effectively teach more challenging content. This article introduces a project designed to explore how state standards and related policies influence teaching practice. In May, 1996, New Jersey announced a new set of "core curriculum content standards" (NJSDE, 1996). These standards began to take practical reality for elementary school teachers wh en state assessments aligned with these standards were introduced in 1998. In the Spr ing of 1999, as the state administered its new fourth grade mathematics and science assess ments for the second time (the first time for which results would actually be released p ublicly), we began a three-year study to examine how teachers in those grades teach mathe matics and science. Using a state-wide representative survey, this article desc ribes three dimensions of teaching
3 of 25practice: the content taught, access to and use of materials, and teaching to the test. In each area, we investigate what in being taught and how equitably practices are distributed among wealthy and poor districts. We al so explore teachers' background knowledge and opportunities to learn about new prac tices. Our preliminary conclusions are that: The introduction of standards and assessments is br oadening the range of topics taught in mathematics and science. A useful baseline measure for assessing teaching to the test can be developed. Opportunities remain limited for elementary teacher s to learn the new knowledge required to improve their mathematics and science t eaching. The inequities between wealthy and poor districts a re complex and may be overstated, but there is clearly more teaching to t he test in poor, urban districts and more hands-on science teaching in wealthier distric ts. Before addressing these issues we describe the cont ext for standards implementation in New Jersey and the research methods employed in the study.The Policy Context In the last decade educational policy in Ne w Jersey has been driven by two related phenomena: school finance litigation and the develo pment of standards and related assessments. Whereas financial resources can influe nce the distribution of OTL, legal battles surrounding the school finance issue also m otivated the adoption of standards. School Finance Litigation Since school finance litigation began in Ne w Jersey thirty years ago, there have been two court cases, eleven decisions, numerous sc hool finance bills, and other laws and regulations (Goertz & Malik, 1999). The litigat ion and related legislation has focused on whether the state was obligated to provi de all children therein a "thorough and efficient education." While these actions have had a number of implications for education in New Jersey, two are especially critica l here: the definition of a thorough and efficient education, and the financial provisio ns to ensure that all children could receive one. The court has been reluctant to define a th orough and efficient education except in the broadest terms: For those special needs districts [the approximatel y 30 poor urban districts identified by the court as inequitably served by th e state], a thorough and efficient educationone that will enable their stud ents to function effectively in the same society with their richer p eers both as citizens and as competitors in the labor marketis an education tha t is the substantial equivalent of that afforded in the richer districts ( Abbott v. Burke 643 A.2d 575, 580 (1994) ) (Abbott III) Beyond stating that children in poor districts shou ld get the same education as those in wealthy districts, this decision provided very litt le guidance; and the court continued its multi-year effort to urge the state department of e ducation to specify criteria in more detail. This was accomplished in part in the Compre hensive Plan for Educational
4 of 25Improvement and Financing (CEIFA), the school fundi ng law of 1996, which defined a thorough education as one in which children succeed ed in meeting the 56 outcomes specified in the Core Curriculum Content Standards. Thus, the standards became the criteria for educational effectiveness, and state t ests administered in 4th, 8th, and 11th grade would operationalize those criteria. The cour t found that these standards and assessments were "the first real effort on the part of the legislative and executive branches to define and implement the educational op portunity required by the Constitution and are facially adequate as a reason able legislative definition of a thorough and efficient education" [ Abbott v. Burke 693A.2d 417, 428 (1997) (Abbott IV)]. This effort was not sufficient to clarify w hat constituted adequate educational funding for all children in the state. Thus, the co urt continued to use a two-part yardstick. First, the poorest districts in the stat e should spend essentially the same per capita as the wealthiest districts (Goertz & Malik, 1999). The state had developed a classification of districts (District Factor Group or DFG) based on a composite measure of community, social, and economic variables such a s the educational and occupational background of the population, per-capita income of the district, and mobility. The DFGs were designated by letter with the poorest district s labeled "A" and the wealthiest labeled "J". Per-pupil spending in the special needs distri cts designated by the court was expected to match that of the highest DFG districts As late as 1993-94, the 14% of districts were spending 22% more than the poorest a lthough their collective tax rate was 43% lower (Firestone, Goertz & Natriello, 1997). Second, in addition to equal base spending, the court required the state to support a series of supplemental programs for the poor urban, districts. Urban schools were expected to implement a whole school reform program model such as Success for All (Porter, 1999), extend early childhood education se rvices to 3and 4-year olds, and began programs to refurbish aging and decaying buil dings. Since these programs could not be supported locally, they had to be underwritt en by the state (Goertz & Malik,1999; Erlichson, Goertz, & Turnbull, 1999). By the 1999-2 000 school year, the equal base funding provisions were in place and implementation of the special programs had begun although not without disputes about the local level of funding and district discretion in designing their whole-school reform and early child hood programs. Equal basic funding is an important develop ment, and extremely unusual in a state noted for inequities in education. In 1996 only two states had a greater dollar gap in spending between the fifth and 95th percentile dist ricts than New Jersey (Quality Counts, 2000). However, the court remedies and new funding formula did not extend to all districts. Schools in DFGs as low as B and into the middle of the fiscal distribution were spending less per child than either the wealth iest or the poorest districts in the state. Standards and Assessments As a normative perspective, standards theor y recommends that state standards become the criteria with which assessments are alig ned. However, like many American states, New Jersey began with assessments rather th an standards. Its first testing system, begun in the late 1970s, was designed to measure "m inimum basic skills" as a means of maintaining the accountability of poor urban distri cts, who at that point were receiving a new infusion of state funds. Several revisions ensu ed, and by the early '90s the keystone of the state's testing system was the High School P roficiency Test (HSPT), administered in 11 th grade as a partial requirement for high sc hool graduation. This test covered mathematics, reading, and writing at a more challen ging level than the earliest test, but passing score was still set at a basic skills level The HSPT was accompanied by an
5 of 25Early Warning Test (EWT), given in 8th grade to hel p schools identify children at risk of failing the graduation test. These tests had specia l significance to educators because patterns of low scores on these tests could become grounds for state takeover of a district. Districts were also expected to administe r conventional achievement tests of their own choice at grades not tested by the state (Firestone et al., 1997). During the 1990s as the standards movement took hold nationally, teams of content experts and teachers were formed within the state t o write the core curriculum content standards in seven curricular areas as well as a se t of cross-content workplace readiness standards. These efforts were heavily influenced by national standards documents in mathematics and science and became official in May, 1996 (NJSDE, 1996). The resulting standards for mathematics and science are listed in Appendix A. These core standards are accompanied by cumulative progress in dicators for grades 4, 8, and 12. Separate documents provide curriculum frameworks to offer guidance to educators in implementing the standards. The state is now phasing in 4th, 8th, and 1 1th grade tests that are intended to be aligned with the standards in each area. The degree of alignment to the standards is difficult to assess becauseas in many statesstric t confidentiality is maintained over operational test items. This creates difficulties f or educators who wish to be given test results item by item in order to seek an easier met hod for aligning their instruction more closely with the assessments. The current tests are an effort to move away from t he basic skills or advanced basic skills orientation that characterized earlier state tests. The 4th grade mathematics tests include 32 closed-ended and five open-ended items; and the matrix for selecting items includes a dimension of "problem-solving skills" wi th categories like "procedural knowledge, conceptual understanding, and problem-so lving skills" (NJSDE,1998, p. 6). The 4th grade science test is similarly organized. One sample openended item and one sample closed-ended item from the test specificatio ns are included in Appendix A. The 4th grade mathematics and science tests were first administered in the spring of 1998, but because of technical problems scores were not r eleased. The following year scores were released in the fall after the spring 1999 adm inistration. The introduction of new standards and asses sments in mathematics and science should provide clarity regarding what is expected t o be taught in each area, and ensure that these subjects receive consistent attention. W hether this attention takes the form of short-term "teaching to the test" or deeper changes in practice, and whether access to new forms of instruction is equally distributed in the state remains to be seen. Recent court and legislative actions may further stimulate access to new forms of instruction. We turn now to the survey designed to address these issues.Study Sample In the spring of 1999, we initiated a three -year study to examine teachers' response to the new testing program in the areas of mathemat ics and science. Data were collected from a statewide sample of 4th grade teachers. Just over 600 teachers were asked to respond to a complex set of instruments. After exte nsive telephone follow-ups and remailings, 245 teachers completed a telephone surv ey, 172 completed an additional mailed questionnaire, and 110 provided examples of mathematics and science lessons they taught, including materials given to students and more detailed reports on teacher and student activities conducted with those materia ls. (Note 1) The sample is highly
6 of 25 representative with regard to district wealth as me asured by DFG (See Table 1). Past research suggests that successful chan ge in teaching practice depends on opportunities for teachers to learn new practices r equired by the policy (Cohen & Barnes, 1993; Firestone et al., 1998). However, the kind of professional development that is most likely to lead to substantial change i n practice continues to be rare (Loucks-Horsley, Hewson, Love, & Stiles 1997). In o rder to assess the effects of professional development, we sought to oversample s chools that were known to engage in extensive professional development with respect to mathematics and science. The New Jersey State Systemic Initiative shared with us results of a survey identifying districts engaged in the most extensive professiona l development in those subjects. We attempted to ensure that 25% of our sample came fro m these districts. In fact 49 of the completed telephone interviews (20%) and 30 of the completed mailed questionnaires (17%) came from high professional development distr icts.Table 1 Distribution of Responses by DFGDistrict Factor Group AB:(Poorest)CD DE FG GH IJ:(Wealthiest)Total Interviews712932243554245 Percent29%12%13%10%14%22%100%Questionnaires492123142540172 Percent28%12%13%8%15%23%100%4th Grade Studentsin State (%) 30%9%15%13%13%19%100% In the following section we explore what content is being taught, teachers' access to materials, the extent of teaching to the test, self -reported knowledge about standards, and teachers' access to professional development.Content Coverage Standards and assessments are supposed to b e able to influence the content taught to children. Smith (1991) and Corbett & and Wilson (1991) found that the introduction of minimum competency tests narrowed the range of s ubjects taught in a school to what was on the test. Firestone, Mayrowetz, & Fairman (1 998) suggested that the introduction of more complex performance assessments can affect the presence and order of topics taught. There is reason to believe that the new sta ndards and assessments are affecting content coverage in New Jersey. Fifteen percent of our sample said they were teaching more math and 14% said they were teaching more scie nce. Noticeable changes are being made within each content area but these are differe nt in mathematics and science. Math Content Traditionally, elementary mathematics has f ocused on basic arithmeticaddition
7 of 25and subtraction of whole numbers with some introduc tion of fractions and decimals and geometric shapes. New Jersey's Core Curriculum Cont ent Standards expect the introduction of a wide range of content at the four th grade level, including a broader range of geometric issues; the foundations of algeb ra; better understanding of measurement; an introduction to statistics, probabi lity, and data analysis; and discrete mathematics (NJSDE, 1996). We wanted to access how teachers were using their time in mathematics and how that time use was changing. In order to avoid influencing respondents familiar with the standards terminology we identified 17 topics that represented a mix of classic elements of the elemen tary mathematics curriculum and areas that were not likely to have been taught befo re the standards were introduced [Appendix C]. We then asked teachers how many lesso ns they taught each of the 17 topics, and whether they had increased or decreased the time allocated to each topic in the last three yearsi.e., when the standards were being introduced and the ESPA was being given for initially. Although we do not have a firm fix on how t ime was allocated to topics before the standards were introduced, it appears that the gap between conventional and newer topics is being reduced with teachers adding time t o newer topics. Working with experts familiar with math teaching in the state, we identi fied three traditional topics: paper and pencil mathematical operations with whole numbers, adding and subtracting decimals via paper and pencil, and place value relationships (whole numbers, decimals); and three newer topics: open sentences, use of variables (str ategies used to prepare students for algebra), probability, and dealing with data (colle cting, organizing, analyzing, and displaying data). Most teachers reported that they spent many lessons on whole number operations: 96% spent eleven or more lessons a year on that topic. In addition, 58% devoted eleven or more lessons to place value relat ionships, and 22% spent that much time on adding and subtracting decimals. Although f ewer teachers devoted substantial time to the newer topics, 50% spent 11 or more less ons on dealing with data. Thirty three percent spent 11 or more lessons on open sent ences, and 14% on probability. Although the larger balance of teaching tim e was spent on older topics, most teachers reported increasing the amount of time the spent on the new topics (Fi gure 1). In general time spent on the older topics remained fairly constant, with the exception of whole number operations. A large portion of teacher s (29%) reported decreasing time spent on whole number operations. Based on this evi dence, it appears that newer topics are taking a more prominent place in the curriculum but not necessarily replacing older topics.
8 of 25 Figure 1. Percent Changes in Mathematics Items We also explored whether the time allocated to topics was the same in wealthy and poor school districts. In 13 of the 17 topic areas there were no significant differences between DFGs. However, in four topics identified as new by our mathematics experts, we noted an interesting u-shaped pattern. Teachers in poor, urban districts and the wealthy districts spent more time on these topics than midd le income districts (Table 2). An explanation for this pattern has not yet been found .Table 2 Differences by DFG in Lessons Allocated to Math Top ics(Percent of teachers devoting 11 or more lessons to a topic, n = 151-154) District Factor Group Abbott*C-EF-HIJ Probability27% 12% 3% 19% Patterns, functions49% 16% 21% 36% Open sentences46% 29% 19% 41% Discrete math54% 25% 16% 36% District wealth is generally measured by DFG. The Abbott districts are all DFG A or B and have been designated by the state Supreme Cou rt as those where spending must be equalized with wealthy districts in the state. T he DFG metric runs from A (districts with large numbers of poor and generally at-risk ch ildren) to IJ with large numbers of children from wealthy families. Teachers from DFG-B districts that are not "Abbott districts" have been excluded from this comparison.
9 of 25 Figure 2. Percent Changes in Science ItemsAccess to Materials New Jersey's Core Curriculum Content Standa rds place an increased emphasis on a more active role for students to take in learning m athematics and science. The mathematics standards require students to "develop an ability to pose and solve mathematical problems, develop reasoning ability a nd become self reliant independent mathematical thinkers; [and] regularly and routinely use calculators, computers, manipulatives, and other mathematical to ols to enhance mathematical thinking, understanding, and power" (New Jersey Sta te Department of Education, 1996, p. 4-9). The science standards require that student s "develop problem-solving, decision-making, and inquiry skills, reflected by f ormulating usable questions and hypotheses, planning experiments, conducting system atic observations, interpreting and analyzing data, drawing conclusions and communicati ng results" (New Jersey State Department of Education, 1996, p. 5-3). These chang es are in keeping with national standards which require more problem solving in mat hematics and hands-on inquiry in science. At the same time they place greater demand s on districts to provide additional materialsmathematical manipulatives, calculators a nd computers, the wherewithal for scientific experimentsbeyond the basic textbooks t hat have been so typical of American teaching (Cuban, 1993). In fact, some text books include alternatives like science kits or math manipulatives. Access to teaching equipment and supplies h as historically been unequal, favoring wealthy districts. In the early 1990s, teachers in poor, urban districts reported less access to both textbooks and computers than their peers in wealthy districts. For a period of time following the passage of the Quality Education Act (QEA) which increased funding to
10 of 25urban districts for a short time in the early 1990s there was some indication that poor districts were working hard to bridge the gap betwe en themselves and wealthier districts. However, they have not been successful (Firestone e t al., 1997). The current study indicates that access to materials may be improving in poor districts. Across DFGs teachers reported having eno ugh materials for most purposes, especially for teaching mathematics. Ninety-five pe rcent of the teachers surveyed reported having enough math textbooks for every chi ld to have one. (Note 2) Ninety-four percent reported having enough manipulatives for ch ildren to share, and 97% reported enough calculators for every child. The situation i s nearly as good in science where 77% of the teachers reported having enough textbooks fo r every child, 76% reported enough science kits either for every child or for children to share, and 85% reported enough measurement and observation tools to share. Use tends to lag behind access. Seventy eig ht percent of teachers report using their math texts almost every day, (Note 3) 66% use manip ulatives once or twice a week, and 53% use calculators once or twice a week. The patte rn in science is somewhat different. While 36% report using a textbook everyday, 40% rep ort using it once or twice a week. Sixty-five percent report using science kits at lea st once a week, and 38% report using measurement and observation tools that often. We did not identify any inequities in acces s to mathematics materials, supported by the high percentage of teachers who reported having enough math textbooks for every child. The situation in science is more complicated because teachers in poor, urban districts appear to emphasize the use of textbooks, while those in the wealthier districts balance textbooks with the use of science kits and other materials (Figure 3). Almost all the teachers in the Abbott districts and mid-wealth districts say they have enough science textbooks for every child and more than four fifths use them weekly. However, less than half the teachers in the wealthy districts have eno ugh textbooks for every child and use them weekly. A third of the teachers in wealthy dis tricts have enough kits for every child and two thirds use them weekly. Figure 3. Access To and Use Of Science Materials
11 of 25Kits are much less accessible in the poor and mid-w ealth districts. Still about half the teachers in urban districts report using them weekl y and use in the mid-wealth districts is comparable to that in the wealthy districts. The pa ttern of access to tools for observation and measurement parallels that to access to kits wi th substantially more teachers reporting having enough for every child in the weal thiest districts. There is a gradual trend of increasing use as one moves from the Abbot t to the wealthiest districts. The reasons for these differences are not clear. Howeve r, the fact that most teachers in the state report little change in their access to mater ials suggests that this pattern reflects a difference in philosophy about how to teach science more than recent changes in funding.Teaching to the Test One of the greatest concerns with standards and assessment-based reform has been that this strategy might lead to teaching to the te st and its concomitant negative effects such as narrowing the curriculum; constricting inst ruction time; increasing the amount of drill while undermining efforts to promote higher o rder thinking skills; and increasing stress for teachers and students (Corbett & Wilson, 1991; Smith, 1991). There is also a fear that teaching to the test will undermine the v alidity of test results by artificially inflating test scores (Mehrens, 1998). There has be en some question about whether these are inevitable effects of highstakes accountabili ty-oriented tests. Some have suggested that changes in test format should include more per formanceoriented items and test items assessing more than mere retention of facts a nd computation skills might lead to tests worth teaching to and encourage teaching that promoted more conjecture, exploration, and active participation in learning ( Baron & Wolf, 1996; Rothman, 1995). To explore the distribution of teaching to the test in the state, we developed a seven-item scale with a mixture of items that seeme d to reflect some of the feared negative effects of this practice and others constr ued as positive. The scale had an alpha coefficient of .71. Specific items included: Teach test staking mechanics like filling in bubble s, how to put your name on the test, or how to pace yourself during the test. 1. Motivate students to make their best effort on the ESPA, such as suggesting they prepare by getting a good night's sleep or encourag ing them to try hard. 2. Have students use rubrics to grade each other's wor k. 3. Teach the regular curriculum using performance-base d exercises similar to the ESPA. 4. Teach test-besting skills like methods for turning story problems into arithmetic calculations or how much to write after an open-end ed math item. 5. Use commercial test-preparation materials like "Sco ring High" and "Measuring Up on the ESPA." 6. Give practice tests with items similar to those on the ESPA. 7. We asked teachers how often they performed these activities (on a scale of 1-4) all year long and the month before the ESPA was given. (Note 4) Figure 4 shows two patterns in teachers' reported teaching to the test First, as might be expected, there is a small increase in activity during the month before the test compared to the entire year (scale mean of 2.50 for the whole year versus 2.86 for the month before the test). Second, there is a distinct pattern of teachers in the Abbo tt districts reporting more teaching to the test than teachers in the wealthiest districts. Tea chers in the midwealth districts fell
12 of 25somewhere in between. Thus, the emphasis on test pr eparation as a separate activity were concentrated in the districts that most need help i n improving student learning. Figure 4. Teaching to the TestFamiliarity with Standards We asked teachers to report how familiar th ey are with state and national standards in mathematics and science. Teachers' familiarity w ith state standards could contribute to changes in content taught, although central office staff who understand state standards and assessments can unilaterally change district cu rriculum. The national standards movement in science, and especially in mathematics precedes New Jersey's efforts by several years; and some districts were using those national standards to guide changes before state standards were adopted or tests were i mplemented. Teachers were much more familiar with state than national standards. Fifty-seven percent said they understood the state's mathematic s standards well, (Note 5) and 53% say they are understand the science standards well. In contrast, only 28% said that they understood the national mathematics standards well and 16 said they understood the national science standards well. Even if teachers o verestimated their understanding of the standards, the state's effort has increased attenti on to standards-based teaching here. For the most part, understanding of standar ds is equally distributed across wealthy and poor districts. The one exception is the nation al mathematics standards where there is a complicated pattern of differences between distri cts (Table 3). Generally, more teachers in the wealthy districts believed that they underst andd the national standards well. However, it is not true that most teachers in the A bbott districts have limited familiarity with the national math standards. The largest conce ntration having moderate familiarity is in the Abbott districts while the almost two thirds of the CE teachers have only limited
13 of 25 familiarity with the national standards. One possib ility is that the wealthy districts have sought to adopt the national standards for a long t ime. Growing familiarity in the Abbott districts may reflect a mix of three factors: a sid e effect of the attention to standards in general from the adoption of state standards, the s pecial pressures placed on the Abbott districts by the state as a by-product of the serie s of court cases and large amount of state money going to those districts (Firestone & Nagle, 1995), and the additional funds coming from CEIFA after the Abbott IV decision.Table 3 Understanding of National Mathematics Standards by DFG(Percent of Teachers, n = 158) District Factor Group AbbottC-EF-HIJ Limited*37%63%32%33%Moderate**47%21%29%28%Extensive***16%16%40%39%* Awareness only and read through once or twice.** Understand somewhat (can implement parts in clas s) *** Understand well (can implement fully in class) and expert (could lead workshop)Professional Development Past research on policy implementation in a variety of fields suggests that regardless of changes in incentives and punishments, teachers will not change their practice until they have learned how to perform the new tasks expe cted of them (Berman 1986, Cohen and Barnes, 1993). Firestone and colleagues (1998) suggest that one reason state-administered performance-based assessment has had limited impact on teaching is because teachers have had limited opportunities to learn the new content and pedagogy required by the new assessments. Teachers reported on several dimensions of their professional development experience. Regarding the source of professional de velopment, most learning opportunities for teachers came directly from the d istrict. Sixty seven percent of teachers reported that some time in their district-provided professional development days in the last year had been devoted to mathematics or scienc e. In the last year, 40% had mentored student teachers or first year teachers, 41% had se rved on district curriculum development or textbook selection committees, and 2 1% had served as lead or specialist teachers helping other experienced teachers in thei r district. All of these are learning experiences even though they may involve helping ot hers. Relatively few teachers had opportunities t o develop new knowledge by interacting with experts from outside the district. Eighteen pe rcent had taken a college course in math, science, or math or science education in the last year. Twenty two percent had participated in one the programs for improving math and science teaching supported by the National Science Foundation through its State a nd Local Systemic Initiatives or the US Department of Education through its Eisenhower g rants to institutions of higher education. Given elementary teacher's reputation fo r aversion to mathematics and science,
14 of 25 these numbers are fairly reasonable. However, since the objective is to achieve statewide high quality mathematics and science teaching, it s eems quite unlikely that teachers' understanding of effective practice will grow quick ly unless more avail themselves of these opportunities. One recurring criticism of professional dev elopment is that it is usually provided through one-shot workshops where teachers receive l imited and often inapplicable information with little or no follow up to help in using what they are supposed to have learned. That seems to have been the case among New Jersey's fourth grade teachers (Table 4). Only about one fifth of the teachers rep orted having more than two days of professional development on either content and inst ruction in science and math. Slightly fewer received more than two days of professional d evelopment on strategies to help students score high in math or science. It is somew hat encouraging that teachers received about as much professional development on the under lying content and instructional issues as they did on strategies to raise test scor es. On the other hand, only one in 20 received more than two days on using assessment res ults. It is particularly disconcerting that teachers received so little support in using a ssessment results to improve instruction, although this may be because the state had not yet reported any ESPA results to schools when this survey was conducted. Not only is professional development limite d, so is follow up. Between 20% and 30% of the teachers report being visited later by a trainer. Follow up by principals is more common, but principals are often less well informed about the content of professional development. Their follow up may show concern and s ignal that the material covered is important, but substantive assistance is likely to be less than that coming from an expert. Nevertheless, between one third and one half the te achers found the professional development they received to be very useful. This m ay be in part a reflection of the growing demand for help in this area.Table 4 Time in Professional Development(Percent Reporting Various Categories) More than 2 days PD in year Follow-up by trainer Follow-up by principal PD is very useful Content and instruction in science 22% 25% 22% 44% Content and instruction in math 20% 25% 26% 48% Using assessment results 6% 21% 35% 30% Strategies to score high in math 19% 29% 33% 48% Strategies to score high in science 14% 22% 29% 41%
15 of 25 Where New Jersey teachers received more pr ofessional development, they found it more useful. The correlation between the amount of time spent in professional development and its perceived utility were .66 for content and instruction in science, .63 for content and instruction in mathematics, and .61 for using assessment results. They were lower for strategies for scoring high in math and science (.44 and .40, respectively). These findings suggest that extensive professional development efforts will be most helpful when helping teachers better understand the underlying material in a subject and effective strategies for helping students learn it. Longer time investments may also pay off for helping teachers to use assessment strategi es to improve practice. Comparable concentrations are probably not as necessary to giv e teachers strategies to raise test scores.Discussion While there are limitations to what can be learned about changes in teaching practice from one administration of a survey that f ocuses on elementary school mathematics and science, the data presented here su ggest some tentative conclusions and raise questions about two issues: ongoing changes i n practice, and differences between wealthy and poor districts. Statewide, it appears that the topics taugh t as part of the 4th grade curriculum are changing. This may have implications for elementary curriculum in general. In mathematics, what had been an unremitting diet of w hole number facts is being leavened with other topics like probability and dealing with data. Generally, more science is being taught, and the small sampling of biology and meteo rology is being expanded. There is a large increase in attention to the process of scien tific investigation, some increase in attention to the introduction of chemistry and at l east a smattering of attention to physics-related topics. These changes help prepare children to use mathematics as part of their adult life and give them an introduction to a broader range of science topics. The simple addition of topics may be a mixe d blessing, however. One criticism of mathematics teaching in the past has been that too many topics are taught at too little depth (Schmidt, McNight, & Raizen, 1997). The addit ion of new topics to the state standards could exacerbate such shallow coverage. T he quality and depth of coverage is difficult to assess with surveys; hopefully, direct observation in classrooms, which is currently underway, will help address this issue. I t will also be useful to collect longitudinal data on coverage of content areas to v erify that the changes we believe are happening are in fact taking place. Teachers are al so becoming more familiar with the state standards, and believe they are more familiar with state than with national standards. We suspect that the extent of their fami liarity is overstated. Again, we hope to learn more from direct observation. On the equity front, the picture is mixed. The good news is that some of the obvious inequities in access to materials that were prevalent at the beginning of the decade appear to be fading. However, there are hint s that two pedagogies may be developing in the state: one for children in distri cts serving the poor, and another for districts serving the wealthy. Pedagogy in the poor districts may come to be dominated by conventional, textbook-oriented teaching and tea ching to the test, while wealthier districts seem to be moving towards more explorator y, active modes of learning that are less dependent on textbooks and less driven by stat e tests. If so, the reasons are likely to have less to do with differences in funding and mor e with heavier pressures to comply with state expectations in urban districts and the challenges that come with teaching
16 of 25poorer children (Natriello, Pallas & McDill, 1990). There is also the issue of those districts in the middle of the DFG distribution. These more working-class districts are not as well funded as either the Abbott districts or the wealthy districts. There are some indication s that teachers in the Abbott districts are moving faster than those in the poorer of the m id-wealth districts to embrace the standards and introduce new topics to the curriculu m. How strong this trend is, whether it will continue, and what its implications are for teaching practice and student achievement remain to be explored through further s urveys and direct observation in classrooms.NotesThis article was presented as a paper at the Annual Meeting of the American Educational Research Association in New Orleans, LA April, 2000. We wish to thank Warren Crown, Roberta Schorr, John Shafransky, Shar on Sherman, and Carol Stearns for their assistance. This research was supported b y a grant from the National Science Foundation. The opinions expressed are those of the authors. Neither the Foundation nor Rutgers University is responsible for them. The teacher work samples are not used in this repor t. 1. The choices offered teachers were none, one or two to demonstrate in class, enough for children to share, and enough for every child to have one. 2. The options were almost every day, once or twice a week, once or twice a month, once or twice a semester, and never. 3. Respondents were asked to report on a 4-point scale where 1 was "almost never" and 4 was "almost always." 4. The actual choices were "Awareness only, read throu gh once or twice, understand somewhat (can implement parts in class), understand well (can implement fully in class), and expert (could lead workshop)." The resp onses reported are for the last two combined. 5.ReferencesBaron, J.B. & Wolf, D.P. (1996). Performance-based student assessment: challenges and possibilities Chicago, IL: University of Chicago. Berman, P.E. (1986). From compliance to learning: I mplementing legally-induced reform. In D. Kirp & D. Jensen (Eds.), School days, rule days (pp. 46-62). Philadelphia, PA: Falmer.Cohen, D.K., and Barnes, C.A. (1993). Pedagogy and policy. in D.K. Cohen, M.W. McLaughlin, & J.E. Talbert (Eds.) Teaching for understanding (pp. 207-239) San Francisco: Jossey Bass.Corbett, H.D. & Wilson, B.L. (1991). Testing, reform, and rebellion. Norwood, NJ: Ablex.Cuban, L. (1993). How teachers taught: Constancy and change in Americ an classrooms, 1890-1980 (2 nd ed.) New York: Teachers College Press.
17 of 25Erlichson, B.A., Goertz, M. & Turnbull, B. J. (1999 ) Implementing whole school reform in New Jersey: year one in the first cohort schools New Brunswick, NJ: Rutgers, the State University of New Jersey.Firestone, W. A., & Nagle, B. (1995). Differential regulation: Clever customization or unequal interference? Educational and Evaluation and Policy Analysis 17(1), 97-112. Firestone, W.A., Goertz, M.E., & Natriello, G.J. (1 997) From cashbox to classroom: The struggle for fiscal reform and educational chan ge in New Jersey New York: Teachers College Press.Firestone, W.A., Mayrowetz, D., & Fairman, J. (1998 ). Performance-based assessment and instructional change: The effects of testing in Maine and Maryland. Educational Evaluation and Policy Analysis 20 (2)95-113. Goertz, M. & Malik, E. (1999) In search of excellen ce for all: the courts and New Jersey school finance reform. Journal of Education Finance, 25 (Summer)532. Loucks-Horsley, S., Hewson, P.W., Love, N., & Stile s, K.E. (1997) Designing professional development for teachers of mathematic s and science Thousand Oaks, CA: Corwin Press.McDonnell, L. M. (1995). Opportunity to learn as a research concept and a policy instrument. Educational Evaluation and Policy Analysis 17 (3), 305-22. Mehrens, W. A. (1998). Consequences of Assessment: What is Evidence? [Online]. Educational Policy Analysis Archives 6 (13). Available: http://epaa.asu.edu/epaa/v6n13.htmlMillman, J. & Green J. (1989). The specification an d development of tests of achievement and ability. In R. L. Linn (Ed.), Educational measurement (pp. 335-366). New York: Macmillan.Natriello, G., McDill, E.L. & Pallas, A.M. (1990). Schooling disadvantaged children: Racing against catastrophe New York: Teachers College Press. New Jersey State Department of Education. (1996). Core curriculum standards Trenton, NJ: Author.Porter, A. (1999). Creating a system of school proc ess indicators. Education Evaluation and Policy Analysis 13 (1), 13-29. Quality Counts 2000. (2000). Education Week XIX (18). Raymond Arthur Abbott, et al. v. Fred G. Burke, Com missioner of Education, et al. 643 A.2d 575, 580 (1994) (Abbott III)Raymond Arthur Abbott, et. al. v. Fred G. Burke, Co mmissioner of Education, et. al. 693A.2d 417, 428 (1997) (Abbott IV).Rothman, R. (1995). Measuring up: standards, assessment, and school ref orm San
18 of 25Francisco, CA: Jossey-Bass.Schmidt, W. H., McNight, C. C., Raizen, S. A. (1997 ) A splintered vision: an investigation of U.S. science & mathematics educati on New York, NY: Kluwer Academic PublishersSmith, M. & O'Day, J. (1991). Systemic school refor m. In S. Fuhrman & B. Malen (Eds.) The politics of curriculum and testing. (pp. 233-67). Bristol, PA: Falmer Press. Smith, M.L. (1991). Put to the test: The effects of external testing on students. Educational Researcher, 20(5), 8-12.Smith, M.S., Fuhrman, S.H. & O'Day, J (1994) Nation al curriculum standards: Are they desirable and feasible? In Elmore, R.F. & Fuhrman, S.H. (Eds.) The governance of curriculum (pp. 12-29). Alexandria, VA; ASCD. Wiley, D.E. & Yoon, B. (1995). Teacher reports on o pportunity to learn: Analyses of the 1993 California Learning Assessment System (CLAS). Educational Evaluation and Policy Analysis 17 (3), 355-70.About the AuthorsWilliam A. Firestone Center for Educational Policy AnalysisRutgers University Email: firstname.lastname@example.org William A. Firestone is Professor of Educational Po licy; Chair of the Department of Educational Theory, Policy, and Administration and Director of the Center for Educational Policy Analysis. His research on the ef fects of both testing and professional development on teachers has appeared in the America n Educational Research Journal, Educational Evaluation and Policy Analysis, and Kap pan. His most recent book is From Cashbox to Classroom: School Finance Reform and Edu cational Change in New Jersey (with Margaret E. Goertz and Gary Natriello).Gregory Camilli Rutgers University Email: email@example.com Gregory Camilli is Professor, Department of Educati onal Psychology, at the Rutgers Graduate School of Education. His areas of research interest include psychometric issues in educational policy, metaanalysis, and differen tial item functioning. Dr. Camilli is a member of the editorial Boards of Educational Measurement: Issues and Practice, Educational Policy Analysis Archives, and Educational Review He is a regular reviewer for Applied Measurement in Education, Journal of Educat ional Measurement, Psychometrika, and Psychological Methods among others. As a member of the Technical Advisory Committee of the New Jersey Basi c Skills Assessment Council, he provides expertise on testing and measurement issue s to the state's assessment program.
19 of 25Michelle Yurecko Center for Educational Policy AnalysisRutgers UniversityMichelle Yurecko, a Ph.D. candidate in Educational Psychology and a research associate at the Center for Educational Policy Analysis. She is a statistician and focuses on educational testing and measurement. Lora Monfils Center for Educational Policy AnalysisRutgers UniversityLora Monfils, a doctoral candidate in Educational S tatistics and Measurement, is a research associate at the Center for Educational Po licy Analysis and a mathematics educator. Her research interests concern large-scal e assessment and modeling differential educational outcomes.David Mayrowetz Center for Educational Policy AnalysisRutgers UniversityDavid Mayrowetz is a doctoral candidate in the Depa rtment of Educational Theory, Policy and Administration, Rutgers University. His interests include policy implementation, inclusion of students with disabili ties, and assessment reform. He is the co-author, with William Firestone of "Rethinking "H igh Stakes:" Lessons from the US and England and Wales" (Teachers College Record, fo rthcoming) and with Carol Weinstein, of "Sources of Leadership for Inclusive Education: Creating Schools for All Children" ( Educational Administration Quarterly, September 1999). He will be joining the Policy Studies faculty of the University of Ill inois at Chicago in January 2001.Appendix A New Jersey's Core Curriculum Content StandardsMathematics: All students will develop the ability to pose and s olve mathematical problems in mathematics, other disciplines, and every day exper iences. 1. All students will communicate mathematically throug h written, oral, symbolic, and visual forms of expression. 2. All students will connect mathematics to other lear ning by understanding the interrelationships of mathematical ideas and the ro les that mathematics and mathematical modeling play in other disciplines and in life. 3. All students will develop reasoning ability and wil l become self-reliant, independent mathematical thinkers. 4. All students will regularly and routinely use calcu lators, computers, manipulatives, and other mathematical tools to enha nce mathematical thinking, 5.
20 of 25understanding and power.All students will develop number sense and an abili ty to represent numbers in a variety of forms and use numbers in diverse situati ons. 6. All students will develop spatial sense and an abil ity to represent geometric properties and relationships to solve problems in m athematics and in everyday life. 7. All students will understand, select, and apply var ious methods of performing numerical operations. 8. All students will develop an understanding of and w ill use measurement to describe and analyze phenomena. 9. All students will use a variety of estimation strat egies and recognize situations in which estimation is appropriate. 10. All students will develop an understanding of patte rns, relationships, and functions and will use them to represent and explai n real-world phenomena. 11. All students will develop an understanding of stati stics and probability and will use them to describe sets of data, model situations and support appropriate inferences and arguments. 12. All students will develop an understanding of algeb raic concepts and processes and will use them to represent and analyze relation ships among variable quantities and to solve problems. 13. All students will apply the concepts and methods of discrete mathematics to model and explore a variety of practical situations 14. All students will develop an understanding of the c onceptual building blocks of calculus and will use them to model and analyze nat ural phenomena. 15. All students will demonstrate high levels of mathem atical thought through experiences which extend beyond traditional computa tion, algebra, and geometry. 16. Science: All students will learn to identify systems of inte racting components and understand how their interactions combine to produc e the overall behavior of the system. 1. All students will develop problem-solving, decision making and inquiry skills, reflected by formulating usable questions and hypot heses, planning experiments, conducting systematic observations, interpreting an d analyzing data, drawing conclusions, and communicating results. 2. All students will develop an understanding of how p eople of various cultures have contributed to the advancement of science and techn ology, and how major discoveries and events have advanced science and te chnology. 3. All students will develop an understanding of techn ology as an application of scientific principles. 4. All students will integrate mathematics as a tool f or problem-solving in science, and as a means of expressing and/or modeling scient ific theories. 5. All students will gain an understanding of the stru cture, characteristics, and basic needs of organisms. 6. All students will investigate the diversity of life 7. All students will gain an understanding of the stru cture and behavior of matter. 8. All students will gain an understanding of natural laws as they apply to motion, forces, and energy transformations. 9.
21 of 25All students will gain an understanding of the stru cture, dynamics, and geophysical systems of the earth. 10. All students will gain an understanding of the orig in, evolution, and structure of the universe. 11. All students will develop an understanding of the e nvironment as a system of interdependent components affected by human activit y and natural phenomena. 12.Appendix B Content Area Topics From The Teacher SurveyMathematics: Paper and pencil mathematical operations with whole numbers (adding, subtracting, multiplying & dividing) 1. Doing mental math operations with whole numbers (ad ding, subtracting, multiplying & dividing) 2. Estimation (magnitude, results of computation, meas urement) 3. Place value relationships (whole numbers, decimals) 4. Adding and subtracting decimals via paper and penci l 5. Identification of geometric figures 6. Area and Perimeter 7. Fraction Concepts (Fractions as parts of a whole, e quivalency) 8. Operations with Fractions (addition, subtraction) 9. Measurement (customary, metric) 10. Probability 11. "Dealing with data" (collecting, organizing, analyz ing and displaying data) 12. Statistics 13. Graphing 14. Patterns, functions 15. Open sentences, use of variables 16. "Discrete math" (Combinations, puzzles, optimizatio n, classification, algorithms, networks, tree diagrams) 17. Science: Understanding natural and man-made systems (recogni zing systems, identifying parts) 1. Investigative skills (observing, classifying, deali ng with data) 2. Using mathematics (measurement, estimating, countin g) 3. Nature and history of science & scientists 4. Selecting and using tools 5. Needs of living things/Life systems 6. Habitats, ecosystems, & adaptation 7. Features and classifications of plants and animals 8. Structure and physical properties of matter 9. States of Matter: Solid, liquid, gas (heating and c ooling) 10. Forces, motion & energy 11.
22 of 25 Invisible forces (gravity, electricity & magnetism) 12. Earth Materials: Rocks, soil, fossils 13. Weather and climate 14. Earth, moon, sun system 15. Stars and galaxies 16. Humans and the environment 17.Appendix C Sample ESPA ItemsTraditional Mathematics Item: Find the exact answer: 110 + 70 18 1. 81 2. 180 3. 810 4. Newer Mathematics Item: Mr. Jones gave each of the students in his class a one-ounce box of raisins. When the students opened the boxes and counted the raisins, they found different amounts. The tally sheet below shows thei r results. Number of RaisinsTallyFrequency 10|111||212|||313|||||514|||315||2 Construct a bar graph to represent the students' fi ndings on the grid in your answer booklet. Be sure to label your graph complet ely. Traditional Science Item: Which thing does a living duck do that a toy duck d oes not do? Floats on water 1. Breathes air 2. Makes a sound 3. Sits still 4. Newer Science Item:
23 of 25 Victor has two glasses. One glass is filled with ic e cubes and the other is filled with water. Give three ways the ice and wate r are different. Copyright 2000 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, firstname.lastname@example.org or reach him at College of Education, Arizona State University, Tempe, AZ 8 5287-0211. (602-965-9644). The Commentary Editor is Casey D. C obb: email@example.com .EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov firstname.lastname@example.org Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Western Interstate Commission for HigherEducation William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX)
24 of 25 Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton email@example.com Hugh G. Petrie SUNY Buffalo Richard C. Richardson New York University Anthony G. Rud Jr. Purdue University Dennis Sayers Ann Leavenworth Centerfor Accelerated Learning Jay D. Scribner University of Texas at Austin Michael Scriven firstname.lastname@example.org Robert E. Stake University of IllinoisUC Robert Stonehill U.S. Department of Education David D. Williams Brigham Young UniversityEPAA Spanish Language Editorial BoardAssociate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico email@example.com Adrin Acosta (Mxico) Universidad de Guadalajaraadrianacosta@compuserve.com J. Flix Angulo Rasco (Spain) Universidad de Cdizfelix.firstname.lastname@example.org Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho dis1.cide.mx Alejandro Canales (Mxico) Universidad Nacional Autnoma deMxicocanalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State Universitycasanova@asu.edu Jos Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es Erwin Epstein (U.S.A.) Loyola University of ChicagoEepstein@luc.edu Josu Gonzlez (U.S.A.) Arizona State Universityjosue@asu.edu Rollin Kent (Mxico)Departamento de InvestigacinEducativa-DIE/CINVESTAVrkent@gemtel.com.mx email@example.com Mara Beatriz Luce (Brazil)Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.brJavier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mxMarcela Mollis (Argentina)Universidad de Buenos Airesmmollis@filo.uba.ar Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mxAngel Ignacio Prez Gmez (Spain)Universidad de Mlagaaiperez@uma.es
25 of 25 Daniel Schugurensky (Argentina-Canad)OISE/UT, Canadadschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil)Fundao Instituto Brasileiro e Geografiae Estatstica firstname.lastname@example.org Jurjo Torres Santom (Spain)Universidad de A Coruajurjo@udc.es Carlos Alberto Torres (U.S.A.)University of California, Los Angelestorres@gseisucla.edu