The International Observatory on Academic Achievement - Documentary search

File number :
CS-APE-27e

Bibliographic reference :

Desimone, L.M., Smith, T.M., Hayes, S.A., & Frisvold, D. (2005). Beyond Accountability and Average Mathematics Scores: Relating State Education Policy Attributes to Cognitive Achievement Domains, Educational Measurement: Issues and Practice, 24(4), 5-18.

Abstract :

Research Overview
In this article, the authors present one of the very few assessments of the direct effect of educational policies on learning. More particularly, the article deals with the effect of the U.S. federal policy No Child Left Behind (NCLB) on mathematics achievement.

The authors addressed three research questions: 1) To what extent do state policy attributes predict change in state-average student achievement between 2000 and 2003? 2) To what extent do the relationships between state policy attributes and changes in state-average student achievement differ by proficiency level (basic, proficient or advanced), ethnicity and socio-economic level? 3) Do policy attributes have differential effects on change in achievement among different mathematics content domains, i.e. procedural knowledge such as rules and strategies, conceptual understanding and problem solving ability?

Policy Analysis Framework
The authors used a theoretical framework developed by Andrew Porter in the 1980s and defined by five attributes that contribute to successful policy implementation:
1) Specificity, the extent to which policies are clear and complete (e.g., clear indications or guidelines provided or not);

2) Consistency, the extent to which all policy components and all policies in place at the same educational institution are aligned with each other (if there are contradictions or not);

3) Authority, the degree to which a policy has full support because of its coherence with social norms, applicability to laws, support by experts, etc.;

4) Power, the recognition granted and sanctions imposed as part of policy enforcement (e.g., funds allocated according to performance); and

5) Stability, the extent to which players, contexts and practices remain in place over time. However, this last attribute was not considered as part of this study.

To assess these different attributes in each state, the authors put together a database (State Policy Database) with four existing information banks.

Academic Achievement Measures
The authors examined the National Assessment of Educational Progress (NAEP) results. Since their goal was to measure mathematics achievement across the three cognitive domains mentioned above, they had to separate NAEP questions into categories corresponding to these three types of learning.

Sample
Selected randomly among schools with similar characteristics, there were 2000 participants from 39 states in 2000, while in 2003, about 3000 students from 50 states participated. All the students were assessed during their eighth grade.

Main Results
With regard to the first research question, it was found that power is the only significant predictor of positive change in NAEP scores. However, this correlation was no longer relevant when all the policy attributes were considered at once.

As for the second question, an increase in the percentage of white students at or above the second level of proficiency (proficient) seemed related to specificity. This attribute predicted a decrease in the percentage of low socio-economic students at or above the basic level of proficiency. It was otherwise for authority, which predicted an increase in this group’s achievement.

In terms of the influence of policy attributes on achievement in the mathematics domains, specificity and authority may be more related to improvements in mathematics procedural knowledge. However, no attributes seemed to have any effect on problem solving and conceptual understanding. In addition, too great power may be associated with a small decrease in achievement in the three learning domains.

The results also showed that changes in the percentage of low socio-economic students who had reached the basic proficiency level between 2000 and 2003 were associated with greater policy specificity. This relationship was not significant for the other study groups.

The authors also pointed out that the states with low NAEP scores and a large number of black and low-socio-economic level students showed greater improvements in their scores compared to states with high NAEP scores. However, the results of this study did not show that policy attributes were related to these gains.

Furthermore, power may be more negatively related to conceptual understanding than to the two other types of learning, while specificity and authority may be related to gains only in the case of procedural knowledge.

Lastly, the poorer NAEP scores were in 2000, the stronger the effect of power and authority was on the gains reported in 2003 in the cases of problem solving and conceptual understanding.

Links :
This journal is also available in electronic format

Key Words :
Federal Policy, Educational Policies, Policy Attributes, Mathematics, No Child Left Behind (NCLB), National Assessment of Educational Progress (NAEP), Ethnicity, Socio-economic Level, Policy Implementation, Educational Inequalities, Secondary/High School, Newsletter2