With the increasing importance placed on testing, it's necessary to look at how tests and assessment tools are designed and the quality of data they yield.

Madhabi Chatterji, Associate Professor of Measurement, Evaluation, and Education does just that. On March 12, she spoke about her new book, Designing and Using Tools for Educational Assessment and her new study "Correlates of Early School Achievement: A Study of Black, Hispanic, Asian and Non-Minority First-graders in the U.S." at an event sponsored by The Institute for Urban and Minority Education (IUME). The study was supported by IUME in the summer of 2002.

In his introduction, Henry M. Levin, the William H. Kirkpatrick Professor of Economics and Education, noted that Chatterji covers a lot of ground in evaluation and "creates a presence in it that is visible."

Chatterji began the discussion by thanking Edmund Gordon, Director of IUME for giving her the opportunity to start in this line of work. Then, she talked about her new book Designing and Using Tools for Educational Assessment. "The things we like to measure in education aren't tangible, so our measurements are prone to all kinds of error," she said. "It's not enough to simply design the tools that help us measure things; we must build into the design process systematic ways to check for errors in our tools and data."

Her book, which is intended for those who design or use tests and other assessment tools for various purposes, including research, describes a process model that includes validation in context, prior to actually using tools. The work of designing, validating, and using data from tests and other instruments follows the quantitative tradition of research. The model is a way of working, designing, and using assessment tools in appropriate ways.

"We need to talk to legislators and let them know that certain tests only give certain types of information, show them the evidence that supports particular uses of tests and their scores," said Chatterji. "They need to evaluate themselves whether or not they can make the statements they are making when there is no supporting validity data for using test scores in the way they want."

In the second half of her lecture, she talked about her "Correlates of Early School Achievement" paper which she says is a work in progress. The study looks at educational statistics from a survey conducted by the National Center for Educational Statistics that began in 1998 using a longitudinal data set from Kindergarten to First Grade. To understand how we can we narrow the achievement gap between minorities and non-minorities, the study attempts to identify child- and school-level factors that influence reading and math achievement of first graders.

This kind of research can help provide information to policy-makers and legislators who make laws like the No Child Left Behind (NCLB) Act, she said. They need to understand the realities that schools face. Right now, laws like the NCLB assumes that it is possible for schools to achieve equal outcomes for all categories of students within fixed time frames.

Findings show that even in first grade, what children come to school with-their risk levels-influence their readiness and what they achieve. Not every child is equally prepared and there is only so much that schools can do. In addition, children's ethnicity, poverty levels, gender, and achievement at kindergarten, influence a school's average achievement as measured by test scores in first grade. The size of the achievement gap compared to a school's average is different for various ethnic groups.

The gaps are different in math and reading. In reading, gender also makes a difference. For example, males tend to bring down a school's mean score by about two points.

Even when Chatterji's analysis put all children at the same poverty and socioeconomic level, age and ethnicity, average achievement scores for schools were influenced by variables such as class size. The higher the class size, the lower the achievement mean for a school. But, the reality is that children attending different schools are never all equivalent. Different schools serve children with different needs. So, the next step will be to see how children's differences interact with what schools do.

"No Child Left Behind makes some unreasonable assumptions," said Chatterji. "It's intent is good, but it's misguided policy to try to use high stakes achievement tests as a means to help schools get better without looking at student differences."

Published Monday, Apr. 7, 2003