Anyone who has attended a local school board meeting knows that democracy is indeed, as Robert Kennedy said, “messy and hard.”  

Now a new online tool recently launched by Teachers College’s Center for Benefit-Cost Studies in Education (CBCSE) promises to improve that process for educators and education administrators.

The tool — called, appropriately enough, DecisionMaker — uses a framework based on cost-utility analysis to help schools, districts and state education agencies make choices about curricula, professional development, after-school programs and other resources. Building on methodologies developed by the late psychologist Ward Edwards and Teachers College education economist and CBCSE founding director Henry Levin, DecisionMaker, in essence, conducts a virtual town meeting, juxtaposing data on costs against  student outcomes and other, harder-to-quantify information such as stakeholder opinions and preferences. Ultimately, the program — which is available free of charge — guides users through a 10-step process, ranging from identifying solution options and screening criteria to incorporating stakeholder values and costs-per-participant for each option.

Creation of DecisionMaker was funded by the William T. Grant Foundation.

[Visit the DecisionMaker website to learn more or register to use the tool.]

“We all do multiple-criteria decision-making in our daily lives,” says Fiona Hollands, CBCSE Associate Director and Senior Researcher. “Let’s say we’re thinking about where to meet for coffee — the criteria for picking a venue might include how far away the place is, how expensive it is, whether they serve food as well as coffee, the service, the décor, who else might be there. We can weigh all that in our heads, but for high-stakes decisions, we need to be able to marshal all the evidence accurately to calculate what’s going to best meet our needs.”

[Read “What Is the Potential for Applying Cost-Utility Analysis to Facilitate Evidence-Based Decision Making in Schools?”, a paper by Hollands, Yilin Pan and Maya Escueta, published in the June/July 2019 issue of Educational Researcher.]

A previous CBCSE tool called CostOut, launched in 2015, has enabled educators to weigh costs and student outcomes, but “the education agencies we’ve worked with tell us that they also need to worry about buy-in from teachers, parents and their boards, because something that’s appropriate for New York City might not fly in, say, Tennessee,” Hollands says. “And they tell us that when decisions they’ve made in the past have failed, it’s often been because they didn’t get stakeholder buy-in from the beginning.”

The education agencies we’ve worked with tell us that they need to worry about buy-in from teachers, parents and their boards, because something that’s appropriate for New York City might not fly in, say, Tennessee. When decisions they’ve made in the past have failed, it’s often been because they didn’t get stakeholder buy-in from the beginning.

— Fiona Hollands

An administrator or teacher using DecisionMaker can solicit opinions or information from others at multiple points in the process. Stakeholders from parents to principals can log in and suggest programs to be considered, upload videos and other testimonials, and propose screening criteria (for example, that a program must be aligned with Common Core State Standards). They can indicate what criteria are most important to them in making the decision, and the project administrator can assign weights to their votes — for example, a teacher’s opinion might be given more weight than that of a student.

“It’s really hard to get people to participate systematically in decision-making without a set way to do it,” Hollands says. “People have different schedules, and if you email them, they often don’t get around to responding. But with DecisionMaker, they can log on at their convenience.” 

DecisionMaker itself was created with a great deal of stakeholder involvement. A team led by Hollands spent three years working with TC’s Summer Principals Academy (SPA) to create the tool and an additional year field-testing and refining it with a state education agency (the Kansas State Department of Education), three school districts (Houston Independent School District, North Carolina’s Wake County Public School System, California’s Fresno Unified School District) and three schools (the 21^st Century Charter School in Gary, Indiana; North Star Academy Washington Park High School in Newark, New Jersey; and the New York City Lab School for Collaborative Studies).

Matt Lenard, Wake County’s former director of Data Strategy and Analysis, says that DecisionMaker was useful in evaluating and scaling the program the district had been using to nurture gifted students in a few schools.

Every district in the country could benefit from a tool like this. You always need to contextualize interventions in a way that accounts for cost and impact, but there are so many line items in the budget to disaggregate, and so many people to talk to. Now we can do that, and also set that information alongside more intangible factors.

— Matt Lenard

“Every district in the country could benefit from a tool like this,” says Lenard, whose responsibilities included leading evaluations and randomized trials of different programs for Wake County. “You always need to contextualize interventions in a way that accounts for cost and impact, but there are so many line items in the budget to disaggregate, and so many people to talk to. Now we can do that, and also set that information alongside more intangible factors.”  

Lenard’s involvement with DecisionMaker came at a point when the tool was still in early development, and Hollands and her team were visiting the district to lead stakeholder meetings and input information from those meetings into the tool’s database. Still he’s confident that even a short online tutorial would enable users now to navigate the tool without other help. Meanwhile, Hollands and her team are creating a course, which they will offer online this coming summer, to train educators in applying the decision-making framework and tool to important decisions of their own. For additional information about the course click here.

Another early DecisionMaker collaborator, Olivia Arroyo, a global history and special education teacher at New York City Lab School for Collaborative Studies, has since used the tool to win buy-in for a history curriculum.

“Within schools we are constantly making decisions and often times these problems get pushed to the side because a consensus is never made due to the lack of time,” Arroyo says. “This tool allows the conversations to continue after the meeting by letting stakeholders add their input after the in-person meeting is over.”

— Joe Levine