The Wallethub state school quality rankings that were released earlier this month add to a growing list of such guides. They join those of the Education Law Center, which has ranked state school systems since 2011 using a four-part funding equity model, Students First’s state report cards, and the Brookings Institution Brown Center’s Education Choice and Competition rankings of large urban districts. There are many others, but these four illustrate some of the diversity in both approaches to ranking schools and types of institutions that rank them.

All four suggest to parents and policymakers that their system identifies the highest quality schools. Yet they produce a very disparate set of “best” and “worst” states (and districts). Two of Wallethub’s top three—New Jersey and Massachusetts—are among two of the the three states that Education Law Center also ranks highest: New Jersey, Massachusetts, and Connecticut. In contrast, two of Students First’s top-ranked three, Louisiana and Florida, are among the lowest on Education Law Center’s sufficiency ranking. The Brown Center gives top billing to the New Orleans’ Recovery School District, New York City, and Washington, DC, and Students First also gives DC high marks, while Wallethub has it dead last, behind Mississippi. And Wallethub ranks Louisiana, Students First’s top-ranked state, 48th of 51.

As the new school year refocuses our attention on education and school quality, what are we to make of these conflicting numbers? Can we use the rankings to help us make good decisions, whether as parents or policymakers?

First, it’s critical to understand the factors that go into rankings. Three of these four indices—Education Law Center, Students First, and the Brown Center—employ only education inputs, albeit very different ones. ELC uses a composite of four measures of state funding for schools: funding level/sufficiency (in dollars); fairness/distribution; “effort”; and coverage (a ratio based on the cost of private versus public school). Students First grades states based on the degree to which they have adopted policies in three categories: teacher “elevation” and evaluation; choice measures that they say empower parents; and spending and governance. The Brown Center’s Choice and Competition Index combines 13 measures of access to alternatives to traditional public schools, including charters, vouchers, and virtual schools, and policies like school closure and parental choice that encourage their use. Wallethub’s metric, in contrast, combines a diverse set of twelve factors, including both inputs, like student-teacher ratios, and outputs, such as reading and math test scores, as well as harder-to-categorize metrics like rates of bullying.

Second, to what degree do these factors correspond with evidence of what works in schools—i.e., what drives student achievement? This is a complex and much-debated question, far beyond the scope of this analysis. But certainly there are basic foundations upon which virtually all education experts, including teachers, parents, and students themselves, agree. In order to fulfill their societal goals of ensuring real opportunity for all and protecting our democracy, U.S. public schools must ensure every child access to a solid education. A school must therefore be able to attract, retain, and support a good teacher in every classroom and a competent principal to manage and lead it. Teachers, principals, and students must be supported by staff, from librarians and aides to nurses and counselors, who ensure that all basic needs are met and that school conditions are conducive for teaching and learning. These, in turn, require sufficient funding, whatever the source, to ensure these educational opportunities.

This fundamental benchmark immediately distinguishes the ELC rankings from the other two that are input-based. Education Law Center makes sufficient funding the first, and, arguably, core metric in its four-part report card. In other words, if a state does not, at the very least, ensure that schools have enough money to provide the basics, it fails the test. The emphasis by both StudentsFirst’s and the Brown Center’s rankings on charter schools and other school-choice options assumes that changes within the public school system cannot produce that solid education for all students. They essentially promote an exit strategy for students who lack that option at the neighborhood or community level. In other words, while all the indices seek to solve the same problem, their proposed solutions differ greatly.

Given their different approaches to assessing school system quality, it is not surprising that, as Figure A illustrates, the Students First rankings don’t match with those of either Wallethub or ELC-Funding. If the rankings, which have been standardized to enable comparison, were aligned, the slopes below would be positive. As we see, New Jersey and Massachusetts are among top Wallethub and ELC states, but among the lowest for StudentsFirst, while the reverse is true for Louisiana and Florida.

It is illustrative to see how these rankings play out in terms of state performance on a test that is comparable across states—the National Assessment of Educational Progress (NAEP), known as “The Nation’s Report Card.” As Figure B illustrates, top ELC (and Wallethub) picks Massachusetts and New Jersey have some of the highest average scores, while Louisiana, favored by the Brown Center and Students First, come in near the bottom. Perhaps more noteworthy, however, is that while there are substantial gaps between higher- and lower-income students in every state, low-income students in Massachusetts perform nearly as well as their higher-income peers in Louisiana: Massachusetts education policies make it truly possible for even disadvantaged children to succeed (in Louisiana, it’s tougher for all kids).

Any index can claim to measure school quality. However, if they are to reflect true quality, rankings must promote funding that is sufficient for all kids to have solid schools, and equitable to meet every child’s needs. As the above graphs demonstrate, indices that ignore these fundamental realities offer parents (and policymakers) poor advice on which schools will help their children succeed.