The content of these pages is developed and maintained by, and is the sole responsibility of, the individual senator's office and may not reflect the views of the Nebraska Legislature. Questions and comments about the content should be directed to the senator's office at firstname.lastname@example.org
Evidence is the foundation of rational decision making. Since governments make decisions that impact the lives of citizens, either positively or negatively, rigorously collected and analyzed data is essential. However, just having mountains of metrics and graphs do not mean they are useful. The improper use of data led Mark Twain to recount that “there are three kinds of lies: lies, damned lies, and statistics”.
Few disciplines are as awash in data as education. A look around the website of the Nebraska Department of Education finds page after page of information about demographics, attendance, salaries, expenditures, and test scores. Despite an extensive investment in time, tax dollars, and human resources at the school district level on up through the state agency, collecting all of this data has led to no improvement in student performance or college readiness. In fact, the current hot topic in policy is “workforce development”, a statewide acknowledgment that a high school diploma is insufficient to equip most young adults with the basic skills needed for entry level jobs.
Much of the information presented on the Education Department website provides a textbook case study in the improper use of data. In his 1954 book “How to Lie With Statistics”, a must read for data nerds like myself, Darrell Huff illustrates the basic principles of statistics by showing common errors used to present data. Simply put, merely applying a quantitative approach or presenting sheer volumes of data is not better than having no data at all. In fact, when the standards of statistics are not followed, misleading data leads to bad decisions. It can also, as Twain’s statement indicates, erode public confidence in the value of evidence.
Tables of numbers can be difficult to read, while graphs easily illustrate differences and trends. When used incorrectly, however, graphs can give the appearance that trends are more pronounced than they actually are. The NDE graph for free and reduced lunches demonstrates this error. The scale on the line graph does not begin at zero, but rather at 34%. The scale ends at 48%. The range of the data is only from 38-46%, a very small change. However, by using an arbitrary and distorted scale on the graph, it gives the casual observer the impression of a much greater change over time. The variation in the state’s 95% attendance rate over 10 years is less than 1.5%, yet the arbitrary scale used on the graph is 70-100%.
Another common abuse of statistics is to demonstrate a correlation between two variables and then imply one variable causes the other. Test scores are frequently plotted against demographic characteristics of students, and the conclusion is drawn or inferred that one factor causes poor learning outcomes. Qualities as diverse as education level of parents, sex, time of day the test was given, household income, race, and dozens of others have a correlation with test scores. Correlation alone does not mean causation. A totally distinct variable that has an impact on the two metrics measured may actually be the cause. Frequently the ranges on these correlation graphs are also truncated as well, visually distorting the data.
Presenting incomplete statistical analysis can also misrepresent reality. Averages without information about the distribution of the data are of little value. Two groups of students may have an identical class average score in the mid-B range. However, in one group half of the class earned A’s and the other half earned C’s, while in the other group every student scored a B. The class average shows no difference between the classes, while in reality the student performance between the two groups is quite different. “Averages of averages”, such as when a school’s average ACT composite score is compared to another, are of even less value representing reality.
Whether by ignorance of statistics or willful misrepresentation of data, the outcome is the same. At minimum, valuable time, dollars, and effort are wasted. Even worse, decisions are made based on bad data that have negative consequences. When our kids and their future are at stake, nothing less than the most accurate, honest approach to evidence is acceptable.