Monday, November 13, 2006

Tennessee Department of Education Is Generous When The Report Card Grades Itself

The release each year by the Tennessee Department of Education of its State Report Card is accompanied by celebration and rhetoric about improving schools, but it’s the educational equivalent of the Detroit Tigers popping the champagne corks after this year’s World Series.

There’s really not much they should be cheering about.

In recent years, DOE has gotten really adept at churning out press releases about the improving school scores in Tennessee, but they’re more about hype than hope. All in all, the students of Tennessee aren’t performing much better than 14 years ago, and in a word, the Report Card is a farce.

It’s one thing to spin the facts. But this is something else altogether.

Government Spin

All of us expect a little spin from government, and to be truthful, all of us like to interpret situations in our own best light, but in this case, the state deliberately misleads the public. After all, surely no one believes – particularly the administrators in Nashville - that almost 90 percent of Tennessee students in the fourth and eighth grades are proficient in math and reading as shown on the TCAP (Tennessee Comprehensive Assessment Program).

A report written earlier this year by Kevin Carey of Education Sector dramatically showed how much our Department of Education is playing loose with the facts. When compared to the other 50 states, Tennessee Department of Education claims that we are among the top 5 in the U.S. in eighth grade math and reading, fourth grade reading and math, and high school reading.

It’s an incredible claim, especially when a more objective national test of student proficiency paints just the opposite picture for Tennessee. In that test – the National Assessment of Educational Progress (NAEP) – Tennessee ranks #40 and its percentage of proficient students is more in the average range of about 25 per cent. In case your math proficiency has been certified by DOE, we point out that this is a difference of about 65 percent

So, how does Tennessee fare so well in its own tests?

It’s Simple In Its Execution

It’s simple, our state lowers its standards to jack up the results. For example, as The Commercial Appeal reported in an outstanding series earlier this year, eighth grade students who answer 40% right in the state’s math test are considered proficient. Just three years ago, they had to answer 51% of the questions right to clear that bar.

At least, it now makes sense why nobody in a fast food restaurant in the state can make correct change these days. They’re getting high marks if they’re only getting 40% right.

But in the interest of fairness, it’s probably unrealistic to expect anything else from DOE. After all, if you were given the power to evaluate your own performance every year, wouldn’t you do whatever it takes to give yourself high marks? Essentially, that’s what happens here, because the much-vaunted No Child Left Behind allows each state to develop their own tests and to define their own levels of proficiency. Faced with loss of federal funding if they don’t make progress under No Child Left Behind, they have strong incentive to massage the results.

To be fair, Tennessee isn’t alone in playing games with the numbers. At least 40 other states are doing the same, which means that No Child Left Behind in the end is the poster child for unintended consequences. Passed by Congress as the way to let the public know if its schools are improving, it does just the opposite by presenting statistics every year that are virtually meaningless if you’re trying to determine if schools are better.

Putting On A Pretty Face

In his report, Mr. Carey puts states on a “Pangloss Index,” named for the character in Candide who, despite all evidence to the contrary, always argued that all was well. On that index, Tennessee’s results rank it as the 11th best state in student achievement when compared to the other 49 states.

Meanwhile, NAEP ranks Tennessee as #40 in student achievement, and more disturbing, since 1992, test scores have been relatively been flat except for fourth grade math.

Since 1994, fourth grade reading scores have moved all the way from 212 to 214; fourth grade math scores have climbed from 211 to 232; eighth grade reading scores have moved a grand total of one point, from 258 to 259; and eighth grade math has gone from 259 to 271.

Curiously, the state Department of Education doesn’t schedule any press conferences to announce these scores, which come from the only national student test that allows us to actually compare students’ performance across state lines.

The Time For National Standards

All of this begs the question of why we don’t have a national standard that allows us to have comparables as part of No Child Left Behind, but in the interest of states’ rights, when the federal law was passed, each state was given the power to interpret their own standards and progress.

As Memphis City Schools Superintendent Carol Johnson said in The Commercial Appeal series: “If every state is going to create its own assessments and tools of what is proficient and advanced, then what’s the point? We’ve got to figure out what is proficient as a nation.”

She’s right, because what’s happening now is perpetuating the cruelest kind of hoax on states like ours. At a time when the economy depends on our ability to produce knowledge workers for the new economy, we’re deluded into thinking we’re making progress. By the time that it becomes clear that we’re not, it will be too late, and we don’t know about the rest of Tennessee, but here in Memphis, we simply don’t have time to waste.

As long as Tennessee is able – and most of all, willing – to set the bar low so proficiency is high, the public is given a false sense of security that the people in the Tennessee Department of Education are taking care of business.

Asking The Tough Questions For A Change

Hopefully, now that Governor Phil Bredesen has breezed to victory and says that education will be his top priority in his second term, he’ll ask the tough questions and demand more out of DOE. He prides himself on his experience as a businessman but what businessman, much less governor, could make wise decision about investment or success if someone is cooking the books.

Tennessee had its own standards in place before No Child Left Behind was even passed in Washington, D.C., and its stated intent back then was to make sure our schools produced students who could compete with students from Singapore and Hong Kong. Over time, this attitude has eroded, with political spin trumping public accountability.

Some things are so important that they should rise above the normal day-to-day politics in Nashville. Surely this is one of them.

Meanwhile, it may have only been curious to us, but after the fine series written by Ruma Banerji Kumar and Halimah Abdullah in March in The Commercial Appeal, the newspaper’s coverage of the release of the Tennessee Report Card last week could have been used to build momentum on those earlier articles. Surely, the paper’s attention span is longer than eight months. They owned this story earlier this year, and it’s a really important one. Let’s hope they stay on it and continue to point out the need for change. Tennessee sure needs it.


George Lord said...

Dear TJ and Smart City Consulting,

Since we have had numerous conversations over the past year regarding the enormous amount of disinformation revolving around education I am surprised that you were willing to go out on this limb. Statistics and test scores abound in the education arena and there is no more politicized research in the US than that which addresses education issues. It is for this reason that we must be ever vigilant as we look at numbers. You have done a very good job of painting a pessimistic picture of the education changes in Tennessee.

I want to paint a slightly more cautious interpretation of these test scores. In the mean time you might want to read a very interesting book by Dr. Gerald Bracey, Reading Educational Research: How to Avoid Getting Statistically Snookered, Portsmouth, NH, Heinemann, 2006.

Let’s examine the data on the state test T-Cap and see where we are. While I agree there are inherent problems with the manner in which the state has changed the operational definition of proficiency, this is a problem through out the US and is in large part due to the unreasonable demands of the NCLB. While NCLB is a good idea, let’s see if our schools are improving and let’s try to remove the achievement gap between minorities and the white majority, the implementation is punitive in attacking the programs which might result in improvements for minorities. And while this is important to understanding the changes you mention, perhaps we should have this discussion at another time and stick with the test scores for now.

Another problem at the root of your analysis is the idea that somehow all students should score 100% on a test, or at least maybe 60 or 70 %, to be labeled proficient. This is not the nature of standardized tests. Please think back to when you took the ACT or SAT; did you or your children, for that matter, score 100%? Or even 70%? Probably not. There is reason for that. Which also goes beyond the scope of the questions I would like to address here. In fact, to explain that may require an additional 3 to 6 college credit hours in an Advanced Tests and Measures class. Since I know you, I will hazard the statement that you don’t want to do this.

However, without getting into the politics of NCLB and the states rights issues (which you know I don’t support) that brought about the current stupidity of each state having a different set of standards. First I focus on the state test (T-CAP) and changes as reflected in those scores. I will focus on 4th and 8th grade reading and math, as those are the two grades and topics which NAEP tests and will allow me to further the discussion on NAEP at the conclusion of the present comments. Between the 2003-04 school year and the 2005-06 school year we saw changes in the numbers scoring proficient or advanced in both reading and math for both grade levels. This was true in Memphis and for the state.
In the table below you will see that Memphis has improved more than twice as much as the state as a whole in reading for 4th grade kids and more than 570% better than the state as a whole in 8th grade reading. The numbers for math are not significantly different. The point is that we are seeing greater improvement in MCS than is occurring in the State as a whole.

% Prof or above 2005-06
% Prof or above % Change
Grade 4 Reading MCS 63.3% 76% +20%
Tennessee 81% 88% +8.6%

Grade 8 Reading MCS 67.7% 82% +21.1%
Tennessee 82.9% 86% +3.7%

Grade 4 Math MCS 63.3% 76% +20.1%
Tennessee 80% 88% +10%

Grade 8 Math 67.7% 82% +21.1%
Tennessee 82.9% 86% +3.7%

Next let’s look at the National Assessment of Education Progress (NAEP) tests. Even the most zealous promoters of a national test have begun to question the utility of some parts of such a test and have retreated to supporting tests that address Math and Science only (see the extensive work of Dr. Diane Ravitch former deputy secretary of education under Ronald Reagan, of NYU, on this topic). However let’s look at the change in the NAEP test scores over recent history. I have data on NAEP from 1992 to 2005. Let me forewarn all that the changes in the test made in 1988 make earlier trend analysis quite suspect. However, the trend in NAEP has been one of constant improvement, slowed only in the post NCLB days. That is while we have seen a national trend of improvement in both math and reading this trend has slowed significantly since the NCLB legislation was passed; I might point out that Secretary Spellings comments in recent weeks to the contrary are just plain wrong. Some attribute recent slowing to the changes in teaching which have occurred regarding teaching to the test. More time is being spent on math and reading in schools to the detriment overall of achievement of our kids. Yet I stray once more.

The NAEP in Tennessee shows improvement in 4th grade reading and math at a rate higher than that in the US. 4th grade % Proficient or Above on NAEP in Math improved by 211% between 1992 and 2005 while the US as a whole only 94% increase in the % proficient and above. Both numbers are good. In 4th grade reading Tennessee showed a 27% increase in the number proficient or above while the US showed an 11% increase. In 8th grade Math TN showed a 75% increase in the % proficient and above while the US showed a 45% increase. 8th grade reading was the only place where we did not see improvement, in TN there was a 4% decline in the % of 8th graders scoring proficient or above while there was no change in the US as a whole for this group on this test.

In short, more minority kids and special ed kids are being tested than ever before and the number of kids proficient or above continues to increase. While we are not where we want to be, 100% proficient and no achievement gap between groups, we are heading in the right direction. If we can figure out the changes necessary for NCLB to be both meaningful and non-punitive for students we will succeed. This is the major education priority before us and it is no small task.

Finally, let me quote Robert Glaser:

“Many of those personal qualities that we hold dear—resilience and courage in the face of stress, a sense of craft in our work, a commitment to justice and caring in our social relationships, a dedication to advancing the public good in communal life—are exceedingly difficult to assess. And so, unfortunately, we are apt to measure what we can, and eventually come to value what is measured over what is left unmeasured. The shift is subtle and occurs gradually.” Robert Glaser and the National Academy of Education, In Lamar Alexander and H. Thomas James The Nation’s Report Card: Improving the Assessment of Student Achievement, 1987.

Among the things we cannot measure with a test, a partial list:



George Lord, PhD

Smart City Consulting said...

We weren't so much trying to be pessimistic, but as to point out the need for the state to get serious about improving student achievement.

We think Tennessee is at best keeping pace with Southern states, and keeping pace just isn't enough. There's no research that we've seen that suggests that what the state is doing is producing the kinds of results that we need to compete.

The problem we see with so much research into education these days is that it seems that the organizations know what they believe and then they drive the research to support it. At times, it just feels impossible to get the unvarnished facts.

We're frustrated and we know that parents are, because it's their children who are being put at risk as a result. We're not yet willing to thrown out NCLB as a failed experiment (although there are days when it looks awfully close to one), because this is largely supported by the public which is hungry for something that works and some way to feel that someone is in charge. That's the problem with the shifting Tennessee standards. They are perpetuating a fraud on the public by suggesting that things are great, students are proficient and schools are improving.

Again, we just think the governor needs to come to grips with the fact that the rhetoric of the Department of Education does not match up with the results of the Department, and that something has to change. We hope that's what he does in his second term.

Actually, in the end, we didn't include a whole lot of data and scores that make Tennessee's performance look even more bleak. Like you, we know it's a mixed bag, but we take a big step by our own Department of Education officials shooting straight with us.

Thanks for the post and the alternate view.

George Lord said...

Couldn't agree more!

Smart City Consulting said...

Amen and Amen.

Anonymous said...

Over all tennessee needs to improve in all areas. I was born and raised in this state and i think that our education system and scoring compared to other states in the nation is horrible.

Anonymous said...

BMP en PDF Convertisseur
GIF en PDF Convertisseur
PNG en PDF Convertisseur
PDF Creator