How to Prevent Creaming in Higher Education Performance Regimes?

One of the most prominent concerns raised about President Obama's proposed performance-based funding plan for higher education is that it could reduce access by encouraging creaming.  In other words, what's to stop colleges and universities from simply raising the bars for entry, tightening their admissions policies, in order to improve graduation rates and lower default rates?

Good question.

I'd like to make a few points and then open this up for discussion.  It's one of the big areas that needs bright minds thinking hard in search of solutions, and I hope you'll jump in with good ideas.  We're going to have to look far and wide for solutions, as we can expect that folks in education probably don't have all the answers.

1. The problem already exists.  The number of colleges raising their admissions requirements over time tells this story.  So let's not pretend like we're creating a new problem. The question is whether we're making it worse.

2. NCLB approached this challenge through the use of value-added modeling.  It didn't work there and it's probably not going to work here either, especially since it's hard to believe that we can possible account for all inputs that are external to college, in order to focus on gains made by the college itself.  Now, I know many people will disagree with me on this, including my former student Robert Kelchen, so be sure to read up on their work on the topic.

3. A weaker version of value-added modeling is risk-adjusted metrics, a regression based approach to accounting for intial student differences when looking at outcomes. I'm not sure this is going to fly either, and the Left doesn't like it since it seems to perpetuate the idea that we should "expect" students from disadvantaged families to do worse in college.  No one actually wants that, and so we try things NCLB-style,  demanding growth in graduation rates for subgroups of students. But that too doesn't prevent colleges from admitting fewer students from a given subgroup.

4. Prohibition of creaming via more metrics.  Let's say we stipulate terms regarding enrollment and admissions, in addition to outcomes.  These may have to be differentiated according to college type.  For example, in order to receive Title IV aid, a college must :

• Enroll at least 100 students who are Pell-eligible (all colleges) and
• Maintain a % Pell that meets or exceeds the state average among high school graduates (all public colleges and universities) 
• Admit at least 50% of Pell applicants (all private colleges and universities)
• OR: Use a lottery process for admission into at least 50% of the entering class

Yes, this means that very small colleges would have to be very diverse in order to participate in Title IV. It also means that all colleges and universities may have to adjust their admissions standards somewhat and change their recruiting practices. Is that a bad thing?  If they don't like it, they simply need to prove their mettle by using a lottery for admission.  Then we can really get a handle on their value-added!

Let's talk this one through further.

5. Prohibition and regulation.  Schools could be selected for an audit based on troubling trends in their admissions data.  If they were found guilty of creaming, they could be put on probation and monitored for a period of time.  If they failed that, they'd be kicked out of Title IV.

We must also ask, how much creaming might be tolerable? If the use of performance standards forced more colleges to help low-income students graduate, while reducing access for some other students, at what point would this become intolerable?


Ok, enough from me-- what are your great ideas?

It's Good to Be an Education Optimist

Once in awhile people ask me how I can continue to call myself an education optimist when the "new normal" is so grim.  They ask, "why do you continue tilting at windmills instead of being pragmatic and accepting the situation?"

The answer is simple: acceptance is unnecessary and defeat is not an option.  I challenge the status quo for the sake of students.

Today, I want to say Take that, windmill!

The University of Wisconsin System just ceded to the demands of students across the State and agreed to cap a tuition increase at no more than 2% for the coming year and eliminate the waiting list for the Wisconsin Higher Education Grant.  This is a stunning reversal, as President Kevin Reilly had been lobbying against students, insisting that no cap was necessary.

What happened?  Well, as I have long insisted, the issue is not entirely about a lack of state funding being provided to higher education but how administrators are spending it.  When the incentives for administrators cause they to advance the interests of institutions over the needs of students, accountability measures are required to prevent that.  UW System just got called out, as an audit just revealed that a $404 million balance from tuition payments in 2011-2012 was leftover, unspent, while tuition was hiked by 5.5%.  SERIOUSLY??? Those cash reserves were being held for "specific planned future activities," according to the System.  Sorry Charlie, no way. That is something you do with appropriations, not tuition.  If you aim to help future students and promote stability, that's a public good, and should be on the public dime. This is an outgrowth of the same mindset that's diminished tuition and pushed students into debt-- the same old public / private benefits nonsense. Honestly, the students should demand NO increase and hold firm on doing it for 2 or more years!

So, here we are-- they said it couldn't be done-- the net price of attending UW System schools will likely stay flat or decline over the next year.  HURRAH!

Second, Minnesota legislators took a major step yesterday towards a tuition cap of their own, as the Senate voted to increase spending on higher education by $263 million in exchange for institutional performance accountability and a freeze on tuition increases at the state's universities.  This is remarkable-- using appropriations to drive down the costs of a college education on behalf of the people of one's state. Gee, whoever thought of that!

So folks, the strategy works. Get out there and insist on budgetary transparency and accountability for our institutions of higher education and simultaneously demand that legislators do whatever they can to drive down tuition.  This is the most effective strategy to reducing student debt in the near term.  We can do it!

Remaking Academia: Improve the Hiring Process

The latest entry in a continuing series here at The Education Optimists


Have you ever sought a job as a professor? Depending on your field and where you’ve applied, it goes something like this:

(1) You send in a letter of interest, a CV, and some publications. Maybe some letters of reference too, or perhaps just contact information for those people. If it’s a teaching institution or a school of education, maybe you’ll also send in a statement of teaching philosophy and some student evaluations.

(2) If the search committee likes what they see in the file, they get in touch. This typically means you’ve published a fair bit, demonstrated that you have some interesting ideas, come from a good graduate program, have very solid letters that say you’re among the very best, can attract grant funding, etc.

(3) Then you either meet with the committee via phone or Skype, or at a conference, or more commonly go to campus. (Sometimes it’s a two-step or three-step sequence, sometimes you just go right to campus.) During the visits, you’ll do a talk about your research (to show how you approach questions, theory and evidence), talk with lots of academics who will ask you about your future research plans and what you like to read and discuss (mainly to see if they think you’re smart and they like you), meet with an administrator or two, talk to students, and maybe give a demonstration of your teaching (e.g. a pedagogical talk).

(4) At the end of the evaluation period, a search committee, or even the entire department, together with the dean, has a set of information about you. It includes a written record of what you’ve done, thoughts about what they’ve heard, some student evaluations on a set of metrics, etc. Then they make their decisions.

Often this results in the offer of a job at a pretty good salary with decent benefits, with a three-year contract, and the possibility of tenure. Or, if you’re lucky, it’s a tenured position—in which case they’ve committed (after a tenure committee does their own review) to hire you “for life.”

This process has long puzzled me for what it omits. And as I listen to heated discussions of ineffective professors and teachers, and watch the advent of a strong debate from k -20 over using metrics to decide who to fire, I have to wonder: why can’t we start instead by using data and standard human resources practices to improve our effectiveness at hiring?

Before I list some suggestions for improvement, let me admit that I have held one academic job for my entire career (which admittedly is just 8 years long). And this area—hiring and evaluation—isn’t the topic of my own research. So I don’t know about every practice used in every college or school, and it’s quite possible some of what I think should be done IS being done—in which case we should get a good census of practices and start evaluating their effectiveness. This is a blog I really hope to get constructive feedback on (yes, more so than usual).

(1) Rethink who does the hiring. Right now prospective colleagues primarily do it. This is good, since they are whom you’ll end up working with and spending time with. They should and must have a role. But those peers were hired because they are talented researchers and teachers, not because they know how to evaluate large numbers of prospective applicants and make terrific judgment calls. Professionalization of this hiring practice is needed, and it must include very experienced people who’ve done hiring in academia for decades. Ideally, they’d be systematically trained in identifying expertise in the competencies academics need to do their jobs very well (see next point).

(2) Bring some additional competencies into the mix. Being a good professor or teacher requires strong time management skills, grit, resilience, ability to respond under pressure, communication skills, drive, ability to implement feedback, performance orientation, inquisitiveness, and cultural competencies as well. Where/how are these being assessed now? Primarily in terms of how much you’ve managed to publish in X time (which doesn’t necessarily tell you how well time was managed since other activities are often sacrificed). There are instruments for measuring such things, and we’re often ignoring them. That’s not good enough. What other competencies predict success in academia? We need to know, and we need to integrate them into hiring.

(3) Lengthen the process. My colleagues will hate me for saying this, but spending only a total of maybe 2-5 days evaluating whether a person should be allowed to teach large numbers of students, enjoy limited campus resources, etc, is far too quick. You need more data and more time to analyze it.

(4) Systematize the evaluation process. We use very superficial forms and often don’t consider the data that result in any sophisticated way. The process of reference and background checks is too personal, political, and idiosyncratic, mainly because people who were never trained to do these checks are in charge!

There’s got to be even more we can do. Sure it has to be a flexible process that can be adapted to public flagships or private liberal arts colleges, as well as community colleges, etc. It also can’t be so expensive as to prevent scaling. And it will need revision and improvement. When’s the last time your department changed how it recruited and evaluated applicants?

The current process skips key steps and fails to assess competencies that when not present, lead to failure and turnover in academia (and k12 teaching). Instead of researching who we should fire, why not focus our attention on improving the hiring process? It seems far more efficient, not to mention equitable and ethical.

ESEA Come, ESEA Go

The chatter among the education cognescenti this week is about what is and what isn't in the bipartisan ESEA draft released by Senate education chair Tom Harkin (D-IA) and ranking member Mike Enzi (R-WY).

Let me repeat my prior contention that, politically, ESEA reauthorization is an issue for 2013 -- not 2011 or 2012. The Republican-led U.S. House is not going to give President Obama any kind of a political victory, despite the solid compromise put forth by the Senate HELP Committee. For that reason, the work currently underway is in part about laying the groundwork for a future compromise, in part a genuine attempt to get something done (despite the House), and in part political cover.

The bill itself represents a sensible step back from a pie-in-the-sky accountability goal of 100% proficiency in favor of annual state data transparency, continued data disaggregation among subgroups, and greater state flexibility over educational accountability. Personally, I am not an accountability hawk and am unswayed by spotty evidence and advocates such as former Florida Governor Jeb Bush who contends that it was Florida's accountability system (rather than its major investment in literacy and other interventions) that fueled student test-score gains. Chairman Harkin nails it by saying that the bill "focuses on teaching and learning, not testing and sanctioning." Amen to that.

Seeing as I have a day job that doesn't allow me to analyze the entirety of 800-page bills, here is my quick take on a few elements in the draft bill:

Positives

• Accountability: Eliminates AYP. Requires states to identify 5% lowest-performing schools and 5% of schools with the largest achievement gaps.
  
• CSR: Tightens up the use of Title II, Part A for class-size reduction to ensure that those dollars are directed at research-based implementation of smaller class sizes. [UPDATE: This could potentially free up some Title II, Part A dollars for teacher professional development and new teacher support.]
  
• Teacher & Principal Training & Recruiting Fund: This Fund would support state & local activities that further high-quality PD, rigorous evaluation and support systems, and improve the equitable distribution of teachers. The bill's language significantly strengthens existing federal policy language regarding the elements of comprehensive, high-quality educator induction and mentoring.

Concerns

• Equitable teacher distribution: The bill would require states to ensure that high-poverty and high-minority schools receive an equitable distribution of the most effective educators as measured by new teacher evaluation systems that must include four performance tiers. Sounds good and fair. But given that teacher working conditions significantly impact an individual educator's ability to be effective in the classroom (and garner a "highly effective" rating [see DC]), wouldn't this just create a massive game of musical chairs and major disruptions in the teaching pool unless a determined effort were mounted to improve the often poor teaching and learning conditions present in high-poverty schools?
  

Good Coverage & Analysis

Alyson Klein - Politics K-12 - Education Week
Joy Resmovits - Huffington Post
Stephen Sawchuk - Teacher Beat - Education Week
The Quick and the Ed (Education Sector)

Live By The Sword, Die By The Sword Redux

A USA Today investigation calls into question "dramatic" improvements in student test scores in select District of Columbia schools due to an "abnormal pattern" of erasures. This occurred during Michelle Rhee's tenure as DC schools chancellor.

Among the 96 DC schools that were flagged for wrong-to-right erasures by the city's testing contractor in 2008 "were eight of the 10 campuses where Rhee handed out so-called TEAM awards 'to recognize, reward and retain high-performing educators and support staff'.... Rhee bestowed more than $1.5 million in bonuses on principals, teachers and support staff on the basis of big jumps in 2007 and 2008 test scores.

In 2008, to her credit, then-DC state superintendent (now Rhode Island education commissioner) Deborah Gist recommended that large test score gains in certain schools be investigated, but as USA Today reported, "top D.C. public school officials balked and the recommendation was dropped."

Such allegations and instances of cheating are not unique to Washington DC of course. In 2010, a New York Times article chronicled erasures in Houston and noted investigations in Georgia (including a criminal probe in Atlanta), Indiana, Massachusetts, Nevada, and Virginia.

This latest development, however, adds a new wrinkle to my 2009 post, "Live By The Sword, Die By The Sword."

Michelle Rhee and other education reform advocates have publicly argued that student performance as measured by test scores is basically the be all and end all....

Student learning, school leadership and teaching cannot be measured and judged good or bad based on a single set of test scores. Test scores must be part of the consideration -- and supporting systems such as accountability, compensation and evaluation must be informed by such data -- but they should not single-handedly define success or failure.

When such huge stakes are placed on a single metric, it raises the likelihood of monkey business. Although it is highly likely this is what occurred in DC, a former employee of DC Public Schools (who tweets as @EduEscritora) makes several smart observations on her blog:

[T]he fact that the number of flagged schools decreased so precipitously from 2008 to 2009 is encouraging, even if we don’t know why that happened.

The decreasing number of schools also doesn’t support the claim that the pay-for-performance system now in place under IMPACT has resulted in cheating; 2010 was the first year that IMPACT existed, and that had the fewest number of flagged schools out of the three years in the study and the fewest number of schools with over 50% of the classrooms flagged – only two!

The problem for an advocate like Michelle Rhee is that she has chosen to largely define success based on a single metric: the test score. If many of these DC test-score gains turn out to be illusory and succumb to what some are calling the "Erase To The Top" scandal, it may spell further trouble for Rhee as a spokesperson for the school reform movement. (Rhee has claimed the largest NAEP score gains in the nation under her leadership, although other analyses have shown that increases began and were larger under Rhee's predecessors.) Her credibility already has been questioned by some as a result of alleged embellishments on her resume about her own teaching record. Without credibility, it is impossible to sell one's wares to anyone but true believers.

From a PR standpoint, this erasure story would seem to call for a measured response that carefully chronicles whatever steps, if any, were taken by DCPS at the time to address the unusual frequency of erasures. Instead, through a spokesperson, Michelle Rhee chose to 'shoot the messenger,' bombastically placing USA Today among the "enemies of school reform." [UPDATE: From the Washington Post's Jay Mathews: "Rhee calls her remarks on test erasures 'stupid'"]

Given Rhee's rhetoric, her policies in DC, and her current focus as head of StudentsFirst (which increasingly appears to be working solely with Republican governors and legislators at the state level), Michelle Rhee has largely pinned her credibility to the test score. If she had chosen to sit on a stool with more than a single leg, she might be sitting more comfortably right now and might not be engaged in a such a precarious and delicate balancing act. No doubt by taking on teacher tenure, she would have made enemies no matter what else she said or did. However, if she touted a more nuanced view of school improvement and student success and didn't poo-poo collaboration, she might not face a growing anti-Rhee cottage industry and her new organization might have had a chance to be a true non-partisan force in education reform.

Becoming Diane Ravitch

Even before Alexander Russo's tweet last week ("I read somewhat [sic] that you should wait at least 30 min between switching sides and diving back into the debate, just like eating & swimming"), I was drafting this blog item about Diane Ravitch and had landed in just about the same place.

I struggle in making a professional assessment of Diane Ravitch's conversion from a Lamar Alexander-era U.S. Assistant Secretary of Education and a No Child Left Behind proponent to chief curmudgeon on all things draped in education reform. Her past explanations about "accumulating evidence" and getting "caught up in the rising tide of enthusiasm" for school choice don't seem to tell the whole story. I'm not suggesting she's insincere, but I just don't understand how she got from here to there.

Don't get me wrong. I find myself in agreement with many of Ravitch's recent statements, especially those about the one-sidedness and rhetorical hyperbole surrounding Waiting for Superman and other education reform PR vehicles, such as NBC's Education Nation. And I think she is right in her efforts to recast what education reform is or should be. So it's not that I think that people don't have the ability to change. It's more about trying to process and understand so fundamental a change that takes someone from being a ringleader for an accountability-driven education system to a few years later being the foremost national critic of educational accountability, charter schools, and business-style approaches to education reform. How could a highly educated person have gotten it so wrong and so immediately reversed herself? Perhaps I should just go and read her book and see if the answer lies within?

Ravitch doesn't make my job of processing her transformation any easier with her misleading tweets and blog posts. On 9/23/2010, Ravitch tweeted about the recent Vanderbilt University teacher merit pay study and its connection to the federally funded Teacher Incentive Fund (TIF):

Vanderbilt U study discredits merit pay so next day USDOE hands out hundreds of millions for...merit pay. Blind to evidence and research.

Really?!? Ravitch is too intelligent not to know that she is engaging in deliberate simplification in support of her apparent stance against differentiated compensation of teachers. With spin like that, she should go run a political campaign. 'Tis the season, after all. Ravitch is engaged in the same kind of hyperbole that she rightly criticizes in what Alexander Russo has taken to calling "reformy types".

The National Council on Teacher Quality provides a wise counterpoint to Ravitch on the merit pay study here:

Good teacher pay strategies are never written in a vacuum: they're part of a well-thought out system of incentives and professional supports designed to attract and keep the best teachers.... First off, it's no surprise that the findings showed no correlation between performance pay and increasing student achievement, meaning that the very premise of the study might be called into question. Performance pay is a reward system designed to send strong signals that the profession honors and rewards results but, perhaps even more critically, it should increase the profession's appeal to individuals who might not otherwise consider teaching, convinced that the profession disdains excellence. It's a silly notion to think that teachers leave their "A" game at home, absent the promise of a little extra pay.

Funded TIF proposals -- and the federal program itself -- are about much more than pay tied to student test scores. Proposals all have a compensation component, but also embed other critical elements such as classroom evaluation, professional development and collaboration. As examples, check out the CLASS Project led by the Chalkboard Project in Oregon, Chicago Public Schools District #299, the National Board for Professional Teaching Standards proposal for the state of Maine and Richmond, Virginia, and several successful Teacher Advancement Program (TAP) proposals, including one in Knox County, Tennessee.

Absolute, steadfast consistency in the face of mounting or available evidence is not my suggested goal here. Blind faith and arrogance are found in too many education advocates and policymakers on all sides of the debate. So, at a certain level, I appreciate Ravitch's conversion. But the credibility of her current positions and statements are, in part, determined by a plausible explanation for that evolution.

As a final thought, I recognize that I've been especially critical of some education reformers and reform ideas as of late (here and here and on Twitter). Given that I place my personal views somewhere in the middle between the most aggressive reformers and the most steadfast defenders of the educational status quo, I only felt it appropriate to share some nagging questions I've had about someone on the opposing side of the debate.

Daily Drivel

It's hard to believe that the Wall Street Journal fancies itself a national newspaper while publishing this largely baseless, political clap-trap on its editorial page:

But national standards are no substitute for school choice and accountability, which are proving to be the most effective drivers of academic improvement.

First of all, to frame education reform as pitting national standards against choice/accountability is ridiculous on its face. It is a false choice. Plus, the Obama Administration's reform blueprint is so much more broad than that. About the only thing that the WSJ editorial gets right is in saying that national standards "won't magically boost learning" by themselves.

Secondly, the WSJ appears to be falling into the "silver bullet" mentality all too prevalent among simplistic education reformers. "Just run schools like a business!" Or, "[INSERT pet approach] is the answer." Yes, we've been down that road before .... small schools, merit pay, open classrooms. The WSJ apparently wants to contribute choice and accountability to the junkyard of spent shell casings.

Third, where is the research evidence to suggest that school choice and accountability should be in the driver's seat? The editorial offers no evidence. The presence of publicly funded vouchers is no panacea. Just look at Milwaukee's experience (here and here). At the recent meeting of the American Education Finance Association, the U.S. Department of Education's senior adviser Marshall 'Mike' Smith offered evidence that rates of gain in student test scores were lower after No Child Left Behind became law than before. We chided Margaret Spellings last year for touting the successes of NCLB on similar grounds. So much for bare-bones accountability.

Does the Wall Street Journal have any editorial standards? Or any shame?

UPDATE: Read Claus von Zastrow's take on this editorial on Public School Insights: "It doesn't pass the laugh test."

Compromise in Central Falls?

The Central Falls (Rhode Island) Teachers' Union, an AFT affiliate, has approached district superintendent Frances Gallo and signaled a willingness to compromise over several reform issues that last month led Gallo and the school board to recommend the firing of the entire staff at Central Falls High School. It appears that this story is still being written. Sometimes dramatic steps are what is needed to achieve compromise.

For more, check out today's Washington Post ...

"I am pleased to reassure the union their place in the planning process," Central Falls Superintendent Frances Gallo said in a statement. She said she welcomes union input in developing "a dynamic plan to dramatically improve student achievement" at Central Falls High School.

Gallo's statement followed an overture Tuesday from the Central Falls Teachers' Union, an affiliate of the American Federation of Teachers. The instructors have offered support for a longer school day, as well as more rigorous evaluations and training, among other steps.

and Providence Journal stories...

Late Tuesday, Central Falls Teachers Union president Jane Sessums made the first move in a news release that said the teachers were willing to embrace a set of reforms that were very similar to changes Gallo initially proposed.

“My heart skipped a beat,” Gallo said after reading Sessums’ proposal. “I thought, ‘They are basically saying they want what we want for the first time, with the kind of assurances I need.’ … This brings the union back with us, in the conversation about meaningful reform. It’s where they should be.”

Less than 24 hours later, Gallo opened the door with a news release of her own, saying she was excited by the prospect of reaching agreement with the teachers.

Education Commissioner Deborah A. Gist, whose order to improve the struggling school sparked the mass firings, said she was encouraged by the rapprochement between the two sides.

“Our focus in everything … is how to ensure the children in Central Falls receive an excellent education,” Gist said, “and that is always going to be improved when all the adults are working cooperatively together.

Related blog posts:

Central Falls Redux (2/26/2010)
Rhode Island District Fires All Of Its High School Teachers (2/24/2010)

Central Falls Redux

I have to side with Rick Hess over Andy Rotherham on the question of whether the mass firing of teachers at Rhode island's Central Falls High School is a portend of things to come. In yesterday's Christian Science Monitor story, Hess calls the situation in Central Falls "a canary in a coal mine." In a blog post yesterday, Rotherham calls is "a bogus trend story."

“This will be a canary in the coal mine,” says Frederick Hess, director of education policy studies at the American Enterprise Institute. Such dramatic moves are likely to multiply as “an increasing crop of no-excuses superintendents and state commissioners” take the view that “it’s essential to clean house” to improve persistently failing schools, he says.

This Rhode Island high school situation sure seems like a bogus trend story. Turnarounds may be a trend but really dramatic moves like this seem pretty anomalous. That whale in Florida killing people seems like a more common trend than schools firing all the teachers en masse. -- Eduwonk

In a Tweet this morning, Alexander Russo sardonically notes that "'mass layoff' sounds so much worse than school 'closing' or school 'turnaround' tho they're all the same thing." Indeed.

This morning word comes from the Providence Journal blog that teachers will appeal their firings. No surprise there.

Related Post: Rhode Island District Fires All Of Its High School Teachers (2/25/2010)

------------

UPDATE: President Obama comments on Central Falls in his prepared remarks before the America's Promise Alliance Education Event on March 1, 2010 (via TWIE, via D_Aarons).

"If a school continues to fail its students year after year after year, if it doesn't show signs of improvement, then there's got to be a sense of accountability. And that's what happened in Rhode Island last week at a chronically troubled school, when just 7 percent of 11th graders passed state math tests -- 7 percent."

Rhode Island District Fires All Of Its High School Teachers

Today's Providence Journal story reports that Central Falls, Rhode Island's "tiniest, poorest city has become the center of a national battle over dramatic school reform." Even the New York Times and the Washington Post have taken notice.

While firing the entire teacher corps at Central Falls High School is a dramatic step, the school board's and superintendent's decision was largely based on the district's track record of very poor student outcomes, the teachers' rejection of a reform plan ultimatum from state Education Commissioner Deborah Gist targeting the state's lowest-performing high schools, and accountability pressures from the federal Education Department. The decision is supported by U.S. Education Secretary Arne Duncan, who recently weighed in on the controversy, applauding them for “showing courage and doing the right thing for kids.” Nonetheless, the impact on individual teachers is great and undoubtedly places their lives into significant turmoil and uncertainty.

Providence Journal (2/24/2010):

Duncan is requiring states, for the first time, to identify their lowest 5 percent of schools — those that have chronically poor performance and low graduation rates — and fix them using one of four methods: school closure; takeover by a charter or school-management organization; transformation which requires a longer school day, among other changes; and “turnaround” which requires the entire teaching staff be fired and no more than 50 percent rehired in the fall.

State Education Commissioner Deborah A. Gist moved swiftly on this new requirement, identifying on Jan. 11 six of the “persistently lowest-performing” schools: Central Falls High School, which has very low test scores and a graduation rate of 48 percent, and five schools in Providence. Gist also started the clock on the changes, telling the districts they had until March 17 to decide which of the models they wanted to use. Her actions make Rhode Island one of the first states to publicly release a list of affected schools and put into motion the new federal mandate.

I expect that this story will be replicated elsewhere. On one hand, dramatic change IS needed in chronically low-performing schools and districts. BUT if educators and prospective educators see the wholesale firing of staff as a likely consequence in such challenging schools and districts, are they less likely to take jobs in such environments? What is the long-term consequence for such schools' and districts' ability to attract and retain high-quality teachers?

Surprised?

Is this really any surprise given an education accountability system that grades performance and issues sanctions based on a single indicator: student test scores?

Hat tip: This Week in Education

Live By The Sword, Die By The Sword?

The problem with Jay Mathews' defense ("Measuring Progress At Shaw With More Than Numbers") of a Washington, DC school principal who did not demonstrate student learning gains at his school after one year is that the principal operates within an accountability system that demands such a result. In this case, both Mathews -- and DC Schools Chancellor Michelle Rhee, as described in Mathews' WP column -- are right not to have lowered the boom on Brian Betts, principal of the DC's Shaw Middle School at Garnet-Patterson, based on a single year's worth of test scores.

The state superintendent of education's Web site says Shaw dropped from 38.6 to 30.5 in the percentage of students scoring at least proficient in reading, and from 32.7 to 29.2 in math.

But those were not the numbers Rhee read to Betts over the phone.

Only 17 percent of Shaw's 2009 students had attended the school in 2008, distorting the official test score comparisons. Rhee instead recited the 2008 and 2009 scores of the 44 students who had been there both years. It didn't help much.

The students' decline in reading was somewhat smaller; it went from 34.5 to 29.7. Their math proficiency increased a bit, from 26.2 to 29.5. But Shaw is still short of the 30 percent mark, far below where Rhee and Betts want to be....

Despite the sniping at Rhee, the best teachers I know think that what happened at Shaw is a standard part of the upgrading process. I have watched Betts, his staff, students and parents for a year. The improvement of poor-performing schools has been the focus of my reporting for nearly three decades. The Shaw people are doing nearly everything that the most successful school turnaround artists have done.

They have raised expectations for students. They have recruited energetic teachers who believe in the potential of impoverished students. They have organized themselves into a team that compares notes on youngsters. They regularly review what has been learned, what some critics dismiss as "teaching to the test." They consider it an important part of their jobs.

That's how it's done, usually with a strong and engaging principal like Betts.

Mathews' take -- including consideration of contextual factors, such as the fact that only 17% of the school's students had attended the prior year and the contention that school turnaround requires more than a single year -- is how the education world should work. Embrace the complexity of learning and trying to measure it! To do so would disallow the use of single-year changes in test scores for making high-stakes decisions about schools and individual school personnel. It would also remove the unrealistic pressure on school turnarounds to bear fruit in a single year. Test scores would be used responsibly in combination with other data and evidence to paint a fuller picture about individual school contexts and inform judgments about school leadership and student success.

But Michelle Rhee and other education reform advocates have publicly argued that student performance as measured by test scores is basically the be all and end all. According to this Washington Post story ("Testing Tactics Helped Fuel D.C. School Gains"), Rhee supports strengthening No Child Left Behind to "emphasize year-to-year academic growth." Such a stance creates a problem for such reformers when they are leading a district and staking their leadership on uncomplicated test score gains. Others will assess their leadership and judge their success by this measure -- an ill-advised one in its simplest form.

I would argue that, in addition to doing the right thing (as happened in this instance), reform advocates and school leaders like Rhee also have a responsibility to say and advocate for the right thing. They have a responsibility to be honest about the complexity of student learning and the inability of student assessments to accurate capture all of the nuance going on within schools and classrooms. While the reformers' challenge of the adult-focused policies of the educational status quo is often warranted, some reforms -- accountability, chief among them -- have been taken too far. Student learning, school leadership and teaching cannot be measured and judged good or bad based on a single set of test scores. Test scores must be part of the consideration -- and supporting systems such as accountability, compensation and evaluation must be informed by such data -- but they should not single-handedly define success or failure.

The complexity as presented by Mathews in his article -- and, more importantly, by existing research (such as by Robert Linn, Aaron Pallas, Tim Sass, and embedded within Sunny Ladd's RttT comments) about year-to-year comparisons of both overall test scores and test score gains -- strongly suggests that educational accountability systems should be designed more thoughtfully than they have been to date, but unfortunately that does not seem to be the direction that policymaking is headed at either the federal or state levels. Part of being more thoughtful is moving away from NCLB-style adequate yearly progress and toward a value-added approach, but thoughtfulness also requires not making high-stakes decisions based exclusively on volatile student data. Do I hear "multiple measures"? Sure, but Sherman Dorn offers some provocative thoughts on this subject in a 2007 blog post.

With regard to educational accountability, policymakers first should do their homework -- and then they clearly have more work to do in creating a better system and undoing parts of the existing system that aren't evidence-based and accomplish only in simplifying a truly complex art: learning.

-------------------

For those of you that have gotten this far, there's a related post on the New America Foundation's Ed Money Watch blog discussing a new GAO report that analyzes state spending on student assessment tests -- $640 million in 2007-08.

The increasing cost of developing and scoring assessments has also led many states to implement simpler and more cost-effective multiple choice tests instead of open response tests. In fact, although five states have changed their assessments to include more open response items in both reading and math since 2002, 11 and 13 states have removed open items from their reading and math tests, respectively over the same time period.... This reliance on multiple choice tests has forced states to limit the content and complexity of what they test. In fact, some states develop academic standards for testing separately from standards for instruction, which are often un-testable in a multiple choice system. As a result, state NCLB assessments tend to test and measure memorization of facts and basic skills rather than complex cognitive abilities.

------------

And here's a new story hot off the presses from Education Week. It discusses serious questions raised about New York City's school grading system.

Eighty-four percent of the city’s 1,058 public elementary and middle schools received an A on the city’s report cards this year, compared with 38 percent in 2008, while 13 percent received a B, city officials announced this month.

“It tells us virtually nothing about the actual performance of schools,” Aaron M. Pallas, a professor of sociology and education at Teachers College, Columbia University, said of the city’s grades.

Diane Ravitch, an education historian at New York University, was even sharper: She declared the school grades “bogus” in a Sept. 9 opinion piece for the Daily News of New York, saying the city’s report card system “makes a mockery of accountability.”

But Andrew J. Jacob, a spokesman for the New York City Department of Education, defended the ratings, even as he said the district’s demands on schools would continue to rise next year....

The city employs a complex methodology to devise its overall letter grades, with the primary driver being results from statewide assessments in reading and mathematics, which have also encountered considerable skepticism lately.

The city’s grades are based on three categories: student progress on state tests from one year to the next, which accounts for 60 percent; student performance for the most recent school year, which accounts for 25 percent; and school environment, which makes up 15 percent.

Mr. Pallas of Teachers College argues that one key flaw with the city’s rating system is that it depends heavily on a what he deems a “wholly unreliable” measure of student growth on test scores from year to year that fails to account adequately for statistical error.