Do Academic Incentives Appear to Augment Financial Aid Effectiveness, Particularly after Enrollment?

The field of financial aid research is rapidly growing and expanding, which is a really good thing since the reliability and validity of evidence on effects pales in comparison to the magnitude of the national investment in aid.  Policymakers shoot me emails almost daily, asking "how can this be?"

Well, it's expensive and difficult to rigorously examine the impacts of expensive, complicated programs. Financial support for aid research is often difficult to come by; seemingly because at least to some foundations and other funders, "we know it all" about aid already and need to move on.

Expert researchers like Sue Dynarski and Judith Scott-Clayton know better than this, and bother to continue studying financial aid and write comprehensive reviews of existing studies on the topic for the rest of us.  I've relied on Dynarski's work continuously since my career began, and continue to be amazed at her ability to conduct incisive, beautifully executed work year after year.  This morning, she issued not one but two new papers from NBER, both on financial aid.  For that, we owe her and her co-authors quite a big thanks.

With that sincere respect for her work in mind, I want to submit one point of disagreement with one of the new papers. It regards a particularly difficult and controversial issue: whether financial aid ought to be reformed to include academic incentives tied to college persistence, to increase its effectiveness.  The abstract for Dynarski and Scott-Clayton's paper reads "for students who have already decided to enroll, grants that link financial aid to academic achievement appear to boost college outcomes more than do grants with no strings attached." This is not a new statement from these researchers, but the paper reiterates it, reviving the debate in the midst of the Gates Foundation's efforts to rethink aid.

A close look at the evidence presented in this new paper leads me to believe that while this is a reasonable hypothesis, it has just as little empirical support today-- or perhaps even less-- than it did a few years ago when the debate over this issue was especially hot.  Let's review.

On the question of the impacts of strictly need-based grants on college persistence, the authors point to two quasi-experimental studies (one by Eric Bettinger, one by Dynarski) with "suggestive but inconclusive evidence that pure grant aid improves college persistence and completion" as well as one study of a program targeting very high achieving students (Gates Millenium Scholars, studied by Steve DesJardins) that found no effects (unsurprising since their high outcomes were hard to improve upon).

In addition, they point to an early working paper (issued in 2011) from my ongoing experimental study in Wisconsin, which at the time, using one cohort of students, found null effects for a private grant program.  This is the extent of the evidence they display about the impacts of grants with no strings attached.  So it's important to note that our paper was updated and re-released last fall 2012 (not uncommon for working papers, and it did get press coverage) to incorporate findings from four cohorts of students and take into consideration that some students saw real increases in financial aid from the grant program while others did not.  The results suggest that grants with no strings attached increased college persistence by about 3 percentage points per $1,000 -- consistent with Dynarski and Bettinger's estimates of aid's impacts from other programs. Sure, those estimates are derived from a quasi-experimental analysis within our experimental study (not dissimilar to the approaches highlighted in the other studies cited), but if you want to be a purist about it, look at the experimental evidence only. That evidence suggests positive impacts as well, and raises the possibility that program complexity is moderating the impacts (another key theme in Dynarski and Scott-Clayton's research from 2006).

On the questions of the impacts of grants with academic incentives, the authors highlight several studies, with two figuring most prominently.  First, the MDRC performance-based scholarship demonstration. They point to evidence from the first, small experiment in New Orleans, which showed positive impacts.  Then, they suggest that the ongoing replication studies of those scholarships "appear to reinforce the findings of the initial study."  Unfortunately, that comment is outdated.  The latest reports on effects are showing null results. The What Works Clearinghouse issued a Quick Review on the New York City results (published in December 2012) last week, indicating the experimental test of performance-based scholarships in that city produced no detectable effects on college retention (the title of the study is "Can Scholarships Alone Help Student Succeed?" but please note that the study is of performance scholarships-- not scholarships alone).  (Full disclosure: I am Project Director on that WWC contract.) Recent conference presentations from the project reveal similar trends-- very little improvement, if any, resulting from the additional of performance based scholarships. Project director LaShawn Richburg-Hayes has been commendably careful to also point out that these scholarships are delivered on top of aid without strings, and also noting that it's unclear why Louisiana's results haven't been reproduced elsewhere.

Next, Dynarski and Scott-Clayton highlight Scott-Clayton's rigorous dissertation study of a West Virginia program. Her quasi-experimental analysis suggests that in West Virginia, effects of tying aid to performance had positive effects.  However, the most important claim for the argument here-- that effects were stronger when academic incentives operated than when they did not--does not tell us that incentives performed better than "no strings attached" for these students.  The two treatments occurred at different points in college for these West Virginia students, with academic performance required early in college, and no performance required later in college.  In other words, there is a key confound (year in college) compromising the results.

The truth is that the experimental work needed to test the hypothesis that academic incentives tied to grant aid outperform grant aid without strings attached hasn't been conducted.  It's clear that the effects of aid are very likely heterogeneous, and there are numerous variations in how aid is designed and delivered.  For this reason, across-study comparisons are very hard to make. We need to set up a horse race between aid and aid+incentives for a sample of students much like those whom we'd hope to reform aid for-- Pell Grant recipients, most likely.   Only then will we know if academic incentives really add value.  And even then, we won't know why-- without rigorous mixed methods research.

For now, the jury is out, and policymakers who pair academic incentives with need-based aid are flying blind.  They may have other rationales for doing wanting to do this-- some people feel better about distributing money when it comes with strings-- but they shouldn't pretend it's an evidence-based decision.  And like merit aid, it emphasizes the "reward" rather than "compensatory" rationales for distributing financial aid; a political norm-laden shift that probably isn't without consequence.

Wishy-Washy Thoughts on Gates

I'm no Diane Ravitch.  If I were, I'd use this blog to bravely state my concerns about the direction the Bill and Melinda Gates Foundation is heading with educational policy. I'd follow her lead and ask hard, pointed questions about the role that people with money play in driving major decisions in a democracy.

But I won't.  Because while I'm tenured, I am still fearful.  I have receiving more than $1 million in support from the Gates Foundation for my research on financial aid, and I am grateful for it-- and in need of much more.  That's the honest truth.  It's harder and harder to find funding for research these days, and while my salary doesn't depend on it, getting the work done does.

So I won't say all that Diane just did.  Yet I have to say something, and as I wrote recently, I always attempt to do so.

Her questions deserve answers.  And they should be asked of the higher education agenda as well.  Why the huge investment in Complete College America, an outfit that is pushing an end to college remediation unsupported by the work of top scholars like Tom Bailey?  Why the growing resistance to funding basic research in key areas where massive federal and state investments persist absent evidence of effectiveness? Why sink $20 million into performance-based scholarships, based on a single tiny randomized trial in one site?

I'm sure there are good answers out there.  It's not the first time I've asked these questions.  And perhaps unlike Diane, the time I've spent with the Foundation has imbued me with some confidence that there are very smart, well-meaning people inside the place-- people I like quite a bit.  There's also a lot of turnover, and the outfit is a bit gangly in some areas, kinda like a teenager.

Actually, that's exactly it. The Foundation is one heck of a powerful adolescent.  And maybe that's ok, as long as it recognizes its stage in life, and continues to seek expert advice and wisdom.  Adolescents are good at asking questions and not so great at listening. That's something to work on. Places like the William T. Grant Foundation are full-fledged adult foundations who make smart and highly effective investments daily.  I'd love for Gates's ed portfolio to seek advice and hear from them.  It'd make a world of difference.

Have I just torpedoed my own chances for future support?  Well, I guess only time will tell....

Renewing the Commitment

This piece is cross-posted from the Chronicle of Higher Education, where it originally appeared as part of a forum on higher education and inequality. I highly recommend reading the full set of posts contained on the COHE website.

In 1947 the historic Truman Commission called for national investments in higher education to promote democracy by enabling all people to earn college degrees. Subsequent expansion of community colleges, adult education, and federal aid occurred not in the name of economic stimulation but to reduce inequality and further active citizenship.

Those ambitions have been steadily corrupted. Today the Tea Party casts the college-educated as snobbish and fundamentally disconnected. Many four-year colleges and universities reinforce that perception by focusing on rankings and alumni satisfaction rather than affordability and national service. Educators speak about business objectives, maximizing revenue through models that charge high tuition and give high aid to needy students, and using a meritocracy narrative that denies the strong role played by a family's ability to pay. The results are stark: Among high-achieving students, just 44 percent of those whose families are in the bottom 25 percent of annual income attend college, compared with 80 percent of those whose families are in the top 25 percent.

In this new world, market-based solutions increase inequality by design. As David F. Labaree documented in Someone Has to Fail, credentialing has replaced learning, and as a result, many students are derided for being what the authors Richard Arum and Josipa Roksa dismiss as Academically Adrift.

The current model ensures that private returns in the form of high-paying jobs accrue only to some, and it justifies minimal public investment in education.

Corporations act as gatekeepers, insisting that only degrees from elite colleges matter, while refusing to pay higher taxes to adequately support public higher education. It is to their advantage: They benefit financially from the for-profit colleges that are filling a void they helped create. The reality is cruel: Many families now dream the same college dream families always have, but run in place in their efforts to achieve it. Heavy debt even drags some backwards.

Colleges and universities vigorously participate in this process. As they lose state support, public flagships turn inward rather than to their communities, focusing on the self-preservation and pursuit of prestige that led them astray. Private universities help their public counterparts fail by promoting idealistic standards of "quality" and practices (such as offering grants rather than loans) that the public institutions simply can't afford. Too many leaders of public universities make common cause with elite private colleges rather than with their public brethren.

Underneath it all resides a fear-driven backlash against educational and economic opportunities for people of color and the working class. Since the civil-rights movement, and especially during President Ronald Reagan's tenure, a focus on private rights and personal responsibilities has replaced concern for social welfare. Remarkably, Reagan convinced the nation that individuals should pay to achieve the American dream. No president since has managed to combat that narrative. In today's focus on paying for performance and metrics, we hear echoes of President Bill Clinton's efforts to reform welfare by telling poor mothers to work for it.

Such rhetoric is fundamentally un-American. As John Dewey reminded us, sustaining democracy requires that we collectively provide for all children what we want for our own children. Anything else simply isn't fair. The politics of austerity have resulted in a paucity of active citizens pursuing democratic ideals by maintaining and expanding public investments. In that climate, New York State stands out for bucking the trend and promoting public higher education with limited reliance on tuition. So, too, do the university presidents urging President Obama to put into effect maintenance-of-effort requirements—requiring states to finance public colleges at a minimum level—to renew the social compact on which our great higher-education system was built. They are bravely rejecting the claim that private markets hold the key to great public needs. In doing so, those leaders may help bring the country back together.