Faulty Chronicle of Higher Education Survey Deserves More Scrutiny

As a person who appreciates insights into the thinking of various actors in higher education, I get excited whenever a newspaper invests its resources in fielding a large survey of college faculty, administrators, students, or those who hire them.

Thus, I was initially excited to see the results from the latest survey conducted by the Chronicle of Higher Education and America Public Radio's Marketplace, on how employers view the skills and talents of recent college graduates.  Reading through the results, one began to think the story contained provocative information on business's view of online education, for example.

But then I got to the story's end, and stopped cold. There lay the description of survey methodology: The findings on these pages come from a survey developed, fielded, and analyzed by Maguire Associates Inc., a higher-education consulting firm, on behalf of The Chronicle and American Public Media's Marketplace. Maguire invited 50,000 employers to participate in the study. Experience.com, a career-services consultancy, helped develop the sample by providing a contact list of employers that recruit recent college graduates. The survey was conducted in August and September 2012. There were 704 responses.

First, that's the sum total of all details we're given about the sample.  We know nothing about the characteristics of the sampling frame or the respondents.  In other words, we have no idea whether these employers are representative of specific sectors, specific states, etc.  And equally importantly, while the author didn't do the math for us, a simple calculation reveals a response rate of just 1.4%.

This is insanely low.  So low, in fact, that I don't think the results should be published at all-- let alone featured on the Website.  And if they are, a major banner is needed proclaiming at the start of the story, "PROCEED WITH CAUTION!"  I said the same thing regarding an Inside Higher Ed survey last year, and that one had a response rate of 7%.

Yes, I understand that surveys are expensive to field and really costly to do well, but at the very minimum the public deserves far more clarity and transparency in reporting on methods than the Chronicle has provided here. In addition, there are clear best practices for fielding web surveys (which I have to assume this was) and it sure doesn't seem like Maguire Associates followed them.  For example, Paul Umbach of North Carolina State wrote a nice piece specifically for those conducting higher education research that lays out the issues point by point. Given that low response rates may lead to sample bias (affecting generalizability), low power, and inaccurate estimations, the researchers should have randomly sampled from the target population only enough people to have sufficient power-in other words, going after a fraction of those 50,000 with solid methods instead of surveying all of them with weak technique.  And the overarching goal should have been to achieve a decent rate of response to reduce the possibility of a non-representative sample-- in this case, the response rate is pathetically low (this is as low as I think I've ever seen) and no information on representativeness is provided.  This doesn't seem uncommon for Maguire Associates-- this 2011 survey report says nothing about their sampling or response rate, nor does this one from 2010, but this one from 2005 and this one from 2008 provide at least some of the information that Chronicle readers deserve.

It's unethical to publish this information like this. So I recommend that the Chronicle do what I asked Inside Higher Ed to do-- take it down! And consider working with a more credible survey firm next time, please.

The MOOC Industrial Complex

These days, every education reform movement seems to generate profit for multiple partners.  Take No Child Left Behind, the latest testing and accountability regime. As many scholars have documented, billions of dollars have flowed to corporations providing the tests, textbooks and "supplementary education services" required by that federal policy.  Advocates say this is appropriate since it means the market is functioning freely to provide high quality services, while critics note that absent government regulation (carefully limited under the law) public goods are quickly becoming private ones.

In recent blogs about MOOCs, I questioned their business model, asking why supposedly cash-strapped universities (like mine) would choose to engage with them when there is no evident monetary return?  I received little response from MOOC advocates on that question. But the answer is becoming increasingly clear.

Many universities have stated that MOOCs are the kind of innovative activity that donors would like to support, and that we would gain new donors if we engaged in creating them.  University foundations, including UW-Madison's, seem quite confident in this-- to the point that they are putting in their own dollars for the initial investments to get MOOCs off the ground.  While it's possible that this is sheer altruism, it's doubtful-- their leaders always expect a return on investment.

Where will it come from?

Why, the industry that supports the MOOCs of course.  For example, today's New York Times  documents the growing testing industry associated with online proctoring, helping colleges "keep an eye" on online test-takers. Unfortunately that article failed to investigate the money involved in this effort, focusing instead on the quality of the services provided to prevent cheating. This is unfortunate, since tracking the dollars created by the work of educators, ultimately benefiting the corporate bottom line is exactly what the "paper of record" should be doing.

What other entities will benefit from the instructional activities provided by public and non-profit universities?  Write in and let's start a list.  And while we're at it, let's think hard about the likelihood that these corporations will eventually constitute those "benevolent new donors" our universities are counting on.


ps. I keep meaning to direct my readers to this excellent commentary in the Chronicle of Higher Education. I highly recommend reading it in full.

Renewing the Commitment

This piece is cross-posted from the Chronicle of Higher Education, where it originally appeared as part of a forum on higher education and inequality. I highly recommend reading the full set of posts contained on the COHE website.

In 1947 the historic Truman Commission called for national investments in higher education to promote democracy by enabling all people to earn college degrees. Subsequent expansion of community colleges, adult education, and federal aid occurred not in the name of economic stimulation but to reduce inequality and further active citizenship.

Those ambitions have been steadily corrupted. Today the Tea Party casts the college-educated as snobbish and fundamentally disconnected. Many four-year colleges and universities reinforce that perception by focusing on rankings and alumni satisfaction rather than affordability and national service. Educators speak about business objectives, maximizing revenue through models that charge high tuition and give high aid to needy students, and using a meritocracy narrative that denies the strong role played by a family's ability to pay. The results are stark: Among high-achieving students, just 44 percent of those whose families are in the bottom 25 percent of annual income attend college, compared with 80 percent of those whose families are in the top 25 percent.

In this new world, market-based solutions increase inequality by design. As David F. Labaree documented in Someone Has to Fail, credentialing has replaced learning, and as a result, many students are derided for being what the authors Richard Arum and Josipa Roksa dismiss as Academically Adrift.

The current model ensures that private returns in the form of high-paying jobs accrue only to some, and it justifies minimal public investment in education.

Corporations act as gatekeepers, insisting that only degrees from elite colleges matter, while refusing to pay higher taxes to adequately support public higher education. It is to their advantage: They benefit financially from the for-profit colleges that are filling a void they helped create. The reality is cruel: Many families now dream the same college dream families always have, but run in place in their efforts to achieve it. Heavy debt even drags some backwards.

Colleges and universities vigorously participate in this process. As they lose state support, public flagships turn inward rather than to their communities, focusing on the self-preservation and pursuit of prestige that led them astray. Private universities help their public counterparts fail by promoting idealistic standards of "quality" and practices (such as offering grants rather than loans) that the public institutions simply can't afford. Too many leaders of public universities make common cause with elite private colleges rather than with their public brethren.

Underneath it all resides a fear-driven backlash against educational and economic opportunities for people of color and the working class. Since the civil-rights movement, and especially during President Ronald Reagan's tenure, a focus on private rights and personal responsibilities has replaced concern for social welfare. Remarkably, Reagan convinced the nation that individuals should pay to achieve the American dream. No president since has managed to combat that narrative. In today's focus on paying for performance and metrics, we hear echoes of President Bill Clinton's efforts to reform welfare by telling poor mothers to work for it.

Such rhetoric is fundamentally un-American. As John Dewey reminded us, sustaining democracy requires that we collectively provide for all children what we want for our own children. Anything else simply isn't fair. The politics of austerity have resulted in a paucity of active citizens pursuing democratic ideals by maintaining and expanding public investments. In that climate, New York State stands out for bucking the trend and promoting public higher education with limited reliance on tuition. So, too, do the university presidents urging President Obama to put into effect maintenance-of-effort requirements—requiring states to finance public colleges at a minimum level—to renew the social compact on which our great higher-education system was built. They are bravely rejecting the claim that private markets hold the key to great public needs. In doing so, those leaders may help bring the country back together.

Straight Up ... Or On The Rocks?

In his Education Week blog "Straight Up", Rick Hess comes to the defense of fellow traveler Naomi Schaefer Riley following her dismissal as a Chronicle of Higher Education blogger. The boom was lowered as a result of NSR's hatchet job, published on the Chronicle's "Brainstorm" blog, of three up-and-coming black-studies scholars. She paints their unpublished dissertations broadly as "left-wing victimization claptrap."

Hess's mounting of the barricades is no surprise as the Right is framing this as a crucifixion driven by political correctness. Ms. Riley's husband, Wall Street Journal editorial writer Jason Riley, is quoted by Hess as saying of his wife's sacking, "The mob rules." Well, there's an independent source. (Also see Mona Charen and Checker Finn for similar takes.)

Sara, my wife, a former Chronicle blogger herself, called for NSR's firing on this very blog. She described NSR's piece as "emotion-laden spewing, a venomous disdainful piece directed at young women scholars of color." Indeed. As a non-higher education expert and non-journalist, but amateur blogger, I perceived NSR's blog post as a screed better suited for a stream-of-consciousness, verbal diatribe on right-wing talk radio or the Sean Hannity show than the virtual pages of The Chronicle.

Hess's defense of NSR is wobbly, or "on the rocks," if you will. First, Hess equates NSR's attacks on junior academics with political protests against an elected official -- Wisconsin Governor Scott Walker. Second, Hess conflates NSR's blog post with scholarly work protected by academic freedom. There is a critical difference between rhetorical flourishes directed at public figures and similar ones directed at private citizens. Such instances are, in fact, treated differently in libel case law, with public figures having a greater burden of proof. "Scholarly concerns for academic freedom" are not incompatible, as Hess suggests, with an opinion that a scathing, personal critique such as NSR's doesn't belong on the pages of a respected media-sponsored blog. Agreeing or disagreeing with her isn't relevant. As the Chronicle editors noted, her post simply did not conform to "journalistic standards and civil tone." Academic freedom, freedom of speech, and the right or privilege to publish a blog or column on a given web site or publication are each very different things.

Conservative blogger and UW-Madison law school professor Ann Althouse offers a refreshingly nuanced take on the NSR affair. She points out that NSR "mocked individual graduate students.... [C]ombining that blogging style with an attack on named, individual students, where you are speaking from a high platform in the established media... that's the problem, and I don't see Riley stepping up and acknowledging it."

That's right. This dust-up isn't much about ideas at all, or freedom of speech, as some have contended. The dispute is fundamentally about journalistic standards in the realm of social media and about the specific personal attacks lobbed by NSR through the Brainstorm blog. The Chronicle and other media outlets should have a higher standard for such blogs -- and if commentators like NSR can't or refuse to meet that standard, they should be replaced by someone that can. If political or philosophic balance is of concern, there are plenty of conservative scholars and thinkers, Hess included, that even on a bad day could more than fill the vacancy created by NSR.

Why the Chronicle Should Fire Riley

It's always been hard to take Naomi Schaefer Riley seriously.  Hers is a brand of politically-motivated "journalism" only considered credible by the likes of people like Bill O'Reilly.  In an effort to expose students to multiple points of view, I recently taught Riley's book The Faculty Lounges in my higher education policy course, and my undergraduates found it laughable.  It's essentially a series of empty claims and distorted facts with little evidentiary basis (seriously, she thinks teacher tenure is a 'main driver' of rising college costs!), much like the little talks she's given around the country promoting it.

So I was a little surprised when the Chronicle of Higher Education hired her to join Brainstorm. But, I thought, perhaps she knows a little more than her book revealed-- and I like to read debates on blogs, so why not bring her on?  A former Brainstormer myself (I blogged from June 2009-July 2010 and resigned only because the time commitment was interfering with raising a new baby), I have long appreciated the freedom the editors provide to bloggers.  But that reign isn't without limits--here's the main caution I was given when I began blogging for CHE: "In submitting postings, you warrant that they are original, do not infringe another’s copyright or proprietary rights, and do not violate any person's right of privacy. You also warrant that your article will contain no libelous or other unlawful material. You agree to cooperate fully with The Chronicle in responding to and defending against any third-party claims relating to your postings."

I took those words very seriously. Before posting, I always asked myself "Is this really appropriate for a large, authoritative venue like CHE?"  If I was being critical of someone else, which I often was, I made certain I respected CHE by assembling all of my facts, linking to citations, and asking someone else to read it over before publishing. And I certainly never aimed to do any harm.  If I had an opinion that I couldn't fully research and prepare a reasonable defense of quickly, I reserved it for my personal blog--read by 100s, not 1000s.

Riley clearly doesn't share my respect for the Chronicle of Higher Education. Why am I not surprised, given her disrespect for academia more generally?  What she wrote this week about Black Studies departments was emotion-laden spewing, a venomous disdainful piece directed at young women scholars of color.  She offered not a single fact on which to rest her case.  She clearly aimed to harm these scholars by calling for the end of their discipline, ridiculing their dissertations, and she did so without even reading their dissertations (e.g. without investigating whether there was any truth to her claims of irrelevance).  The last issue regarding libel--whether she caused actual harm--remains to be determined.  The truth is, it's quite possible: let's ask these women whether Riley has cost them valuable time better spent on their work or whether they are receiving hate mail causing emotional distress.

Admittedly, I'm no lawyer.  But whether or not she really broke her contract by writing something libelous, Riley definitely thumbed her nose at CHE and undermined the paper's credibility, damaging its relationship with scholars nationwide. That's a damn shame.  She ought to be fired for that abuse of power. CHE need not continue to lend her its platform.  Let her go.

Who Gets Paid What?

Today the Chronicle of Higher Education released information on the "Median Salaries of Senior College Administrators by Job Category and Type of Institution, 2011-12. "  Giving concerns about administrative costs contributing to rising tuition-- or at minimum, the difficulty in keeping tuition flat giving the expansion of some administrations, this is data we should all pay attention to.

How does UW-Madison stack up? A quick comparison of the salaries of a sample of 20 or so indicate that overall, Madison pays slightly better than the median salaries at doctoral-grant institutions.  But that conceals some heterogeneity which may deserve further attention.  What jumped out most to me is that while men like Vice Chancellor for Administration Darrell Bazzell, and Vice Chancellor for University Relations Vince Sweeney, and Dean of the Graduate School Martin Cadwalleder earn $30-60,000 more than the median for people in their position, three of the key female stars on campus earn below-the-median salaries:  legal counsel Lisa Rutherford, Vice Provost for Enrollment Management Joanne Berg, and Director of Academic Planning and Analysis Jocelyn Milner.  Education dean Julie Underwood, and former Chancellor Biddy Martin earn(ed) well above the national median, which offset the lower salaries of women overall.

My point in making these comparisons is not to suggest that everyone isn't earning their keep 'round these parts--we are all working hard and making do with less these days--but the comparisons serve as important reminder that we should always, always remain vigilant about the specifics of salary distributions.  I know Madison tries hard to attend to gender equity, and administrators at other colleges and universities should too.  After all,  higher education should lead the way in ameliorating inequality-- not perpetuating it!

Hat-tip to Tora Frank for able research assistance! We used 2010-11 salaries since that's what's available-- and as we all know, raises aren't happening these days...

Squeeze Public Higher Education-- And Watch it Squirm

Make no mistake about it-- a conservative agenda in public higher education, quite similar to the one for public k-12, is steadily progressing across the nation.  The multi-pronged attack includes cutting budgets ("we have no choice-- look at the deficit!"), deriding outcomes ("college is worthless, students are partying"), and applying business models to evaluating success ( c.f. all the efficiency talk).

Today's news is rife with stories suggesting that under attack, public colleges and universities are abandoning their missions, adapting market-centered approaches, fighting with each other, and otherwise jumping ship.

• A new survey reports that 143 public colleges and universities now have differential tuition -- a policy that seems efficient on its face, but may well further stratify opportunities, leaving behind those with the least information and least ability to pay.

• In Florida, a week after announcing substantial budget cuts to state U's, the legislature is pushing for the most elite public universities to jack up their tuition.  Let's create that 1%, why don't we?

• And in California, the Master Plan is nearly dissolved, as fault lines become battles lines and the new cry is "Forget you, state U-- we are better..."

If you weren't alarmed by events last year in Wisconsin, and what they suggested about national trends, you ought to be waking up now.  And assuming you are concerned, come out this week and hear about how public higher education can fight back.  Gary Rhoades is in town, at a visit sponsored by the Wisconsin Academy and the Wisconsin Alumni Association, and he's giving three talks.  Please join us!  The first one is tonight at 7 pm at MMoca.