The dot-edu sector isn't yet connecting the dots

A consulting firm has just attempted to do to doctoral universities what US News and World Report famously does to American colleges: use various quantitative data to arrive at easily-understood rankings. Inside Higher Ed has a thorough writeup posted.

The consulting firm's chosen headline for their data (that public universities are falling behind private ones in their productivity of PhD-level research) seems highly debatable for a number of reasons covered in that article. Well down in there an administrator from Arizona State, after blurting the usual silly cliche about how "you can say anything with statistics", actually does a nice job detailing core flaws of the specific data being used here. The response from the consulting firm is quite unpersuasive.

For me though this small tempest fits into a broader storyline about this country's educational institutions and experts. Like charitable foundations, that sector still seems mostly to think that 21st-century America is still happy with its 20th-century social contract. There are lots of signs to the contrary, such as the growing public interest in some way to compare institutions' actual productivity other than just taking their word for it. The educational system's customers, which is everybody, are less and less willing to do that and the sector itself continues to fail to offer any other robust way to measure and compare its own output.

Hence we see the current standardized-testing mania that has infected our primary and secondary schools, which everyone agrees has tons of drawbacks -- but so long as educators offer no practicable alternatives that would measure educational results on a wide scale, the parents will keep voting for politicians who impose standardized testing. Looking at colleges and universities its clear that a similar dynamic is underway: the educators insist that their output cannot be measured objectively and respond to college and university rankings mainly by deploring the very idea.

That "trust us to know what's best for your children" attitude will not wash in today's world, and a good thing too. Moreover this society's expectations about transparency have moved way past what the .edu sector gets -- 15 years ago Congress had to pass a tough federal law just to get universities to admit how many young women were being date-raped on their campuses. If colleges and universities don't feel like getting serious about identifying measurable, transparent, regular ways to document and compare their output, it will be done for them. Slowly, erratically, clumsily and who knows how intelligently -- but it will.

How are we doing?

A few days ago I mentioned Charity Navigator, it's a non-profit devoted entirely to tracking and reporting on non-profits. Its primary intended audience is donors but the information they put together, at impressive scale and depth, is of great interest to anyone working in the field.

An important caveat: of course they haven't solved, any more than most of us have, the problem of measuring the results of our efforts towards our missions. If a non-profit's mission is for youth to become deeply engaged in the natural world, how do we truly measure how much or well that is happening? We know that anecdotes aren't evidence and that measuring such change is harder than counting widgets sold, and that's about all we can yet agree on. In our weaker moments we fall back on witless cliches about "lies, damn lies and statistics" or how "you can make numbers say anything"; hopefully that's just honest frustration getting the better of us. It's the top issue for our sector and tends to dominate discussion at every conference I attend these days.

Wow that was quite a lot of throat-clearing wasn't it...it's a touchy subject. Anyway what Charity Navigator can do is provide some consistent measures of the caliber of our efforts, and that is certainly worthwhile regardless of the above. They seem pretty balanced in terms of putting the spotlight on both the good and bad, such as with their "Top Ten" lists. For instance I'm glad to have worked for #7 on this one; very glad never to have contributed to the follies at any of these. (And anybody on staff at one of these needs to be very concerned!)

The "Hot Topics" section is quite nice, where they highlight charities related to a currently-newsworthy subject; so for example for October, Breast Cancer Awareness month, they provide a one-stop list of all groups working on that. There's lots of good info on each non-profit and it appears to be kept quite up-to-date. It's probably worth searching for the place you work for, and I hope it turns out to be operating as well as you thought!