I had written an extensive analysis of a potentially fraudulent licensing exam scheme in the field of social work, where the exams appeared to have been invented by processes very different from what their creators claimed. As far as I could tell from the available information, people were being prevented from obtaining and advancing employment in their chosen field on the basis of exams that failed to test what they were supposed to test.
I had previously criticized state bar exams for their contribution to reduced competition and inflated fees among attorneys, while substantially depriving the public of legal recourse to an extent that would be considered absurd in other countries. I had also noticed that rampant licensing schemes were preventing competition, in fields bereft of any burning need for public protection (e.g., hair-braiders and athletic agents).
It now occurred to me that such abuses probably occurred in other fields as well. In that regard, there would be a variety of potential questions to explore. For example:
- Are the licensing, certification, or other credentialing processes valid? That is, do they give employers, clients, and the public a solid reassurance that a person so credentialed is reliably qualified, and that one lacking such credentials is not?
- Are the credentialing processes redundant? For instance, do they impose, as a lawyer might put it, an unconstitutional restraint on interstate commerce — by, for example, requiring newcomers to a state to jump through hoops that prove nothing, for no real reason except to protect the state’s favored practitioners against competitors from elsewhere? Do credentials offered by competing corporations overlap at students’ expense, such that credentialing would be more useful if it were administered by a neutral and effective governmental or nonprofit agency?
- Are the credentialing processes wasteful? An example: higher education, where people are often required to obtain degrees that amount to little more than sheepskins to hang on the wall.
- Do conflicts of interest impair legitimate testing? An example would arise where a credentialing organization charges high fees for its proprietary exams and/or study materials, and then structures its exams so as to produce a high rate of failure and expensive reapplications.
As these questions illustrate, there was no magic bullet. Any credentialing organization, whether governmental, nonprofit, or for-profit, could be at risk of corruption and/or incompetence. One might guess, pending research on the matter, that incompetence would be especially likely in entities that are themselves shielded from effective competition (classically, governments and monopolies), while financial corruption would be most immediately likely in for-profit corporations whose profits are enhanced by abuses in what the credentialing process costs and what it achieves. But these sorts of assumptions were not bulletproof: my social work analysis (above) demonstrated that both corruption and incompetence could perdure at a nonprofit credentialing organization classified as a public charity.
Concerns about incompetence extend to the foregoing question of whether the credentialing process is “valid.” Contrary to popular belief, a survey is not just a bunch of questions strung together and presented to some innocent passer-by, and an exam is not just a string of questions or problems presented to an arguably innocent student. In both settings, there are issues of context as well as content. You can ask someone the most ordinary questions and yet elicit extraordinary anxiety and hesitation — when, for example, you are the lawyer and that person is sitting on the witness stand. Proper testing is complex and expensive. It requires consultation with experts in the field of testing. Otherwise, you might come to discover that you have been measuring something very different from what you thought you were measuring. Or, worse, you may never make that discovery. It is one thing to skip the expert consultation when you are a high school English teacher writing a quiz. It is another thing when you are charging hundreds of dollars for a putatively authoritative exam.
At this point, I did not have time to conduct an open-ended inquiry into the extent of such problems outside of social work. But it did seem that problems like those mentioned above might arise in a number of fields. For instance, one discussion forum contained several posts questioning the validity of credentialing obligations imposed by the National Board for Respiratory Care. As another example, a website suggested that the Council on Naturopathic Medical Education and related state licensing boards were essentially engaged in quack medicine.
Within the computer field specifically, there seemed to be many possibilities for the kinds of problems discussed above. For example, in the Linux world, an old Slashdot article noted that Red Hat prohibited test-takers from offering specific criticisms of its exams. That rule, which is apparently still in place, would be a good way to insure that credentialing examiners would remain ignorant of feedback that could improve the quality of their determinations. Judging from remarks by Sam Marsh, that sort of avoidance of quality improvement could have the effect of unfairly enriching Red Hat through deliberately incompetent testing processes. A webpage offers comparable complaints about Microsoft certification exams — and so does another. There seem to be general debates about such matters — including, for instance, Orland’s Ars Technica piece criticizing Microsoft certification fees, and an argument from Katz, on behalf of the training industry, responding to various complaints about certification.
The concern presented here is simply that abuses can and do occur, within the dominating and sometimes corrupt and inept world of credentialing. This post is intended as a starting point for further discussion and analysis of credentialing in the computer world.