I was studying for the Graduate Record Exam (GRE) General Test, produced by Educational Testing Services (ETS). As discussed in a separate post, the Analytical Writing portion of that exam required an essay on an issue. It also required an evaluation of an argument. That latter task is the subject of this post. (I briefly posted this and several related items in my Improving Higher Education blog, but I believe they belong here instead.)
As shown in that other post, I went through the process of writing up some responses to some of the issues that ETS said might appear on the exam. In that process, there seemed to be a number of questions worth asking about the issue essay assignment. I mean, there were questions about how to prepare for it, but also questions about what its score might signify. These efforts and questions reduced both the time and enthusiasm that I possessed, by the time I turned to the present topic of evaluating an argument.
As with the issue essay, ETS provided a Topic Pool, listing the various arguments that might appear on the GRE. Before looking at specific examples of such arguments, I looked at ETS’s instructions for the argument task. Possibly their most significant statements were these:
- You are not being asked to discuss whether the statements in the argument are true or accurate.
- You are not being asked to agree or disagree with the position stated.
- You are not being asked to express your own views on the subject being discussed (as you were in the Issue task).
Instead, you are being asked to evaluate the logical soundness of an argument of another writer and, in doing so, to demonstrate the critical thinking, perceptive reading and analytical writing skills that university faculty consider important for success in graduate school.
There did seem to be some similarity with the issue essay. Either way, I had to recognize the pros and cons of a viewpoint. In the issue essay, I had to write them out; in the argument evaluation, I had to determine how well the writer had expressed them, or what would be involved in addressing them. But ETS indicated that the use of examples might be helpful, but was not important, for the argument evaluation essay. Generally, the mission here was to critique and investigate, not to make a case.
As in the issue essay, there were sets of standard instructions that would accompany the text being evaluated. These instructions seemed to vary more than those of the issue analysis. A rough summary of the eight sets of standard instructions for the argument evaluation might go something like this: “Discuss the evidence, assumptions, questions, or alternative explanations that need to be considered in evaluating this argument’s strengths and weaknesses.” In other words, the general assignment seemed to be to explain how the item specified in the instructions (e.g., assumptions) would affect the stated viewpoint. Or as ETS put it, ultimately I would have to “evaluate the logical soundness of an argument.”
This could sound a lot like a discussion of whether the argument’s statements are accurate, which ETS (above) said we should not do. The difference seemed to be that we weren’t saying whether the position itself was right or wrong; we were just supposed to say whether the argument provided, in support of that position, was persuasive, and why or why not. If we didn’t like the argument as stated, that wouldn’t necessarily mean it was wrong; someone could still come up with other evidence or arguments in support.
As with the issue essay, ETS provided a sample argument and examples of essay responses that, they said, would deserve a score of 6, 5, 4, 3, 2, or 1. Again, length seemed to be a factor; the lower-scored examples were much shorter than the longer ones. In this case, the six-point essay contained 540 words, while the five-pointer contained 506 and the four-pointer contained only 260. And again, there was precious little difference between the scoring rubrics for the five- and six-point essays. The key variations seemed to be that the higher score required “insightful” rather than merely “generally perceptive” examination, “cogent” rather than merely “clear” development of ideas, “compelling” rather than “generally thoughtful” support for key points, and conveying ideas “fluently and precisely” rather than just “clearly and well.” “Cogent,” defined as compelling or convincing, did not seem to add anything here, and perhaps the same would be true of “perceptive,” which could be defined as insightful. The basic idea seemed to be simply that they would like some essays more than others. Considering the emphasis upon precision, it was ironic that there was such a vague differentiation of a 6 from a 5.
Consider, for example, the evaluation of the essay to which the grader awarded six points. The grader had nothing but positive things to say about that essay. But do we really want to encourage grad school applicants to hold forth confidently on things that they don’t necessarily understand? For example, the last sentence of that essay’s second paragraph said, “Unless the survey is fully representative, valid, and reliable, it can not be used to effectively back the author’s argument.” I would think the more accurate statement would be that a survey drawn from a nonrandom sample would provide less than conclusive support, but may nonetheless yield worthwhile insights, particularly in the absence of other empirical data on residents’ preferences. Having said that, there was a further problem. The argument did not actually say anything about how “the survey” was done. The writer was elaborating upon his/her rank speculations about what the survey asked, whether the sample was representative, and so forth. In other words, s/he devoted a full paragraph — 119 words, or more than one-fifth of the total — to a tangent. It was a line of argument that might have no relationship to the actual situation. Moreover, s/he slipped into referring to “the survey,” in the singular, when the text indicated that there were “surveys,” plural, possibly done using a variety of methods by multiple investigators. Hence, already in the second of five paragraphs, there was tangential reasoning, distortion of the available information, and misstatement of that information’s potential significance.
It seemed that the grading of an essay like this one might be done by professors or doctoral students who were not social scientists and, as such, might be impressed by a show of quasi-familiarity with research concepts, rather than being dismayed that the writer did not seem to know quite what s/he was talking about. Perhaps similar scrutiny of the remainder of the essay would resolve that question one way or the other. My point here is simply that the reader, gushing over the pluses of this essay, made no mention of this. Generally, the grader failed to provide a convincing indication of why this essay would necessarily be superior to one that was both more concise and more accurate. From a test-taker’s perspective, it was nice to know that I might get a good score for a somewhat rambling and wrongheaded rant; but from the perspective of someone concerned about the future of higher education, I had to wonder what kind of student, and what kind of attitude toward evidence, was being favored by ETS.
As in the issue essay, there were dozens of potential topics in the argument topic pool. These, however, were longer and full of specific fact statements. There did not seem to be much chance of boiling them down into a smaller set of general topics. In addition, since I did not plan to write sample essays even for the reduced set of aggregated issue topics that I had developed from the raw material provided by ETS, I certainly was not going to try to write up 61 pages’ worth of essay assignments (as calculated using Microsoft Word). I actually did not plan to even read all of them. Two concerns did emerge, though, as I flipped through some of those topics. First, in some cases these arguments being evaluated would pose a real problem of tedium. Second, as a related point, to perhaps a greater extent than with the issue essays, it did appear that prior — contrary to ETS’s assurances — prior familiarity with the assigned topic could be quite valuable at test time.
The first item in the pool illustrated both of those concerns. It read as follows:
Woven baskets characterized by a particular distinctive pattern have previously been found only in the immediate vicinity of the prehistoric village of Palea and therefore were believed to have been made only by the Palean people. Recently, however, archaeologists discovered such a “Palean” basket in Lithos, an ancient village across the Brim River from Palea. The Brim River is very deep and broad, and so the ancient Paleans could have crossed it only by boat, and no Palean boats have been found. Thus it follows that the so-called Palean baskets were not uniquely Palean.
Write a response in which you discuss what specific evidence is needed to evaluate the argument and explain how the evidence would weaken or strengthen the argument.
My first reaction to this item was that I knew nothing about it, and really didn’t have much to say. At test time, my expected reaction to this sort of assignment would be to get nervous, kiss a score of 6 goodbye, and flail around for a while, writing random gibberish, until some loosely related idea or recollection came to mind, perhaps lending a belated structure or at least some sense of purpose to the exercise. All of these reactions were very different from what I would expect if, instead, my exam presented me with one of the items on bicycle helmets or college degree programs, about which I actually knew or at least believed something. It did seem, in other words, that, because of the topic assigned, some students would be disproportionately advantaged or disadvantaged, not only in terms of what they would know, but also in their comfort and confidence with the writing assignment.
As I thought about it, I decided that what I would do, if confronted by an argument topic that I found truly tedious and/or alien, would be to read it through once and then go back to the start, looking at each part of the assigned text, and proceed until I found something that seemed to relate to the assignment. In the example quoted above, I would start with “woven baskets.” There didn’t seem to be anything in the text about other kinds of baskets, or about bags or other alternatives to baskets, so I decided that first term was not a good place to start my written analysis. Moving on, I came to the baskets’ distinctive pattern. The text didn’t seem to have much on that either, nor on the ensuing contrast between “previously” and “recently” found Palean baskets. But now it seemed that the intended focus of my essay was to be found in the second half of the text. As I reflected on that, I wondered if I should assume that the early parts of an unpleasant essay assignment on the GRE would tend to be just setup for the punch line. So I skipped ahead and looked at the final sentence: “Thus it follows that the so-called Palean baskets were not uniquely Palean.” This, I thought, would be a good starting point for my essay, with a leadoff paragraph indicating that there seemed to be some basis for doubting that Palean baskets were uniquely Palean, but that further information was needed — about the possibility that the Brim River may have been smaller, thousands of years ago, at least in times of drought, and about the likelihood that Palean boats, made of organic materials, would probably not have survived since prehistoric times.
But now that I’ve said that, I still wasn’t sure what else I would say, to fill a half-hour and write a respectable essay. It seemed that what I just wrote, in the final sentence of the preceding paragraph, captured much of the situation, and that took about two minutes to write. Plainly, I would be at a severe disadvantage, in this particular essay, against an archaeology major, who no doubt could go on and on about various techniques and questions that, to him/her, would just pop out of that sample text. What other evidence would be needed, besides information about the river and boats? I could probably think of more, if I sat there and brainstormed for a while; but meanwhile my archaeology colleague at the adjacent desk would be busily writing away, and by that point I would have the additional burden of fighting panic, fearing that I was going to produce one of those three-point essays.
So I looked at ETS’s example of a three-point essay, on the topic of that community survey discussed in the six-point essay (above). It contained only 295 words. The grader faulted the three-point essay for failing to “explain how the survey might have been flawed.” That expectation would tend to favor social science students, with training in research methodology, over literature majors. The reference to a “survey,” in the singular, suggests that this grader, like the other one (above), failed to recognize that there were multiple surveys, and that his/her expectation was therefore, in fact, that the essay writer would take the time to speculate about the many possible forms that such surveys might have taken. Was it the same survey over and over, or were there different surveys, conducted by different investigators? Were they all conducted at the same time of day and year? Did any of the surveys include interviews? Have the results been written up and peer-reviewed? A person could go on and on in this vein, but it is not clear how this would be directly responsive to the text. Why not just say, contrary to the grader’s apparent expectation, that surveys would be likely to have various limitations, and that, in lieu of writing a textbook to elaborate upon that point, it would be helpful to have further information? Judging from the six-point essay, the answer appears to be that, consistent with the impression that this is all a sort of play (see my other post, regarding the issue essay), the writer is supposed to go through the motions of pretending that s/he has extensive knowledge about surveys (even if that’s not really true), and will just be mentioning a few possible concerns (arising, again, from some exposure to survey methodology), as a potentially faux display of the sort of expertise that s/he could demonstrate at greater length, time permitting.
The three-point essay’s grader went on to say that “the response drifts to irrelevant matters” including “the problem of pleasing city residents.” The text itself said this: “In surveys Mason City residents rank water sports (swimming, boating and fishing) among their favorite recreational activities. . . . For years there have been complaints from residents about the quality of the river’s water and the river’s smell.” Plainly, the problem of pleasing city residents is relevant to this text. The grader seems to have been referring primarily to three sentences in the three-point essay:
The real issue is not the residents use of the river, but their desire for a more pleasant smell and a more pleasant sight. . . . If the budget is changed to accomodate the clean up of the Mason River, other problems will arise. The residents will then begin to complain about other issues in their city that will be ignored because of the great emphasis being placed on Mason River.
The grader’s view seems to be that the writer should not have addressed the concern that redirection of funds to river cleanup will generate complaints about underfunding in other areas. In the grader’s words, these remarks “introduce unwarranted assumptions that are not part of the original argument.” The writer facilitated that criticism by professing that this was, indeed, an assumption. It was presumably good that the writer was aware that s/he might be making an assumption; yet that awareness is not graded as a positive attribute.
Substantively, it is not at all clear that the writer committed a major faux pas. Rather, it seems that s/he was seriously dinged for a matter of style. Suppose s/he had instead said that other problems may arise and that residents may complain. Those aren’t assumptions; they are implicit questions, and the six-point essay writer used a lot of them. By the logic of this three-point grader, one could criticize the six-point essay for introducing the unwarranted (and in flat lands, ridiculous) assumption that it would make sense to ask residents about “a hydroelectric dam,” when the original argument makes no reference to any such thing, and for assuming that the text’s reference to “numerous complaints” about water quality could plausibly have come from just “one or two individuals,” when the original argument makes clear that surveyed residents (very likely comprising far more than one or two individuals) want to be able to engage in water sports and presently cannot.
These observations raised the question of whether graduate education (or at least ETS’s grading) had become self-selecting for people who were good at putting on a show of using preferred phrasing and of employing examples and questions in an approved way. In other words, did the graders prefer the six-point essay because they, themselves, were that sort of writer? The three-point essay certainly lacked style, when compared to the six-point essay. But the three-point grader did not seem to think that s/he was being unduly swayed by style. In the context of the GRE, the grader’s criticisms amount to a flat assertion that the three-point writer has vastly inferior capabilities in critical thinking and expression. Style aside, the question posed here — from the perspective of a test-taker who has just quailed at the prospect of having to write an essay on Palea and the Brim River — is whether that assertion of inferiority would stand up after taking into account such background factors as the two writers’ exposure to relevant fields of study (e.g., research methods) and their apparent differences in eagerness to engage in appropriately phrased flights of fancy (regarding e.g., hydroelectric dams). A case might be made that, for purposes of competent graduate work, the scoring was distorted if not actually reversed: verbiage aside, the six-point writer did not acknowledge his/her own implicit assumptions, and seems to have been less cautious about making erroneous statements in writing.
The net message, regarding the possibility that I would have to write an essay on something like Palean baskets near the Brim River, seemed to be that I was supposed to just throw out something, anything, rather than sit there tight-lipped, worried about what I did and didn’t actually know. Donald Rumsfeld’s reference to unknown unknowns came to mind: I couldn’t say what I didn’t know about those damned baskets and such, because I didn’t even know that there were things of type X and Y that one could know about such matters.
But OK. Brim River, Palea, baskets. Just say something. Well, maybe the fricking chimpanzees swam the river and took those baskets along. Maybe the baskets floated across during a flood. Maybe the natives base-jumped across the river, starting from its enormous cliffs on each side. Or, in honor of the six-point writer, maybe God put a hydroelectric dam there at one point, so people could walk across, but then changed his mind and removed all traces of it. Asteroids? Earthquakes? But no, let us not be silly. The original writer is right: except for the possibility that aliens carried those baskets, or that the baskets were actually composed of a ferrous material that was magnetically drawn across the river in a previous epoch, when the Earth’s magnetic poles were in the east and west rather than the north and south, no doubt the passage does accurately convey a problem for the earlier theory, and at this point all we can do is to dress it up with caveats and cautions about how we need more information on means of transportation, about questions as to whether every other native tribe along the river had boats and therefore the Paleans surely did too. Not a bad theory, now that I look at it — but, as I say, it took me ten minutes of screwing around to get to it.
It seemed that, in a pinch, facing a truly inscrutable essay topic, my best strategy might actually be to sit there telling myself jokes about it, making it even more ludicrous than it seemed to be, until the Truth of the matter popped out, whereupon I could commence an earnest inventory of assorted speculative concerns, all phrased very nicely, so as to convey the impression that I actually believed there was something here to talk about. Seen thus, the exercise could be mildly entertaining.
When I googled the question, I saw that other GRE takers had invented all kinds of interesting things I hadn’t thought of — that, of course, humans could have been swimming along with the chimpanzees, for example, or that the river would freeze in winter. These answers seemed obvious, now that I saw them. They might have occurred to me immediately, if I had not been freaking out (as I might well do in exam conditions) about the prospect of writing on a topic like this.
I noticed, when I did that Google search, that a number of the resulting websites were Chinese. It occurred to me that maybe those wacky Asians had already done all the hard work of inventing possible answers to all of those dozens if not hundreds of argument essay topics — that perhaps this was what some of my Chinese doctoral classmates had been doing, when they were spending six months to a year studying for the GRE. Maybe they were memorizing their own preformed answers to hundreds of goofy ETS essay topics. The very thought was dismaying. What a waste. I mean, my own exam was a week away, but I wasn’t thinking that I should have been doing likewise. There was no chance of that. As a native English speaker, with decent writing ability and the likelihood of getting a topic that was not as bad as this one about woven baskets, I had the luxury of believing that I would not score less than a 4 on the essay, and I was willing to risk that, rather than spend months (never mind a year) in that sort of exam preparation.
A number of hits, in the Google search, seemed to refer to this particular argument as Topic No. 037. Thus I discovered the urch.com forum on GRE argument essays, with posts dating back to August 2002; and in that forum, I found a post dating from September 27, 2002, by Erin, who provided a list of what was then “all 242 GRE Argument topics.” Sure enough, a version of this thing about Palean baskets was no. 37 on that list. Elsewhere, I found a 104-page document comprised of (a) 68 pages containing what appeared to be a more recent list of 174 GRE argument topics and (b) 36 pages of relatively brief commentary on (or perhaps sketches responding to) those items, in Chinese. I say this 104-page document appeared more recent because, as in the current ETS list of topics, the Palea item was first, not 37th.
It seemed, in short, that there might be several different lists of roughly 200 topics, phrased in various ways and redundant of one another to an unknown extent, and that people (especially but not only non-native English speakers) might be devoting inordinate amounts of time to assorted processes of reading, responding to, and possibly memorizing aspects of these variously interesting, tedious, significant, or trivial essay topics. The big picture seemed to be that, for some years now, American universities had been telling thousands upon thousands of international students that, if they displayed the kind of determination necessary to master English responses to those ~200 argument essay topics (plus the list of issue essay topics), they would improve their chances of scoring well on this least important aspect of the GRE — and apparently quite a few of them were willing to do that, whereas others (of potentially comparable or superior ability) might not.
It tentatively appeared, then, that ambition and hard work were being substituted, to some extent, for intelligence and ability. A person who had nothing better to do with his/her time, and who could endure the grinding tedium of working through answer sketches for all those argument topics, would stand a good chance of writing a nice, long essay about the river and the baskets, whereas a test-taker who might be more qualified for doctoral research could easily choke up and turn in an inferior performance on this portion of the GRE — and that could be enough to make the difference between admission or rejection at a competitive school.
In that sense, the GRE’s Analytical Writing section consisted of two different tests. For those who did not prepare answers to approximately 250 to 300 issue and argument essay topics in advance, this portion of the GRE was to some extent a test of the ability to compose under pressure. For those who did work through those topics in advance, it was a test of the ability and inclination to invest large amounts of time and effort to game the system — to develop, that is, an edge over the majority of fellow test-takers, based not on one’s ability in the actual material being tested, but rather on his/her confidence that there was nothing better that s/he could be doing with his/her energies and abilities. It was as if ETS had decided to use a new kind of math in the quantitative portion of the GRE — one that used unfamiliar symbols and rearrangements of mathematical concepts, to see which test-takers were desperate enough to spend months learning it. These seemed to be interesting wrinkles in society’s apparent belief that it was sending its most capable people to grad school.
So. I had arrived at some conclusions about the argument evaluation assignment. I was not going to prepare responses to all of those ~200 possible assignments. I was simply not pursuing a PhD in order to become that kind of student. Instead, like many other test-takers, I was going to enter that portion of the exam with some brief exposure to its basic expectations, and with a hope that the assigned topic would provoke enough thoughts to inform a half-hour’s worth of writing. If that hope failed, and I found myself facing a topic of unknown unknowns, my strategy would be to jot down amusing responses, in hopes that this form of cranial stimulation would lead eventually to something putatively intelligent.