Putting it together

I was really surprised by the article that I ended up reviewing because it only tangentially referred specifically to reading assessment and it was rooted more in the field of psychology. The article was surprisingly accessible, but more importantly it helped to focus my attention on the process of reading. That in turn helped me to identify what I truly hoped to measure in the assessments that I plan for my reading and writing students.

As I mentioned, the assessment that I use currently would be considered holistic in that it’s contextualized and asks students to complete a task that is authentic. They pre-read an article, then summarize it. I use the annotations they make during pre-reading to assess their prereading/reading skills and their summaries to assess their comprehension. The problem that I faced with using summaries is that it depends entirely on their writing skills. I didn’t grade for grammar or spelling in their summaries, but rather how well they were able to write about their understanding of the main idea. This dependence on writing made me somewhat wary of how valid the test was in terms of giving me information about their reading comprehension.

I have all of these books on reading and they all offer a host of options when it comes to assessing reading, but until I read the article about Dynamic Text Comprehension, I don’t feel that I had an appropriate lens with which to evaluate the assessment tools presented in the book. Often the purposes of the assessments are not clearly articulated. In many cases, the creators of the tasks simply assert that they are not traditional as though that were sufficient to make them useful. I want an assessment that provides washback, that is it gives students’ some insight into where they may have a gap in their process. It should also specifically focus on their reading process, though I do understand that writing is a necessary part of reading instruction, I felt that all of my assessments relied on students’ abilities to write which may or may not have a disproportionate effect on their performance.

Ultimately, I want the best of both worlds. My students have many more traditional assessments ahead of them and I want them to be able to maintain those test taking skills that they’ll need to reach the next level of their education. But I want the tests to provide more useful information to me and to them about where gaps in their knowledge or process exists so that we can address them. To this end, I want it to look like a duck, but not to quack like one.

When I went back to look at the assessments available to me, I had a fresh way of looking at them and this is what I decided on. This first comes from Zully G. Tondolo and is included in the book, New ways of classroom assessment.  Students are given a passage with sentences that are in the wrong order and they must first order the sentences, and then they are given a few short answer questions. Here’s an example from the book:

Directions: Number the sentences in logical order from 1 to 6.

A. Devi and the Tree

_____Sometimes she also liked to climb the trees and sit there in her secret place.

_____One of the trees was Devi’s special reading tree.

_____Five hundred years ago, a young girl called Devi lived in a town in the mountains in India.

_____Her family’s house had a big garden.

_____In later life, Devi liked to sit under the beautiful trees in the garden and read a book.

_____Sometimes Devi and her friends had picnics or played games together there.

The content may seem simple, however, I think that is one of the benefits to this task. It can easily be manipulated for the level of my students which varies a great deal from semester to semester. In fact, what I like about this activity is that I can control the content, vocabulary, grammar and other features. Following this activity, I can check in with students to find out why they ordered sentences the way they did and might be able to see what’s helping or hindering them.

The next activity is a little more complicated, but also focuses on reading with little need for students to write. It comes from the same text from a teacher by the name of Patrick Rosenkjar. For this it is important to use a self-contained reading passage, a short article of ten to fifteen paragraphs. Between one-third and one-half of the paragraphs should be paraphrased by the teacher to accurately reflect the main ideas, but should carefully avoid using key words contained in the original. Students must then decide which paragraphs were paraphrased by the teacher.

I particularly like this last assessment because it too can be modified depending on students’ levels and also reinforces other skills that they learn later in the semester. It is also focused on asking students to understand the text in order to complete the task without asking them to take on the additional cognitive burden of writing. Like Amber said in her post, I’m not in the business of reinventing the wheel. The activities are out there, but now that I have a better understanding of what I want to actually test for, I can judge which of these activities will suit the goals established in my own class.

What am I missing?

Rapp, D., & van den Broek, P. (2005). Dynamic text comprehension: An integrative view of reading. Current Directions in Psychological Science. 14(5), pp. 276. http://www.jstor.org/stable/20183043

The primary problem I’ve noticed in all of the literature that I’ve read is that they all point to the use of alternative assessments for reading: portfolios, annotation, summary writing. Part of the problem for me as a TESOL instructor is that some of these alternative asessments don’t address one of my primary concerns. It is possible that students are comprehending more of the text than they are able to communicate through writing, even low-stakes journaling. I know this for several reasons, not the least of which is my own language learning experiences which I described in a previous post. This is particularly true with difficult texts in which the reader uses context to guess the meaning of difficult vocabulary.

I felt slightly vindicated when reviewing a bit of reading I did in English 709. In From Reader to Reading Teacher, Aebersold and Field (2006) discuss challenges face by teachers in designing assessment tools for ESL/EFL students. The question of validity is chief among them, that is, are you truly testing the thing you are indeed seeking to test (176). For example, short answer questions or summary writing could involve testing students’ vocabulary, knowledge of and ability to write using English rhetorical conventions, and/or cultural knowledge. They also support my view that teachers should recognize that students will face a variety of assessments throughout their academic careers, and the use of assessments in the classroom should reflect that variety and prepare students to be successful (177). With this, I felt satisfied that traditional assessments do have a part to play within the context of my student population and the goals of our English for Academic Purposes (EAP) program.

But, how could I design assessments that help prepare students for such traditional assessments while remaining true to my goal of giving students authentic language tasks and opportunities for washback. One problem that presents itself is the competing views of reading (Rap & van den Broek, 2005). One is a memory-based perspective from Gerrig & McKoon in which items a reader processes activates other information, in TESOL we often refer to this as schema or background knowledge. In this view, the process occurs on a subconscious level or during on-line processing (the simultaneous processing you do while decoding). Think about this:

Which is correct to say, “The yolk of the egg is white” or “The yolk of the egg are white?”

As you read this sentence, you activate your egg schema: maybe something like eggs have a shell, a yolk and a white. But the question is designed to distract you from the content by focusing your attention on the grammar. The answer is neither is correct. Go ahead and check. I’ll wait. For more like this click here. If you didn’t need to check your answer then maybe you should stop reading now and head over here.

The competing view originated in the work of Piaget and Vygotsky in the constructionist perspective in which readers are actively engaged in a process of trying to understand the text and resolve their difficulties into understanding. I imagine this might be like trying to read Lewis Carrol’s “Jabberwocky.” We all use context clues, syntax, and morphology to try to construct meaning out of the poem. These are more typical of off-line processing that one does after decoding. It follows the conventions of a valid literary form: poetry. Therefore, there must be meaning to be made from it and we consciously seek to draw out inferences from the text.

I had never considered the two views as mutually exclusive before. However, Rapp and van den Broek note that researchers often focus on one or the other as they seek to control as many variables as possible. Consequently, the research is often skewed in one particular direction. Though this has begun to be superseded by the landscape model, which would take another twelve blog posts to explain. If you’re interested you can read more about it here, here, or here. The landscape model essentially says that the two are complementary and occur in cycles as we read, reflect, and construct, revise and reconstruct meaning from reading.

The authors essentially argue that the memory-based and constructionist approach taken separately are lacking. They combine them in the term Dynamic Text Comprehension (DTC) to account for the fact that the process of reading is recursive and that as we decode, we do indeed experience subconscious on-line activation of schema and that after we read, we then reflect (off-line) on our understanding at which point we make conscious choices that in turn lead us to other on-line processing. At this point, schema that is relevant is strengthened in our processing of the text and irrelevant information is discarded.

The authors do not in fact present any original research to support the claims, however, they do review several studies that have been conducted specifically using this model to account for data that supports both the memory-based and constructionist theories. They go to great lengths to point out that they are not in fact attempting to supplant either of these theories, but rather to unify them into a coherent whole that can be studied in its own right.

They posit that both mechanisms are necessary to account for reading. They classify the on-line processes as cohort activation. This occurs when a concept, and any associated pre-existing knowledge, is activated. The second mechanism which they call coherence-based retrieval simultaneously begins and organizes the information into a coherent structure based on the reader’s expectations. While we typically think of one as occuring on-line and the other as off-line, the authors suggest that the two processes occur simultaneously, though they don’t specifically provide any evidence to support this. A glaring omission considering the on-line/off-line debate that runs through much of the debate. They do cite a number of studies in which this model successfully predicted outcomes better than either of the theories used in isolation.

The article proved to be far more interesting than many of the others that I read specifically about assessment because I realized that I what I was hoping to do was to find a way of writing a test for students that engaged them in the process of reading, while still being contained in a familiar format. Think aloud protocols, reading journals, portfolios, summary writing are all well and good. I do use them to some degree or another in my teaching and will continue to use them for assessment. However, I realized that in understanding this surprisingly accessible article helped me to think about what I can do to make a reading test that is valid, i.e. it actually measures to some degree whether students are successfully moving through the reading process rather than can they write about what they read or eliminate wrong answers. My next step is to identify some ways to apply this new understanding to the types of traditional assessment I’m trying to incorporate into my class.


Aebersol, J. & Field, M. L. (2006). From reader to reading teacher: Issues and strategies for second language classrooms. New York: Cambridge University Press.

Rapp, D., & van den Broek, P. (2005). Dynamic text comprehension: An integrative view of reading. Current Directions in Psychological Science. 14(5), pp. 276. http://www.jstor.org/stable/20183043

Blog # 7 (The Final Showdown…of Reading Rates)

In honor of the gravitas of this momentous concluding blog post, I’d like to begin with a fittingly dramatic analogy: Let’s imagine The Avengers have been assigned a new reading teacher (it could happen – they need to read the manual for the flying fortress, the missives of destructive evildoers and the fine print on exorbitant spandex dry cleaning bills), and you are it. You step in front of your pupils on the first day and immediately recognize this is going to be no easy task: Iron Man, though a capable student, seems mostly interested in reading technical manuals, while Captain America only reads 40s era comic books and The Hulk has problems with becoming frustrated while reading (big problems). What do you do?

   (You guys wanna be able to read your own comics, right?)

This is a situation faced every day (in slightly less dramatic fashion) by reading teachers of every level around the country. And instead of the four or five students that a potential Avengers instructor would be faced with, the average reading teacher has a class of 15, 20, 30 or even more students to deal with. Students who come to reading with a variety of skill sets, interests, motivations and personalities, so where to begin indeed?

In my last blog I found the beginnings of an answer to this dilemma in an article by Mara Sapon-Shevin entitled “Ability Differences in the Classroom: Teaching and Learning in Inclusive Classrooms” . In this article, Sapon-Shevin discusses what she terms “purposive heterogeneity” in classrooms, which basically boils down to the idea of treating student differences as a source of strength rather than a weakness in the classroom. Meaning it does not always have to be a source of frustration – what a concept!

Frustrated teacher

                                  (Me, pre-Sapon Shevin article)

Sapon-Shevin talks about how the diversity of a heterogeneous classroom is actually a much better reflection of what students will face in the real world. She also delves into the mutually beneficial aspects of having stronger students work with weaker ones, as well as the idea of having classes work together on large projects entailing many different, individualized assignments suited for the needs of each student. At the post-secondary level, I imagine this might work something like the structured group discussions which many teachers already use in class. These usually entail assigning a different role to each student, for example facilitator, time-keeper, note-taker etc., so that they remain involved in the discussion and move it forward productively. The same could be true of a larger project or assignment, with the difference that the teacher (or possibly the students themselves, depending on how we wanted to work the assignment), would select the various roles of the students based on their relative strengths or weaknesses. In creating a presentation, for example, one student could be assigned to research sources, another to read them, one other to synthesize them and still another to present them to the class. The advantage of this being that each student could remain productively engaged in the tasks they were assigned while none of them would be either left behind because of difficulty,  or left bored because of their relative aptitude in comparison to other students. We might further structure the assignment so that if some students did work faster than others on their portion, they could be assigned to either conduct further research, or possibly aid the slower students with their assignment. This is just a rough sketch of what this kind of assignment might entail, based on my understanding of it, but in any case I think it’s becoming clear that there are a myriad of ways to approach it, which is perhaps one of the best signs of a pedagogy which is adaptable and potentially useful in the class.

After reading the article, I was intrigued by the ideas that Sapon-Shevin proposed, but of course, being the diligent researcher that I am, I was not satisfied with reading just one article and then putting it into practice. I set out therefore to find other articles which would either corroborate or problematize Sapon-Shevin’s. The next source I looked to was a book by Carol Ann Tomlinson and Jay McTighe entitled Integrating Differentiated Instruction and Understanding by Design.

 I immediately recognized that the two authors were grappling with the same questions as I when reading an early passage which said, “…teachers find it increasingly difficult to ignore the diversity of learners who populate their classrooms. Culture, race, language, economics, gender, experience, motivation to achieve, disability, advanced ability, personal interest, learning preference, and presence or absence of an adult support system are just some of the factors that students bring to school with them in almost stunning variety” Whew! Variety indeed, but this is exactly what I had been thinking of. Moreover, as the title of the book suggests, the authors are adding another element to the mix, that of Understanding by Design, or UbD. According to the authors, this is a method primarily based in curriculum design, whereas Differentiated Instruction, or DI, is focused on instruction. The authors claim that these two methods are not only complimentary but reciprocally necessary – we can’t have one without the other. This is because (as any SFSU Reading/Comp professor can and repeatedly will tell you), instruction is not sufficient in the absence of solid course design, and vice versa.

This was an important concept to be reminded of, and so I read on with a keen eye towards understanding the theory of UbD. The authors illustrate the concept by giving example scenarios from  the classroom of “Mr. Axelt,” a teacher on top of his game when it comes to applying the theories of UbD and DI integration. We learn, amongst other things, that Mr Axelt has a thorough plan to teach the US Constitution to his class that accounts for all of his 32 students, that he plans to have them apply learned knowledge gained in differentiated group work, that he adapts assignments and instruction as the unit progresses, regularly reviews progress, discusses with colleagues and lastly views himself as a learner alongside his pupils. Like I said, the man is on top of his game. In all seriousness though, I found this material, and especially the realistic manner in which it was presented to be highly illuminating and helpful in my research.

                             (Mr Axelt, as I imagine him anyways)

By this point, I was seriously considering applying Differentiated Instruction in my classes, as well as its corollary Understanding by Design. Correspondingly, I found two articles that dealt with the use of DI in class, “Lesson Planning Tips for Different Student Levels” by Dorit Sasson, and another article by Carol Ann Tomlinson entitled What is Differentiated Instruction?”.

Both of the articles begin with a definition of DI, which I was pretty familiar with by this point, and then move on to explain its importance as well as how to use it. Sasson’s article focuses on the idea of having students working on the same activity or project while doing different tasks that are suited to their particular needs. Sasson lists some ideas for reading activities that students could do at different levels. This was interesting for me because in reading through these lists, I realized that many of these things were concepts that I had already been applying in the classroom without acknowledging it, and I suspect this may be true for many other teachers as well. We have all, for example, faced the situation where a group of students reads a passage faster than the rest. What do we do in that situation? Asking those students to lay their heads on the desk and take a nap while the others finish is certainly not an option. So, we engage them, ask them questions about the reading or challenge them to go back and find details to deepen their understanding. This is, I think, in a nutshell the same thing that Sasson is describing, the advantage being that we should recognize that we are doing this and will need to do this so that we can plan accordingly.

The Tomlinson article is in many ways a more simplified version of what she describes in her book, though it helped to see the ideas listed in abbreviated form. In it she gives short list of concepts to consider in the areas of Content (reading materials/reading buddies/group work), Process (tiered activities/personal agendas/variation), Products (rubrics/assignments), and Learning Environment (room environment/routines). I feel as though the article, though certainly not as exhaustive as her other work, would work well as a sort of checklist for teachers who are preparing, implementing, or assessing Differentiated Instruction in the classroom.

And so this final blog draws to a close. I will not say that I have reached a conclusive answer to my original question, as the article on Understanding by Design, a concept I was unfamiliar with before but which could prove crucial, showed me that there is more research to be done. I will say however that what I have found so far has been fruitful and I look forward to trying it out in my classes in the future (be they super-powered or otherwise).


Blog # 8: Portfolios and Rubrics: An Assessment of the Good, the Bad, and the Ugly

Reading back through my blog posts about assessment and student engagement in the reading/writing classroom, I realized I wanted to explore a few questions in-depth.

1)      Is there more to the term motivation than we take for granted?

2)      Can grading be formulated to enhance student motivation in a learning-oriented classroom?

3)      Portfolio grading sounds great, but is it effective for both teachers and students?

Ultimately, I’m trying to find out if there are reasonable, practicable (at the university level) grading systems that do not impinge upon students’ motivation or quash the very idea of learning for the sake of learning. I’ve found, if not an abundance of information, a few articles whose authors attempt to reconcile grading with progressive, rather than traditional, teaching. After a long stint in the stacks of JSTOR and Google Scholar, for which specific search terms do not always yield desired results, I was relieved to find articles that represented all sides of the spectrum. In this final blog, I’m invoking a variety of perspectives, from those of anti-assessment educators like Alfie Kohn, to those who believe assessment practices, such as rubrics, need to be changed to fit progressive curricula, and finally to those who see assessment as contributing to student motivation (however extrinsic).

As I was reading about assessment and student engagement, I realized I needed to conduct a more thorough examination of term motivation. Aside from distinguishing extrinsic from intrinsic, many authors who address issues of student motivation do not offer a specific definition. To my mind, the word implies self-propulsion toward a goal of some sort (I always think of running to, rather than running from, something). All three Dictionary.com definitions include variations of the word “motivate,” which must mean it is difficult, even for lexicographers, to pen a specific meaning. Perhaps, then, there is more to the idea of motivation than we suppose.

For instance, motivation does not always come with positive implications: “It’s remarkable how often educators use the word motivation when what they mean is compliance. Indeed, one of the fundamental myths in this area is that it’s possible to motivate somebody else. Whenever you see an article or a seminar called ‘How to Motivate Your Students,’ I recommend that you ignore it. You can’t motivate another person, so framing the issue that way virtually guarantees the use of controlling devices.” This perspective, offered up by Alfie Kohn in an interview with Ron Brandt (for the Association for Supervision and Curriculum Development), speaks to the vagueness of motivation and the many ways in which we throw the word around. Can’t we be responsible for motivating others? Kohn’s most likely referring to intrinsic motivation, and neglecting to make the distinction. Or perhaps he’s leaving out the distinction with the purpose of implying that extrinsic motivation doesn’t really count for anything (and it really doesn’t, in his book).

But is extrinsic motivation really synonymous with compliance, and is the current institutional focus on assessment ultimately responsible for the dissipation of students’ natural curiosity in the learning process?


Some say no way. In his article, “Intrinsic Versus Extrinsic Motivation in Schools: A Reconciliation,” Martin V. Covington “examine(s) critically the assertion that these (assessment) processes are necessarily antagonistic, such that the will to learn for its own sake is inhibited or even destroyed by the offering of extrinsic rewards.” His goals are realistic: he wants debunk anti-assessment theories, like those propagated by Kohn, and to help teachers figure out how to grade effectively and relevantly without compromising student engagement with learning.

However, Covington acknowledges that certain grading systems don’t work: “In many class rooms, an inadequate supply of rewards (e.g., good grades) is distributed by teachers unequally, with the greatest number of rewards going to the best performers or to the fastest learners. This arrangement is based on the false assumption that achievement is maximized when students compete for a limited number of rewards.” However, he does not put much stock in the “overjustification effect” disseminated by theorists like Kohn, which is the idea that rewards from teachers stifle students’ natural curiosity and intrinsic motivation, promoting instead performance-oriented task fulfillment rather than learning for its own sake. He believes assessment does not have to be damaging, but can actually motivate students, which means that not all extrinsic motivation has a negative impact.

Hmm. That’s going against the grain. I tend to think of Kohn as the radical version of the teacher/philosopher I want to become, so it is difficult to side with the assessors on this point. However, in the territory underlying my dreams and ambitions, where I can see clearly the real challenges and obligations of teaching in a community college or university system, Covington’s ideas make more sense.

Motivation, fuzzy term though it may be, is crucial to both students and teachers in college-level reading and writing classes. Nobody wants to teach a group of bored, distant, back-of-the-room eye-rollers, and no one wants to sit in class day after day with a frustrated teacher. So, if there are ways to give feedback and grades and keep students interested, engaged, and (at least) extrinsically motivated, what are they and how can we get on board?

In my search for student-friendly grading methods, I came across articles touting new and improved rubrics, which led me to wonder whether the rubric (that dirty word!) could actually be part of the answer. Because much of what I’ve read about them emphasizes their ties to directive instruction, I’ve always thought rubrics were basically a reinforcement of the banking method. For a long time, I’ve been under the impression that they work like a narrow list of instructions that lead students to write in prefabricated, unimaginative ways; even worse, teachers like me end up with a stack of near-identical papers that threaten to bore us to death over the weekend. But how well-founded are these assumptions, really? If instruction methods can be changed, why can’t methods of assessment? Perhaps even rubrics can be altered to reflect student-centered assessment principles.

Some educators believe rubrics can be altered to make assessment more student-centered. Heidi Goodrich Andrade’s views on the subject are worth noting: “Research has shown that feedback can improve learning, especially when it gives students specific information about the strengths and weaknesses of their work. The problem is that giving focused feedback is wildly time consuming. A good rubric allows me to provide individualized, constructive critique in a manageable time frame.” Moreover, rubrics, she claims, keep her honest and fair in the grading process and provide students with clear assignment guidelines that encourage responsibility and self-efficacy. Also, she says, “Instructional rubrics allow me to assign more challenging work than I otherwise could.” However, she warns, good rubrics are not a substitute for good teaching.

I believe that in order for assessment to be “good”, or “effective,” it must coincide with student-centered instruction. Good teaching involves caring about students’ success—not just in class but in the world at large—and helping students overcome feelings of failure, even when they are confronted with bad grades. It means not treating students like numbers, and not making them feel as if they’re being judged by the grades they’ve made in the past. Looking toward the future, we have to see our students as capable learners so that they can start to see themselves that way. We cannot achieve that goal using assessment methods that are based on arbitrary standards, just as we cannot (given the choice) perpetuate archaic teaching models that do not reflect our informed beliefs. Right now, there is no assessment system in place that works for all students (if there was, we’d have heard about it!), but it nice to hear that some teachers have managed to give grades and keep students engaged.

The most student-centered approach to assessment I can think of (besides student self-assessment, which is essentially effective but not generally used to determine students’ final grades), is portfolio grading. Portfolios, many experts claim, provide teachers and students with a unique opportunity for collaboration and responsibility-sharing. In the article “Portfolio Assessment: Some Questions, Some Answers, Some Recommendations,” Cindy S. Gillespie asserts, “The major advantage of portfolio assessment is that it allows students to actively participate with teachers in the evaluation process.” Though portfolios mainly have to do with writing, there are ways to use them to assess reading as well. Reading logs and journals can be collected and graded as a portfolio, for instance. In a very memorable literature education class I took, we took that idea a step further and wrote reflection papers on how our reading and writing processes had changed throughout the course of the semester.

Also in Gillespie’s study were many other poignant advantages of using portfolios:

1) They allow students to see how they’ve grown as readers/writers through the semester.

2) They allow students to reflect on reading, writing, and thinking as interrelated processes.

3) Work on portfolios can be done collaboratively through peer review.

4) They promote students’ responsibility for their own learning.

5) They increase student self-awareness and self-esteem.

Sounds good, right? But what are the drawbacks, and how much extra work is it for teachers?

After listening to the opinions of educators who likewise believe that this method benefits students, I can honestly see why very few busy teachers end up implementing it. “According to many of the authors, the greatest weakness of portfolio assessment is the increased workload for the teacher.” Another weakness, Gillespie notes, is that “Portfolios may encourage teachers toward a ‘one assessment tool fits all,’ mentality.”

So how much work do we really want to do? Rubrics are quick, easy, and relatively painless, although they do generate a lot of controversy, considering their traditional alignment with the banking method rather than progressive methods of instruction and assessment. Portfolios evaluate the whole student, his or her progress throughout the semester, and increase metacognition. However, according to Gillespie, they may “present unique data that may be ignored or criticized by school-related constituencies.”

Ultimately, I think the best way to come up with a rock-solid assessment plan is for teachers to figure out that they don’t have to stick with one method if it is not working out! We should be willing to change our methods to fit our instructional principles and beliefs, and not give up because we think we have no options. We do have options. It’s just that there is no one-size-fits-all approach (not that there is one to anything), and that can seem daunting. But there are ways to help students learn—and stay motivated—by giving thorough feedback, holding conferences, and grading as fairly as possible within the extant grading systems, imperfect though they may be. The important thing to remember is that good assessment practices are nothing without good teaching.


  1. http://www.alfiekohn.org/teaching/pdf/Punished%20by%20Rewards.pdf
  2. http://www.canyons.edu/faculty/labriem/Psych101/IntrinsicvsExtrinsicMotivation-inSchools-a-reconciliation.pdf
  3. http://ci443-gallagher.wikispaces.com/file/view/Rubrics+4.pdf
  4. http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_nfpb=true&_&ERICExtSearch_SearchValue_0=EJ525767&ERICExtSearch_SearchType_0=no&accno=EJ525767

Getting more specific about “Assessment”

I started down this road of looking at assessment, but I’ve begun to lean more towards specifically looking at traditional assessments, i.e. exams. Throughout my studies in grad school, I haven’t paid much attention to traditional assessments. Despite the fact that I use them in my teaching, my Prometric gig was where I really learned the jargon and techniques for designing multiple choice assessments. I want to take the holisitic approach that considers the whole student, his or her skills and development over the course of the semester. The diagram above shows some of the features of traditional assessments versus alternative assessments from Brown & Abeywickrama’s Language assessment: Principles and classroom practices (2010). What I would like to do is remind myself of the benefits of each, and then look through the literature to try and find assessments that occupy that steel blue space between the cornflower of traditional and alternative assessments.

I’d like to start by defining some of the terms that are relevant to discussing assessment. The product orientation of Traditional Assessments refers to the fact that students get one shot to do well which is why it also fosters extrinsic motivation. They are taking the test in order to get a grade. This contrasts with Alternative Assessments, portfolios for example or other extensive projects that students work on over time. As they engage in the process, teachers can provide ongoing (formative) feedback and ideally, students become more concerned with the doing of the work rather than focused on the end result. This is also reflected in the practicality, at least for teachers, of traditional assessments. Grade exams once and you’re done, you give feedback (summative) once at the end. Most alternative assessments involve periodic feedback which means students’ work is likely to be discussed, revised, and reevaluated.

Traditional Assessments tend to be more reliable, meaning that if the same test is given to a similar student population then the scores will be roughly the same. You can use the same multiple choice test semester after semester which also adds to the practicality – you know what you’re students’ scores are likely to be and anyone with the answer key can score them. Alternative assessments however tend to reflect authentic tasks that learners will do, texts similar to what learners will read, and the process is (ideally) truer to life. I seem to be making more of an argument for Alternative Assessment, but there is another important way that separates Alternative and Traditional Assessments.

Traditional assessments tend to have disproportionately high levels of Impact. This means that the consequences for failing a traditional assessment can be quite serious. Many of my students are studying in the US because they failed to score adequately on university entrance exams in their home countries. Some must continually show improvement on their TOEFL® scores in order to retain government scholarships that allow them to study here. I will continue to use alternative assessments (portfolios, the reading “mid-term” I mentioned in my last post, and journaling) to provide both formative and summative feedback and evaluations in my class. However, considering how many high impact traditional assessments they have in store for them, I’d like to give them some low-impact, low-stakes opportunities to take some traditional assessments, but only if I can find a way to fit them in with the goals of my class and institution.

I have also noticed that students are incredibly motivated just by the thought of having an exam. To some degree they’ve internalized the extrinsic motivation of getting a good quiz score. And giving them a multiple choice test does seem to give them a cognitive break. As long as you don’t do this to them. →→→→→→→→→→→→→→↑

In my reading and writing class, the first thing students do every day is read and then write about what they’ve read. From my own experience as a language learner, I know that it takes a great deal of mental energy to read and write in one’s L2. Students seem genuinely relieved when they know we have something coming up that they can study for. So what I’m hoping to find is some way to reinforce the work they’re already doing, give them a little cognitive break, keep their test-taking skills sharp and do it with a low-stakes traditional assessment that doesn’t insult their intelligence, or mine, I mean I will have to grade them after all.

With those goals in mind, I’ve rounded up some articles that I think will help me to make the most of students’ motivation to do well on traditional assessments, but using them in a way that reinforces the importance of thinking over choosing the right answer (or eliminating the wrong ones). I’d like to be able to write a test that doesn’t seek to trick them, but still gets at finding out what they know and provides some positive washback that students’ can use to improve their skills. I’ve already mentioned the first text, Brown & Abeywickrama, that I used to focus the aspects of assessment that I felt were important to my goals. Their focus is on TESOL, however, their discussion of the general framework of assessment is relevant:

Brown, D. & Abeywickrama, P. (2010). Language assessment: Principles and Classroom Practices. White Plains, NY: Pearson Longman.

Some articles that I’ll use to explore further:

Afflerbach, P., & Kapinus, B. (1994). Developing alternative assessments: Six problems worth solvingReading Teacher47(5), 420. [This one looks promising as it discusses some of the issues I may run into when trying to tweak Traditional Assessments into performing a little more like Alternative Assessments.]

Antón, M. (2009). Dynamic Assessment of Advanced Second Language LearnersForeign Language Annals42(3), 576-598. doi:10.1111/j.1944-9720.2009.01030.x [This was intersting, though a little far afield from what I’m looking at. I typically only use Dynamic Assessment, in which the teacher plays an active role in the test task, for diagnostics on the first day of class.]

Dennis, D V. (2012). Matching our knowledge of reading development with assessment data. In Using informative assessments towards effective literacy instruction.  (pp. 177). [This looks like it might give me some insights towards better connecting students performance with their location within a stage of development.]

Liu, P., Chen, C., Chang, Y. (2010). Effects of a computer assisted concept mapping learning strategy on EFL college students’ English reading comprehension. Computers & Education, 54(2), 436. [This again is a little far from my search, but it was one of the few that addressed technology which is slowly beginning to have more of an effect on assessment at my institution.]

Rapp, D., & van den Broek, P. (2005). Dynamic text comprehension: An integrative view of reading. Current Directions in Psychological Science. 14(5), pp. 276. http://www.jstor.org/stable/20183043 [This nearly didn’t make it into my list because it is firmly rooted in Psychology, which is rather outside my comfort zone. Though I would recommend reading it, it is surprisingly accessible.]

Smagorinsky, P. (2009). The Cultural Practice of Reading and the Standardized Assessment of Reading Instruction: When Incommensurate Worlds CollideEducational Researcher October. 38: 522-527. doi:10.3102/0013189X09347583. [This looks quite promising because it is recent and also deals with the intersection of traditional and alternative assessments.]

Lit in Comp: Straightening out the Confusion

While designing a freshman comp class for my GTA application, I found myself conforming to the trend of avoiding literature in composition class, yet with no conviction on the value of including or excluding it from comp. Discussion of the Lit in Comp  debate has occurred in none of my reading or comp pedagogy classes but I did one day raise my hand and ask the question: Why is literature now obsolete in comp classes? The teacher straightened up in her chair and said: “It’s politics.”  The topic was opened and closed with the history of the power struggle between literature proponents and rhetoricians. The value in teaching literature in composition class or the value in not teaching it, never got addressed—apparently a tendency also present in the community of English Academia.

Despite the tendency, and also in spite of it, works that try to delve deeper into the pedagogical implications within this debate have begun to appear. I will talk about “A Place for Lit in Freshman Comp” by Gary Tate, “Literature and L2 comp: Revisiting the Debate” by Belcher and Hirvela and “Lit in comp classes: case against” by Francis and Barbara Lide.

I have yet to figure out where I stand in this debate, and the purpose of this research is exactly that. Though many current faculties might be tired of the argument and avoiding it in the classroom, prospective composition teachers are graduating without exploring a question that is of both academic and cultural importance: Can the addition of literature to the required texts of composition classes be of benefit to the students, or would it merely be a distraction and waste of time?

A Place for Literature in Freshman Composition, Gary Tate, 1992.

Gary Tate writes this piece to initiate the missing conversation within the Literature in Composition debate. His stance: “It is time for us to adopt a far more generous vision of our discipline and its scope, a vision that excludes no text”. His opinions are responses to the writings of others in the community and are informed by his taste (admittedly) as well as his long experience teaching composition. His opinions also come from a very wide outlook on education and the English class, one that encompasses all that transcends academia and focuses on the various themes and issues that students encounter and struggle with. “I am convinced that true education, as opposed to training, is concerned with much more than what we find in the various academic disciplines,” he asserts. It sounds a lot like Twain’s “I have never let my schooling interfere with my education.”

Tate speaks from his heart—from an affinity for literature and an enthusiasm for grappling with the issues of the world. He does however underline a salient point: “They should be denied no resource that can help them.”

The statement begs the question “how would literature help them?” setting up the conversation he is inviting.

Literature and L2 Composition: Revisiting the Debate, Diane Belcher and Alan Hirvela, 1999.


In this article, Belcher and Hervela cut the politics and get to the point. Their purpose: to explore the different theories and teacher experiences already archived, in order to find out if literature does indeed offer any pedagogical benefits. In the process of reviewing other articles on the topic, Belcher and Hervela encounter arguments by various scholars making the case that literature can indeed be of particular use for developmental writers, including L2 students.

Basic writing specialists like Bartholomae, Petrosky, Hull and Rose recognized that reading literary text can have triumphant results amongst basic writers:

  • Hull and Rose argued that literature invited “dynamic involvement in generating and questioning knowledge” a skill that Spack agrees is useful in other discourse communities as well.
  • For these basic writers who often turn out to be minorities, the use of multicultural texts can, as Severino (1997) puts it “help students recognize cultural differences and similarities” as well as provide “an antidote to the racial and ethnic strife that afflicts many communities and much of the world ( p. 116).”
  • Winddowson claims that literature promotes “conscious[ness] of the procedures we employ in the understanding of any discourse” (1979, p. 159) by requiring the exercise of interpretation.
  • For Krashen (1983), pleasure-reading which tends to involve light literary texts is “the most natural approach” to language proficiency.
  • Kramsch (1993) explained that the use of literature offers students variety by exposing them to aesthetics, and, narrative forms of discourse, instead of mere information-based reading and writing.

Hirvela and Belcher’s work in assessing the theories of those who have been active in the debate has a particular significance to the TESOL field. TESOL teachers are going into the teaching field without proper education on the benefits of literature to L2 students, and are hence not reaping the benefits. As a result of their illiteracy on this specific subject, TESOL professionals are also uninvolved in further enriching the information pool with their own findings.

The summary of literature advocacy that is provided in this piece also informs L1 English level by suggesting literature as one other means for students’ critical thinking and writing success. Though each of the above arguments are valid, they don’t assert the necessity of an education exclusively in literature. They do however insist on its inclusion to the variety of text genres used.

Literature in Comp Class: Case Against, Francis and Barbara Lide (1993).


Francis and Barbara Lide take a firm stand against the arguments that literature merits being in composition classes. Since literature-using composition classes rely on reader-response writing assignments, they argue, students are in a position where they read one discourse and write in another, wasting the learning-by-imitation chance that exists in a class where the reading and writing are of the same genre. The authors contend that the discrepancy in discourse between the assigned reading and the assigned writing “makes itself felt in the very verb tenses of the writing: narrative literature is written in the past tenses and in the pluperfect; it is written about in the eternal present as well as in rhetorical choices, vocabulary, concepts, etc…, making the help of literature rudimentary to non-existent (112).” Lide and Lide emphasize that literary discourse is relevant in few professional fields and academic disciplines, thus failing to be relevant in much of students’ lives outside of the English class. The authors urge curriculum makers to reevaluate the benefits of literature vis-a-vis composition when allotting the limited time in the composition course to literary texts.

*           *

The articles I have reviewed here are not based on statistical research or experiments and rely instead on the authors’ and their colleagues’ theories and experiences as instructors. This should in no way undermine their contribution to the debate as they each highlight points that can’t be neglected in the overall consideration of the English class’ purpose and its place in the university. Still, we all ache for the correct and only answer to “should literature be taught in composition class or not?”

The controversy and even antagonism present in this debate are largely due to the variation in people’s view of the purpose of the composition class. The matter seems to lie entirely on a simple difference in value: some are for the arts, some for students’ success in the university and others for the academic pursuit of social equality. Most would agree that students need all three, plus preparation for the professional world. Many would advocate for an exposure in multi-genre literarcies.  The limited time that a freshman year comp course offers prevents teachers from catering to all these educational needs and forces them to choose fewer discourses to focus on . Instructors end up choosing what they are most passionate of, which in my opinion is a perfectly good justification as they are bound to teach best what they are most interested and best versed in. However, when making their choices, instructors should ask themselves: At what cost am I choosing what I am choosing? What is my pedagogical priority? What is the purpose of freshman year composition?

We cannot be sending students into the world without professional writing skills. We should actually go through the trouble of making sure they have the crucial and practical skills to write CVs, formal letters, grant proposals, and so on. These skills fall completely in English jurisdiction, as I see it.

We also cannot send students into the world without being confident of their ability to process the world. Teaching the world means teaching applicable literacy and teaching for well-roundedness, which life in our diverse population absolutely requires. It involves teaching the professional discourse, media literacy, social and cultural literacies and art appreciation. Art is a prevalent discourse, special in its ability to invoke beauty.  We can’t allow the art of the written world to eclipse into extinction.

The general education system should be sharing the responsibility of teaching some of these literacies and composition instructors should craft their curriculum avoiding redundancy. San Francisco State University restructured its general education curriculum to require a literature course that is its own entity, alleviating freshman composition’s need to teach written art.  Having segments like media literacy and social/cultural literacies would also help tremendously. If the responsibility to teach critical thinking—which is what these literacy classes amount to—is delegated to General Education, composition teachers would finally get to teach composition—that is, the craft of writing. Additionally, they would have the time to cover more genres of writing, including professional writing, blog writing, persuasive writing, etc…Only in the composition class that teaches composition would there really be an education in discourses—emphasis on the plural.

Assessment vs. Exams

To begin, I feel I should make a distinction between an assessment and a test. I am constantly assessing my students through activities, discussions, etc. In fact, I place more stock in assessing students reading comprehension through short low-stakes writing assignments than in exams. However, I do plan to teach in countries like South KoreaSaudi Arabia or China, that place a high premium on students’ abilities to pass exams. Most of them will have to pass an English proficiency test in some form or another.

In some ways, I would like to help prepare them for passing such exams without simply giving them some generic test-taking strategies, but also because tests are important to my students. I don’t think they should be and I take every opportunity to minimize the relation of exams to their ability to communicate in English. Some of my students with the highest TOEFL® scores are almost incomprehensible when engaging in mundane chit chat. Likewise, some of the lowest scorers could function perfectly well if they didn’t have to write research papers. I stress that test-taking is a skill and their scores mostly reflect how well they take tests rather than how well they communicate their ideas in English. I began to wonder, if I could design a test that could be more useful as an assessment tool rather than simply training students for future tests. What I am looking for is a cross between what I do in my current class and the traditional multiple choice tests that students are familiar with.

First, let me tell you what I do in my current class. I think it is an improvement on multiple choice tests, but is not without its problems. The methods that I have been using so far seem more appropriate for testing students’ retention of the skills that I teach them. For example, the first exam that I give in my reading and writing class consists of giving students a passage to read, asking them to annotate it as they read it for the first time. To show their use of reading strategies, they are explicitly instructed to mark the passage as they read it. For example, students are told to highlight or underline the parts of the text that they read first. Furthermore, they are told to make notes about any questions that come to mind as they read the text. “What does this word mean?” “I don’t understand this.” “Maybe this is important.” These are all things that students are encouraged to make note of during their reading.

The stated goal of this exam is to try and make the tacit explicit; that is, to show the processes that students are going through as they read. You may have noticed by now that during the first part of this exam, no mention is made of the traditional measure of reading assessment: comprehension. In fact, the traditional comprehension questions are no where to be found in this class. Reading comprehension is checked through the second part of the exam where students are asked to write a summary of the article that accurately represents the author’s ideas while (mostly) using the students’ words. All parts of this exam are completed in class although the summary writing occurs on a subsequent day. This allows students the chance to take the article home, read it more deeply, look up vocabulary, and plan their writing.

I feel that the format of this exam succeeds in focusing on reading as a skill rather than focusing on students’ abilities to pull information from an article. This is important for two reasons. First, it provides a model for which students can develop their own reading habits that they are encouraged to carry over into their other classes. Secondly, it discourages the strategies that students have learned to help them succeed at multiple choice tests such as the TOEFL®. I must mention that I have worked as a writer of test items for Prometric, a subsidiary of ETS, the company that designed and maintains the TOEFL®, SAT® and other tests.

     Before I accepted this job, I thought about the implications of working on a standardized test and how I felt about participating in creating an assessment that I feel fails to accurately evaluate students’ skills and abilities. Ultimately pragmatism won out over my idealism and I decided that standardized tests are a part of students’ lives. I wanted to participate in the process so that I could better understand the nature of the demands that students face. What I learned during the process has informed my own practices of assessment in the classroom. I don’t think anyone in our ENG 701 class would advocate using multiple choice tests to assess students’ reading, but I think it worth discussing the reasons why.

First, multiple choice tests are decontextualized. Regardless of whether the questions are about short passages that students read in class or extensive reading done outside of class, the student is only asked about information that fits within the space of a sentence or two. This relies on students’ memory as much as it does reading comprehension. Because the questions are so decontextualized, many things can lead students astray. A key word that appears in the right position, may persuade students to check the wrong answer. In fact, in designing the test we didn’t call them wrong answers, we called them “plausible distractors”. To be sure, the plausible distractors needed to be clearly wrong, but they are designed to confuse students.

To illustrate what I mean, here is an example of a vocabulary item. Please note that ETS, vigorously defends their copyrights so this example isn’t the actual item we discussed, but it does accurately represent the type of item and the issues that were raised in our discussion.

I enjoy watching the sunset over the lake in the afternoon. It’s always so serene at that time of day.

SERENE probably means:





In the above example, the issues raised were primarily with the distractors. Originally, two of the distractors had been nouns, however, reviewers felt that the student population would likely be able to eliminate them based on syntax alone. Then we changed those two distractors to the adjectives FUN and LOUD, but then it was decided that LOUD stood out too much because it didn’t end with a nasal as the three others did. It was thought that students would either select it or disregard it based on this distinction alone. WARM was ultimately selected because it fit the syntactic and phonological patterns, but also because it had the added distinction of introducing slightly more confusion complexity. The stem mentions sunset, the sun is warm, therefore students might select warm. And here’s the logic: this will help us determine if students know the meaning of the word SERENE. Really, I think it tests whether students can spot the wrong answers, rather than testing whether they know the right one.

I’m using the example of a vocabulary question, but the principles involved apply to other types of questions including reading comprehension. The basic idea is that multiple choice questions have a set of strategies that can help students be successful. Test writers know this and so write questions and distractors to confound students as much as to try and measure how much of a reading passage students comprehend. I don’t really know if it’s possible to design a hybrid test that provides students some familiarity with the format, while still truly measuring how well students are reading. Students have a lot of anxiety about the tests in my class, in part because they don’t know what to expect. There is no real way for students to study for them, except perhaps by practicing their reading and summary writing skills. While I want to be ready for the types of writing tasks that they will have to do in the US, I also do want them to improve their ability to score well on proficiency exams because they’ll likely have to take the TOEFL® or similar tests throughout their academic careers. I hope to find some articles that can help me strike a balance.