FROM SINGLE TO PASSAGES: CONTEXTUAL EFFECTS ON PREDICTIVE POWER OF VOCABULARY MEASURES FOR ASSESSING READING PERFORMANCE


The Writer: David D. Qian
The Hong Kong Polytechnic University

Reviewed by: Firman Parlindungan
The Islamic University of Malang

A. REVIEW
The study presented here evaluated the equivalency of the two forms of presentations of vocabulary items, that is, presenting vocabulary items in isolation and in context, both forms used as part of a reading comprehension test. Here is background of the study why the researcher conducted this research.

Vocabulary knowledge and reading context are among the most important factors involved in reading comprehension. Recent research has confirmed the important role of vocabulary knowledge in assessing second-language reading performance. Recent research on TOEFL has reported strong and positive intercorrelations between vocabulary knowledge scores and reading scores, suggesting that the construct of vocabulary knowledge overlaps considerably the knowledge construct required for reading comprehension.

It is also widely recognized that context plays an important role in helping comprehend a text. Oxford and Crookall (1990) criticized purely decontextualized vocabulary leaning as undesirable because the word is not in a context of communication. Even they know their semantic meanings because those words were learned independent of context.

Based on an involvement load hypothesis for second-language vocabulary learning, Hulstijn and Laufer (2001) expressly pointed out three factors—need, search, and evaluation—are important for learning and retaining new retention outcome. Based on this hypothesis, assessing vocabulary in context, which encourages learning words in context, can be assumed to directly promote beneficial vocabulary learning activities.

However because contextualized vocabulary assessment changes the way the vocabulary items are conventionally presented in a test, the construct of the contextualized vocabulary tests will also differ from that of the traditional discrete-point vocabulary test. in the discrete-point test, the focus of the assessment is, generally on the meaning of single words either in isolation or with very limited context, whereas in a test composed of contextualized vocabulary items, what is assessed is mainly the meaning of target words as they fit the context semantically, grammatically and, more important, communicatively. This change is test construct will naturally result in modification of what is measured and assessed.

There is evidence that traditional discrete-point vocabulary test work well in predicting reading comprehensions abilities, but apart from Henning’s (1991) study, little is known about the ability of contextualized vocabulary test to predict reading comprehension in comparison with the discrete-point test.

A.1 Development of the TOEFL vocabulary subtest: A historical overview
As a test provide information on English proficiency of nonnative speakers intending to study in American universities, the first official administration of the TOEFL test took on February 17, 1964, to candidates at 55 test centers located in 34 countries. In the first generation of the TOEFL test running from 1964 to the mid-1970s, the TOEFL contained five sections: listening comprehension, English structure, reading comprehension, vocabulary, writing ability.

A few years after the first administration of the TOEFL, the Educational Testing Service (ETS) started to feel the pressure to re-examine the format of the test and in particular, the Vocabulary subtest. Applied linguists’ criticized that the format of the vocabulary items in the TOEFL test provided great incentives for test candidates to memorize list of words and their synonyms out of context, which would not help develop actual language ability necessary for real-life communication. In response to the critics, EST proposed a new items type.

A.2 This Study
Naturally, language testers were keen to know whether the new format could provide as much prediction as did its predecessor, the discrete-point vocabulary items format. Therefore, there was a clear need for further research to address this question because the new format is now continuously employed as the primary vocabulary item type in the TOEFL iBT, which is a high-stakes test to money. Our study was conducted in this context with three specific research questions:

RQ1: How do the contextualized vocabulary items compare with the discrete-point vocabulary items in terms of predicting reading performance?

RQ2: How do the contextualized vocabulary items compare with the discrete-point vocabulary items in terms of the difficulty level for test takers?

RQ3: How do the contextualized vocabulary items compare with the discrete-point vocabulary items in terms of discriminating power?

A.3 Method
A.3.1 Participants

Two hundred and three international student from an intensive English-as-a-second-language program at a major Canadian university participated in the study. They were recruited from classes at the intermediate proficiency level and higher and came from 16 first-language backgrounds, including 72 Korean students, 46 Japanese, 32 Spanish, and 25 Chinese.

The education levels of the participants ranged from the complication of high school, master’s degrees. Most of the participants came from a large variety of academic fields totaling 65 disciplines, ranging from humanities and social sciences to natural sciences to natural sciences and engineering.

A.3.2 Instruments
The instruments used in this research were two TOEFL subparts. The first subpart was the Discrete-Point Vocabulary Measure (DV), and the second, the Reading Comprehension subtest (CV).

A.3.3 Procedures
All instruments were administered to the participating students in quite lecture room. The instruments were in a mixed order. A half participants were asked to complete DV first, and then CV. whereas the remaining participant receive the CV first, and then DV. The participants were all asked to complete a background questionnaire before taking the test. It’s for demographic information.

A.3.4 Analyses
Beside descriptive statistics, statistical procedure applied to the data set included Cronbach’s alpha reliabilities, Pearson product-moment correlation, a Hotteling t test, hierarchical regression, item difficulty indexing (IF),item discriminability indexing (point-biserial correlation coefficient), and independent sample t test. The level of significance level was at p < .01, instead of the normal p < .05 level.

A.4 Summary the Findings
The result of Pearson product-moment correlation and hierarchical multiple regression analyses showed that CV and DV scores statistically significant to RC score. Hotteling t test indicates that the amount of scores statistically equivalent. The result from item difficulty analyses show that CV and DV items fall within fairly similar ranges of difficulty level. The point-biserial correlation coefficient indicates that CV and DV provides similar and adequate discrimination powers. Independent-samples t test confirms that there was no statistical difference between the mean ID values of the two measures.

B. COMMENT AND DISCUSSION
Vocabulary, one of the three elements in language, is of great importance in the whole system of language. Words are the basic building blocks of language, the units of meaning from which larger structures such as sentences, paragraphs and whole texts are formed.

It is often said, “Without grammar, little can be conveyed; without words nothing can be conveyed.” With regard to second language acquisition, vocabulary is an indispensable and basic part. Both teachers and learners spend lots of time and energy on vocabulary acquisition. From the millions of vocabulary textbooks in the libraries and bookstores, we can have a brief idea that how much importance we attach to vocabulary (Nation, 2001).

A good knowledge of English vocabulary is important for anyone who wants to use the language, so knowledge of vocabulary is often tested. It is important that the test maker be aware of what he/she is doing when testing vocabulary. Vocabulary knowledge can be divided into four types (Kitao, 2006). The first is active speaking vocabulary, that is, words that the speaker is able to use in speaking. The second is passive listening vocabulary, which is a word that the listener recognizes but cannot necessarily produce when speaking. The third type is passive reading vocabulary, which refers to words that a reader recognizes but would not necessarily be able to produce.

Finally, there is active writing vocabulary, which is a word that a writer is able to use in writing (Oxford, R.L., & Crookall, D, 1990).
As an important part of language testing, vocabulary testing has also been influenced by the development of linguistic theories. Constructivists insist that language is composed of a set of structures and all languages can be analyzed from structures and can be divided into millions of small and independent linguistic elements.

Learning of language is the acquisition of all these elements. Therefore, discrete-point test, in which every language skill is tested separately in different items, was adopted to test language competence and it was regarded as the most valid and effective way of testing (Putri, 2010).

Nowadays, with the further research and development on linguistics, linguists have discovered that language competence is a multi-element entity, which are different from the traditional linguistic elements and skill theory and the mono-element theory. Both the integrative tests and discrete-point tests have advantages respectively.

As pointed out early in the article, a major advantage of adopting contextualized vocabulary assessment in the TOEFL Reading Comprehension subtest should be the likelihood that, by getting rid of discrete-point vocabulary items in the test to discourage test takers from having to memorize wordlists out of context, the new CV format may produce positive washback. This was clearly also the main rea¬son for the TOEFL Committee of Examiners to decide to devise the new format in the first place.

The statistics generated from this study have made a strong case for the con¬tinued adoption of the CV format, in terms of predicting reading performance, the CV format is just as closely related to overall RC as is the DV format. How-ever, because the CV format seems so likely to provide positive washback whereas the DV format has been linked to negative wastback, the former is clearly a preferred task format. Under the circumstances, there is no longer a need for decontextualized vocabulary measures in the TOEFL test, or in other large-scale tests of a similar nature.

As shown, the new contextualized vocabulary format encourages understand and learning new vocabulary in context, which is particularly meaningful for learning polysemous lexical items. The need to memorize long wordlists to score high on the vocabulary section has been a major cause for the criticism of some standardized English proficiency tests, including the pre-1995 TOEFL version. The pressure to achieve high scores on high-stakes tests and some cultural tradi¬tions that endorse rote learning have made it difficult for students not to memo¬rize wordlists when there are discrete-point vocabulary items in the test. Naturally, this will also affect how teachers teach and how students learn when the stakes of the test are high for them.

Presumably, the revision of the item format for the vocabulary assessment in me new TOEFL versions has mitigated the criticism of the discrete-point vocab-ulary format since the format changed in 1995, even though there are still doubts whether contextual clues provided by the passage are actually needed by test takers in working out the meaning of the target words. At least the new item type provides little motivation for prospective TOEFL takers to memorize lists of difficult words out of context, and for teachers to urge, and in some cultures even monitor, their students to remember and recite wordlists.

For teachers who would like to develop in-house vocabulary tests for forma¬tive or summative purposes, the CV item type used in the present study can also be recommended for classroom use because this assessment format encourages understanding and learning words in context. This would bring the learning closer to the actual communicative application of the language in the real world. Classroom assessment based on such a format should not be too difficult for the teacher to develop.

C. CONCLUSION
Generally speaking, more and more tests begin to test vocabulary in contexts. For instance, TOFEL test vocabulary by giving only a word to be tested and four choices of single word at the beginning. When more and more criticism appeared against such form, sentence context was adopted in TOFEL vocabulary text. From 1995, TOFEL began to test vocabulary in reading materials. Other standard test, such as CET, PETS, GMAT and IELTS, either adopt sentence context in vocabulary testing or paragraph context.

In addition, there some points that we need to pay attention to. The first is washback has in recent years been a hot research topic in the field of language testing. Language testers have devoted an enormous effort to determining how tests can produce positive washback, which has also become an aspect for judging test quality. However, washback research has turned up mixed results. Although tests may produce positive or negative washback effects, they do not necessary always exist even when test developers intended to produce them.

The second, there is paucity in empirical research directly linking vocabulary tests to washback although language practitioners generally assume context-dependent vocabulary assessment will lead to beneficial learning activities whereas context-independent vocabulary assessment may likely promote unwanted learning activities. It is therefore highly desirable to conduct research looking directly into how various forms of vocabulary tests will affect learning activities.

Usually vocabulary can be tested through the following ways: multiple-choice items, associated words, matching items, word formation test items, items involving synonyms, rearrangement items, and completion items and so on. Each form of testing has its own advantages and disadvantages. In the most public testing, multiple-choice items are preferred, for the convenience of scoring and its objectivity. No matter what way is adopted; vocabulary-testing items can be divided into context-independent items and context-independent items, when context is concerned. A context-independent vocabulary test presents words to the test-takers in isolation and requires them to select meanings for the words without reference to any linguistic context. While context-dependent vocabulary test is designed to assess the test-taker’s ability to take account of contextual information in order to produce the expected response. In the context-dependent vocabulary tests, context can be sentences or passages (Chun-Mei, 2007).

REFERENCES
Chun-Mei, GAO. 2007. Influence on Context of Vocabulary Testing. US-China Education Review Journal, 4(7):1-5.

Kitao, Kathleen K. & Kitao, Kenji. 2006. Testing Vocabulary. (Online), http://www.cis.doshisha.ac.jp/kkitao/. Accessed on 27 June 2012.

Nation, I.S.P. 2001. Learning Vocabulary in another Language. England: Cambridge University Press.

Oxford, R.L., & Crookall, D. 1990. Vocabulary Learning: A Critical Analysis of Techniques. TESL Canada Journal, 7(2):9-30.

Putri, Helena Fariska. 2010. The Importance of Vocabulary in English Learning. (Online), http://universityofibnkhaldunbogor-indonesia.blogspot.com/2010/01/importance-of-vocabulary-in-english.html. Accessed on 27 June 2012.

Qian, David D. 2008. From Single Words to Passages: Contextual Effect on Predictive Power of Vocabulary Measures for Assessing Reading Performance. Language Assessment Quarterly, 5(1):1-19.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s