CALL (Call Assisted Language Learning): CALL related to literature 2

5. Recent Developments in Technology and Language Learning: A Literature Review and Meta-analysis

YONG ZHAO
Michigan State University

Abstract:
The purpose of this study is to assess the potential of technology for improving language education. A review of the effectiveness of past and current practices in the application of information and communication technology(ICT) in language education and the availability as well as capacities of current ICTs was conducted. The review found that existing literature on the effectiveness of technology uses in language education is very limited in four aspects: a) The number of systematic, well-designed empirical evaluative studies of the effects of technology uses in language learning is very small, b) the settings of instruction where the studies were conducted were limited to higher education and adult learners, c) the languages studied were limited to common foreign languages and English as a foreign or second language, and d) the experiments were often short-term and about one or two aspects of language learning (e.g., vocabulary or grammar). However the limited number of available studies shows a pattern of positive effects. They found technology-supported language learning is at least as effective as human teachers, if not more so.

KEYWORDS

Effectiveness of Technology in Language Education, Limitations of Research Studies, Literature Review, Meta-analysis

INTRODUCTION

This review study is intended to address three related issues in technology and language education. First, policy makers and the general public are interested in learning about the effectiveness of using technology in language education because they need that information to help decide future investment decisions regarding technology (President's Committee of Advisors on Science and Technology (Panel on Educational Technology), 1997). Second, researchers and

developers are interested in knowing what has been done and what we already know about using technology to enhance language learning. Such knowledge will hopefully guide their further explorations and development. Third, language educators want to know what works and what does not so that they can make informed decisions in selecting the appropriate technology to use in their teaching.

Answering these questions is not easy for a number of reasons. First, technology is an ill-defined concept that encompasses a wide range of tools, artifacts, and practices, from multimedia computers to the Internet, from videotapes to online chatrooms, from web pages to interactive audio conferencing. These technologies vary a great deal in their capacity, interface, and accessibility. It is thus misleading to think the effects of videotapes are the same as those of the online chatrooms just because they are all called "technology." Second, the effects of any technology on learning outcomes lie in its uses. A specific technology may hold great educational potential, but, until it is used properly, it may not have any positive impact at all on learning. Thus, assessing the effectiveness of a technology is in reality assessing the effectiveness of its uses rather than the technology itself. Since most information and communication technologies (ICTs) can be used in a variety of ways, some more effective than others, it is inappropriate to overgeneralize the effectiveness (or lack thereof) of one way of using the technology to the technology itself. Third, to further complicate things, the effectiveness of an educational approach is highly mediated by many other variables—the learner, the task, the instructional setting, and of course the assessment tool. Thus, even the same use of a particular technology in different instructional settings may result in different learning outcomes.

Clearly it is unreasonable to expect any single study to tell us to what degree technology is effective in improving language learning. However, a comprehensive review of many studies can get us closer to an answer (e.g., Cavanaugh, 2001; Chapelle, 1997; Lou, Abrami, & d'Apollonia, 2001; Salaberry, 2001). With the help of a research method called meta-analysis (Glass, 1977; Hedges & Olkin, 1985; Lyons, 1995a), we can assess the effectiveness of technology uses in language education by analyzing findings of numerous empirical studies. A carefully conducted review can also help us develop a map of past and current work in the field of technology and language education. The map should reveal what we know, what we have done, what works, and what does not. The study in this paper offers such a review.

Focusing on the issues of effectiveness, this review attempts to achieve three goals: (a) assessing the overall effectiveness of uses of technology in language education through meta-analysis, (b) exploring patterns of recent efforts in using technology to improve language learning, and (c) identifying effective ways to use technology in language education.

METHOD

Selection of Studies

There is a long history of using technology to improve language learning (Salaberry, 2001). The review in this paper is limited to research published in referred journals during the last five years, from 1997 to 2001. The decision to limit the review to this period of time was motivated by the concern for relevance. The primary purpose of the review is to seek evidence and ideas that will guide our future work, rather than paint a comprehensive historical picture of research in computer-assisted language learning, which can be found in many existing publications (e.g., Chapelle, 2001; Levy, 1997; Salaberry, 2001). Thus, it is reasonable to focus on studies of technological applications that have the most relevance. Relevance is considered in two areas: technology and pedagogy. As we know, technology changes constantly and rapidly. The technological innovations that we are most interested in and that will most likely have an impact on language education in the future are: (a) multimedia computing; (b) the Internet, especially the web; and (c) speech synthesis and recognition. These innovations were a fairly recent development, and efforts to apply them in language education occurred even later. Focusing on the research publications over the past five years in this way should give us sufficient insight into the applications of these relatively new technologies.

There was also a major paradigm shift in the pedagogical and research focus of technology applications in language education recently (Chapelle, 1997, 2001; Pennington, 1996; Salaberry, 2001)—a shift away from traditional drill-and-skill computer-aided instruction (CAI) models toward multimedia, intelligent CAI, and integration models. Studies about applications of these newer models appeared more recently as well.

Works included in this review were identified from five representative journals devoted to research on second/foreign language education and technology and language learning.

Selecting Representative Journals

A four-step process was followed to identify the representative journals. First, a key word search using "computer assisted language learning" was performed on ERIC through FirstSearch. The search was limited by year, document type, and language. Only journal articles published from 1997 through 2001 in English were included. The search resulted in a total of 389 articles. Second, all articles that did not have the key word "second language" were excluded, which resulted in a total of 355 articles. Third, all articles that appeared in nonpeer-reviewed, irregularly published, or practice-oriented journals or magazines were excluded, resulting in a total of 156 articles. The fourth step was to calculate the distribution of the articles and their sources. These articles were published in 22 different journals. The journals fell into three categories: (a) technology and language learning journals that specifically address issues in applications of

technology in language learning and education; (b) language learning and education journals that address issues in language learning and education in general, of which technology application is a subissue; and (c) educational technology journals that address issues in the application of technology in education in general, of which language learning and education is considered a subarea of study. Figure 1 shows the distribution of articles by journal categories.

Figure 1

Distribution of Articles by Journal Category

However, in many cases, a journal may have had only one or two articles. Nine journals had more than three articles. (see Figure 2).

Figure 2

Distribution of Articles by Journals

As Figure 2 shows, most articles appeared in journals devoted to educational technology, especially language educational technology. The three journals devoted to language learning and technology (System, CALICO Journal, and Language Learning & Technology) published nearly 70% of all the articles.

Further, the articles appeared mostly in two journals: CALICO Journal and Language Learning & Technology (see Figure 3).

Figure 3

Distribution of Articles in Technology Journals

Based on this analysis, the CALICO Journal and Language Learning & Technology were first selected to represent studies published in journals devoted to technology and language learning. The Journal of Educational Computing Research was selected to represent journals devoted to educational technology. The Modern Language Journal and TESOL Quarterly were selected to represent journals in language learning and education. These two journals were selected over Foreign Language Annals (FLA) because they are generally considered more research oriented than FLA. This selection represents 72% of all articles found to be related to computer-assisted language instruction from the ERIC database during the period of 1997 through 2001. Considering that 13% of the articles are scattered in 12 other journals, the selection should be considered as a reasonable representative sample of journals that may publish studies in technology and second language learning and education.

Selection of Studies

Having selected these five journals, the researcher read the abstracts of all articles published in the journals since 1997 to identify possible studies to include in this review. After identifying the possible articles, the researcher read all identified articles to select the ones suitable for a meta-analysis. The following criteria were used for the selection:

1. The article had to report results of an empirical study or multiple studies on the effectiveness or effects of a technological application aimed at improving language learning. Informational articles describing the process of technological applications or products and opinion articles presenting theories or review the literature were included as part of the review but not for the meta-analysis of effects.

2. Technology was more broadly conceived than just computers because the convergence of media is an emerging trend in language education technology. Thus, studies about the application of video, audio, computer-assisted instruction programs, the web, computer-mediated communication, simulation, speech technologies, word processing, e-books, and grammar checkers were all included.

3. The studies included for the final meta-analysis had to have measures of improvement of language proficiency. Self-assessment of improvement or attitudinal surveys were not acceptable.

A total of 9 articles were found to meet the above criteria for inclusion in the meta-analysis although many more articles published in these journals dealt with technological applications in language learning. While the meta-analysis was conducted on only the 9 studies, the review here draws upon discussions and findings from the other articles as well as a number of books published around the same period of time.

RESULTS

This section is organized into three parts. Part one presents an overview of the literature on technology applications in language education. Part two summarizes how technology has been used to support language learning and the effectiveness of these uses. Part three reports the results from the meta-analysis study intended to assess the overall effectiveness of technology-supported language learning.

Overview of the Literature

The review of recent research on technology-supported language learning reveals a number of interesting points regarding existing research in this area. First, the total number of well designed experimental studies on the effectiveness of technological applications in language learning is very limited. For instance, the four issues in Volume 16 of the CALICO Journal, which were published in 1998 and 1999, contain 10 feature articles. Only two of the 10 met the selection criteria to be included in the meta-analysis. Of all the 51 feature articles published in Language Learning & Technology, one met the criteria. The majority of the articles are either description of cases—uses of technology in language education and processes of software development—or theoretical discussions of principles of technological applications. Some of the limited

empirical studies did not measure learning gains, relied solely on learner self-reports as measures of effectiveness, or were not well designed.

Second, the studies were limited to college level language learners. None of the nine studies was conducted in K-12 settings. Only two of the nine studies were not about college students—their participants were military linguists, also adults.

Third, the target languages under consideration in the studies were also limited. French and Spanish were the most studied (10 out 16 or 63%). Other more commonly studied languages were English as a second language (ESL), German, and Arabic.

Fourth, most of the studies were about the application of a single application instead of systemic large-scale integration of technology. Only two of the 16 studies were about long-term technology integration in the language learning environment. Thus the treatment reported was also short term, lasting from a few hours to a few weeks.

Lastly, the applications of technology to enhance language learning have been wide ranging, both in terms of the types of technology used and the issues language education needs to address. The studies investigated the uses of most available technologies including video, audio, multimedia, communication, network, and speech technologies. These technologies have been applied to support the teaching of various aspects of language learning including vocabulary, grammar, reading, writing, speaking, listening, and culture.

Uses and Effectiveness of Technologies in Language Education

The following paragraphs present detailed discussions of the specific applications that have been studied over the past few years. The discussion is categorized into four groups: access to materials, communication opportunities, feedback, and learner motivation.

Providing Access to Linguistic and Cultural Materials

Access and exposure to engaging, authentic, and comprehensible yet demanding materials in the target language is essential for successful language learning. However for many language learners, whether in classes or self-study settings, such access is often limited. Thus language educators have long looked at information and communication technologies (ICT) as possible solutions to this problem (Egbert, Chao, & Hanson-Smith, 1999; Hanson-Smith, 1999; Salaberry, 2001). The uses of ICT to provide better access and exposure to linguistic and cultural materials fall into the following three areas.

Enhancing access efficiency through digital multimedia technologies

Digital multimedia technologies were used to make access to learning materials more efficient than print media or audio recorders because multimedia (visual, audio, and text) presentations can create stronger memory links than a single

medium alone and digital technology allows instant and accurate playbacks, which helps the learner to access specific segments much more easily without spending time to locate them—a tedious and time consuming process (Hanson-Smith, 1999; Thorton & Dudley, 1996). Shea (2000) compared the time students needed to complete their language learning tasks using captioned video versus interactive videodisc (IVD) and found that the students using IVD completed the tasks significantly faster (p <. 05). Labrie (2000) found that although students spent more time learning a set of French words on paper than those who studied on computer (where they could hear a word pronounced and see a picture about the word), they did not learn more words. In another study, Nutta (1998) examined the learning of students who spent the same amount of time (one hour per day for seven days) learning verb tenses in English in two conditions: attending a regular class and receiving instruction from the teacher versus using a multimedia computer program (audio, video, recording capabilities etc.). Nutta found that the ESL students using the computer program performed as well or significantly better (on three out of six measures, p < .10) than their counterparts attending the class.

Enhancing authenticity using video and the Internet

Video materials can bring natural and context-rich linguistic and cultural materials to the learner, while the Internet enables the learner to access authentic news and literature in the target language, which can reflect current cultural changes more effectively than printed sources (Bacon & Finnemann, 1990; Hanson-Smith, 1999; Herron, Cole, Corrie, & Dubreil, 1999; Herron, Dubreil, Cole, & Corrie, 2000; Kitajima & Lyman-Hager, 1998; Lafford & Lafford, 1997; Lee, 1998; Weyers, 1999). Weyers (1999) studied the effectiveness of authentic video on college Spanish students. He had one class of students watch a Mexican television show as part of a second semester Spanish class that met 60 minutes daily for a total of 8 weeks, while the other class followed the regular curriculum without the video. He found that the video group's performance on both listening comprehension and oral production to be significantly better than the regular group (p < .01). The video group also outperformed their counterparts on other measures of communicative competence. Herron (2000) found that video also helped their first-year college French learners develop significantly better understanding of the target culture. In another study, Green and Youngs (2000) substituted regular classroom instruction with web activities one class period per week for beginning college French and German students. After a semester, they found that

the substitution of one class day for directed, pedagogically sound Web activities seems to have allowed the treatment groups to continue to progress toward their personal and professional goals and allowed them to learn language at a rate similar to that of their peers in the control groups. It also appears, in general, that the students had a positive experience using the web … (p. 108).

Enhancing comprehensibility through learner control and multimedia annotations

Comprehensible input is necessary for language learning, but useful learning materials must also contain enough unfamiliar materials (Krashen, 1985). For language learners, especially beginning and intermediate ones, authentic materials are often beyond their language proficiency and may become incomprehensible without help. To enhance comprehensibility of spoken materials, full caption, keyword caption, or slowing down the speech rate have been found to be effective (Shea, 2000; Zhao, 1997). Zhao (1997) found that the ESL students who were able to flexibly slow down or speed up the rate of speech had significantly better listening comprehension than those who did not (p < .05). For reading materials, glossing or multimedia annotations have been effective means to enhance comprehension (Al-Seghayer, 2001; Chun & Plass, 1997; Johnson, 1999; Lyman-Hager, 2000). Al-Seghayer (2001) compared ESL students' vocabulary learning in different annotation conditions and found that

a video clip in combination with a text definition is more effective in teaching unknown vocabulary than a picture in combination with a text definition … . The variety of modality cues can reinforce each other and are linked together in meaningful ways to provide an in-depth experience (p < .001 ) (p. 225).

Providing Opportunities for Communication

Engaging in authentic communication in the target language is another essential condition for successful language learning yet such opportunities do not exist for most learners. ICT has again been used in many different ways to create opportunities for language learners to communicate in the target language (Hanson-Smith, 1999; Kelm, 1998; Muyskens, 1998; Warschauer & Kern, 2000). Efforts in this area can be summarized into two groups: interaction with the computer and interaction through the computer with remote audiences.

Interactions with the computer

Communicative interactions can occur in either written or spoken language or a combination of both. At the simplest level, a computer program can generate utterances either orally or in writing that require the learner to respond by selecting an answer with a mouse click or providing simple writing responses (Hanson-Smith, 1999). With the advancement of speech synthesis and recognition technologies (Ehsani & Knodt, 1998), the learner can also carry on near natural conversations with a computer program around preselected and programmed topics (Bernstein, Najmi, & Ehsani, 1999; Egan, 1999; Harless, Zier, & Duncan, 1999; LaRocca, Morgan, & Bellinger, 1999; Wachowicz & Scott, 1999). The learner can also give either written or spoken commands to a computer program in a simulation and game environment. The computer program would then perform the command (Holland, Kaplan, & Sabol, 1999; LaRocca

et al., 1999). Harless et al. (1999), for example, tested the effectiveness of a virtual conversation program in Arabic at the Defense Language Institute. The program enabled the students to interview virtual native-speaking characters orally with speech recognition technology. After interacting with these virtual characters for at least 8 hours per day for 4 days, the participants' reading and speaking skills increased significantly (p < .05) while their listening skill increased "convincingly." In another study, Holland (1999) and her colleagues found that a speech-enabled interactive microworld program which allowed the learners of Arabic to construct objects by speaking to the computer improved student motivation and oral output.

Interactions with remote audiences through the computer

Computer-mediated communication (CMC) and teleconferencing technologies have been used to create authentic communication opportunities for language learners since the 1980s (Beauvois, 1997; Pennington, 1996). The uses of CMC technologies, such as electronic mail, bulletin boards, and chatrooms have been found to have many benefits for language learners (Beauvois, 1997; Cahill & Catanzaro, 1997; Kelm, 1998; Salaberry, 2001; Warschauer, 1998). CMC brings the much needed audience to the language learner (Johnson, 1999). It also promotes more equal and better participation, leading to more output in the target language (Beauvois, 1997; Gonzalez-Bueno, 1998). It fosters negotiation and form-focused learning (Pellettieri, 2000). CMC was also found to enhance the writing process and improve student writing (Schultz, 2000). Although CMC communication is, in most cases, conducted in writing, it has been found to improve oral proficiency as well. For instance, Beauvois (1997) found that second-year French learners who held their discussions online achieved better oral proficiency than those who discussed the texts orally in the traditional classroom setting (p < .05).

Providing Feedback

The capacity for computers to provide instant and individualized feedback has long been recognized by educators, including foreign language educators (Chao, 1999; Salaberry, 2001). While early applications tended to follow the behaviorist tradition by simply assessing the learner's performance and providing simplistic feedback in a correct-or-incorrect fashion, more recent applications are much more contextualized and pedagogically sound (Salaberry, 2001).

Computer-based grammar checkers and spell checkers

Computer-based grammar checkers and spell checkers represent potentially powerful ways to provide feedback to students' written output (Jacobs & Rodgers, 1999). Although the feedback provided by current grammar checkers is not always accurate—albeit immediate—due to its inability to perform semantic analysis and process deep level structures, Burston (2001) found that advanced

students of French benefited tremendously from a French grammar checker. In this study, the students in the treatment group used a French grammar checker while writing their essays, whereas the control group did not. The results suggest that "the effectiveness of the use of Antidote in improving morphosyntatic accuracy in assigned compositions were overwhelmingly positive." (p. 507). The treatment group's first essay scored on average 70%, compared to 20% of the control group. The second essay showed similar results: 85% for the treatment group and 54% for the control.

Automatic speech recognition technology

Automatic speech recognition technology holds the potential to provide feedback that would otherwise be impossible. Pronunciation is a fundamental element of language learning, but providing feedback that can be easily accessible and useful is difficult. In traditional instructional settings, feedback and modeling are often provided by an instructor, who may or may not be good at judging the student pronunciation in the first place. Typical ways to provide feedback often include having students repeat the pronunciation or explaining how the sound should be produced in a very abstract fashion. With the advancement of speech recognition technology, the student can receive feedback in more effective ways (Dalby & Kewley-Port, 1999; Ehsani & Knodt, 1998; Eskenazi, 1999; Mostow & Aist, 1999). Mostow and Aist (1999) have suggested visual, template-based, and model-based feedback. First, a computer program can analyze a student utterance and display the features visually, perhaps with a comparison to that of a native speaker. The program can also display the position and movements of the tongue when a student produces an utterance, which can also be displayed in comparison to that of native speakers. Second, computer programs can compare student pronunciation of individual words or sentences to prerecorded templates. For example, good agreement (r =.81 for high quality speech and r = .76 for telephone-quality speech) was found between automatic and human grading of the pronunciation of English sentences produced by Japanese English learners (Bernstein, Cohen, Murveit, Rtischev, & Weintraub, 1990). More recent studies have found different levels of correlation between machine and human graders: from 0.44 to 0.85 (Bernstein, 1997; Ehsani & Knodt, 1998). Coniam (1998) also found that such high correlation can be achieved at the discourse level. Third, pronunciation can be evaluated against pronunciation models. In this approach, student pronunciation is not limited to preselected words because the model is a generalization of a template.

Tracking and analyzing student errors and behaviors

Tracking and analyzing student errors and behaviors is another approach language educators have experimented with to provide more helpful feedback. Computer programs can store student responses, which can then be analyzed either by a human instructor (Sinyor, 1997) or the computer (Nagata, 1993).

The effectiveness of this approach remains to be determined although Nagata, summarizing her research findings, suggests "traditional feedback may be as good as the intelligent feedback for helping learners to correct word-level errors (e.g., vocabulary and conjugation errors), while the intelligent feedback may be more helpful for understanding and correcting sentence level errors (e.g., particle errors)" (p. 337).

Integrating Technology in the Language Classroom

As mentioned before most of the empirical studies were about a single application used in a few days. We were fortunate to have found two articles that evaluated the effectiveness of more comprehensive uses of technology over a longer period of time (Adair-Hauk, Willingham-McLain, & Youngs, 2000; Green & Youngs, 2001). These efforts were all carried out at Carnegie Mellon University. The first study (Adair-Hauk et al., 2000) was conducted in 1996, and the second study took place in Fall 1998 and Spring 1999 (Green & Youngs, 2001). Participants of the first study were second-semester French students and those of the second study were first-semester French students and first- and second-semester German students. Both studies followed the same format: the treatment group participated in technology-enhanced language learning activities, while the control group attended a regular class for one of the class periods each week. The technological applications included computerized multimedia grammar and vocabulary exercises, instructional video, online spell checker, French-English glossary, and the web. Measures of listening, reading, writing, cultural knowledge, and student attitudes were taken during the course of both studies. Speaking was assessed in the first study. The findings are summarized below.

1. For study 1, when change over time was considered, there was no significant difference between the treatment group and the control group in cultural knowledge, speaking or listening. For study 2, there was no significant difference in any of the skills measured (cultural knowledge, listening, reading, and writing) (p < .05).

2. However, the difference in writing was significant in study 1. The control group's homework writing scores decreased, while the treatment group's increased. Writing test scores also indicate a significant difference between the two groups favoring the treatment group (p < .001). The treatment group also scored significantly better than the control group in reading (p < .001).

3. Both studies found that students in the treatment group spent about the same amount time completing the tasks as their peers in the control group.

Both studies concluded that technology-supported independent language learning is as effective as classroom instruction, if not more.

Assessing the Overall Effectiveness: A Preliminary Meta-analysis

In order to gain a better sense of the overall effectiveness of technology applications in language learning, a meta-analysis was conducted of the studies that included enough data for such an analysis. Meta-analysis is the analysis of analyses—a statistical technique for aggregating the results of multiple experimental studies (Glass, 1976, 1977; Hedges & Olkin, 1985; Lyons, 1995a). The result of each identified study is converted into a measure called effect size. An effect size is obtained by transforming the findings from each study into a standard deviation unit. The effect size indicates the extent to which experimental and control groups differ in the means of a dependent variable at the end of a treatment phase. An effect size (d) is calculated as the difference between the means of the treatment group and the control group divided by the pooled standard deviation.

For the meta-analysis presented here, more than one effect sizes was calculated for several studies because they had more than one measure (e.g., listening, reading, and writing). But in order to satisfy the independence assumption of meta-analysis (Hedges & Olkin, 1985), only one effect size per study was entered into the study. When two or more effect sizes were calculated, they were averaged. The effect sizes used in this analysis are weighted ds, which corrected sample size biases (Hedges & Olkin, 1985). The calculation was performed using Meta-analysis Calculator (Lyons, 1995b), a computer program designed for meta-analyses. Table 1 summarizes the results of the meta-analysis study.

Table 1

Overall Effect of Technology Applications in Language Learning

Notes: Averaged shows the result when only one averaged effect size was included per study, while All shows the result when all effect sizes are included. K = number of effect sizes. N = number of subjects. The number of subjects was repeatedly counted for each effect size for studies that used multiple measures.

As Table 1 shows, the mean effect size of the 9 studies is quite large, indicating an overwhelmingly positive effect of technology applications on language learning. The confidence interval at the .05 level further confirms this finding. Thus judging from this analysis, it is reasonable to conclude that technology has been shown by the published empirical studies to be very effective in improving student language learning. What is worth mentioning is that this analysis put all technologies and their various applications in language learning

together. It did not differentiate among the areas of improvement in the target language either. In other words, this is a summary of the empirical findings of the effects of a variety of technological applications on virtually all aspects of language learning (e.g., vocabulary, grammar, reading, listening, writing, speaking, and cultural understanding.). Table 2 summarizes the technological applications and measures under investigation in the 9 studies comprising the meta-analysis sample.

Table 2

Summary of Technology and Content of Studies

DISCUSSION AND CONCLUSIONS

This review study was conducted to achieve three goals: (a) to assess the overall effectiveness of uses of technology in language education through meta-analysis, (b) to explore patterns of recent efforts in using technology to improve language learning, and (c) to identify effective ways to use technology in language education. In this final section, the findings of the study are summarized and their implications are discussed for future research and development efforts in technology supported language education.

In terms of overall effectiveness of technology on language learning, there is evidence suggesting that technology-based language instruction can be as effective as teacher-delivered instruction. Although the number of available experimental studies is limited, a consistent pattern of positive effects is found across the studies. However, this finding should be interpreted with extreme caution for a number of reasons, in addition to the limited number of studies. First, there may be a tendency for journals to publish studies that report significant positive gains. In other words, studies that found less significant or even

negative effect of technology may not have been published. While there is no simple way to verify this assumption, it is to some extent supported by a recent meta-analysis study about the effects of social contexts on computer uses in learning, which found that published studies have a bigger effect size than unpublished studies (Lou et al., 2001). Second, most of the studies had fairly small sample sizes and rarely employed random sampling. Third, the fact that all studies were conducted on college students and adult learners raises questions about the generalizability of the finding to other language learners who may differ in motivation, language background, learning style and ability, and instructional context. For instance, it is very likely that college students are generally more motivated and better learners than K-12 students as a whole. Lastly, in most cases, the researchers of these studies were also the instructors who designed, implemented, and evaluated the technology uses. It is conceivable that the classical "Pygmalion effect" (Rosenthal, 1973) could affect the results. It is also the case that most of the instruments were designed by the researchers, who were also the instructors, instead of independent standardized instruments. It is possible that these measures might have a bias in favor of conditions where technology was applied.

This review found that recent efforts in applying technology in language education share three interesting characteristics. First, many of these efforts were carried out by individual instructors or small groups of individual instructors with limited resources. Consequently, efforts were of smaller scale. Very often only individual technology was used to affect a very specific part of language education. The review found a very limited number of efforts that attempted comprehensive applications of multiple technologies to the whole process of language education. Second, most efforts involved the development of products which then were used in language teaching. Commercially available language software or tools were rarely used in these studies. As a result, the review found many publications describing the development process, while only few articles reported the effectiveness of these products. In the meantime, the numerous commercial products readily available and widely used in classrooms were not studied. Third, the review found that in general current attempts to use technology in language education were not connected and ignored the language learning at the precollege level. This finding is very surprising, especially viewed in the context of technology applications on other content areas, such as mathematics and science, where a large number of studies of technology applications were conducted at the K-12 level.

What are effective uses of technology in language education? This review shows that the application of technologies can be effective in almost all areas of language education. Modern technology can help enhance the quality of input, authenticity of communication, and provide more relevant and useful feedback. In particular, communication technologies such as the Internet and satellite television have been found to be widely used as a way to bring authentic materials into the classroom, involve learners in more authentic communications with

distant audiences, and provide researchers the opportunity to better examine the language learning process. Additionally speech technology, while still not quite ready for full implementation for language education, has already been shown its potential for supporting language learning.

Findings from this review study have significant implications for future work. It is apparent from the literature review that technology, when used properly, can have a positive effect on language learning. It is also apparent that the availability and capacities of information technologies have not been fully taken advantage of by language students or educators. To truly capitalize on modern information and communication technologies to significantly improve language learning, a number of issues must be addressed.

Issue 1: Comprehensive and systematic development of curriculum and content

Technology capacities need to be translated into pedagogical solutions and realized in the forms of curriculum and content for language learners. Current uses of technology, as revealed by the literature review, are fragmented and isolated. There are very few comprehensive technology-based curricula that fully take advantage of the power of available technologies. Thus, in the future, what is needed is the development of full curricula that are supported by available technologies instead of individual tools that are only used infrequently or as a supplement to a primarily print-material-based curriculum.

Issue 2: Basic research to explore effective ways of using technology

The effectiveness of technology on language learning is dependent on how it is used. Certain technologies are more suitable than others for certain learning tasks for certain learners. Therefore research about appropriate ways and contexts of technology use is much needed (Salaberry, 2001).

Issue 3: Technology uses in the classroom

Technology is underutilized in classrooms (Cuban, 2001). The finding that none of the studies found in the major language education and technology journals is about technology use in K-12 classrooms is shocking because studies of technology applications in other subject areas (e.g., mathematics, science, social studies, and language arts) have taken place in mainly K-12 classrooms. The fact that almost all the authors of the reported studies were also the instructors in the experiments suggests a possible explanation: K-12 teachers are not using technology in their teaching and there is a lack of interest among university researchers in studying technology applications in K-12 language classrooms. This finding raises two issues: how to promote technology use in K-12 classrooms and how to encourage more research about technology use in K-12 language classes.

Issue 4: Large scale systematic empirical evaluation of technology uses in schools

There is a clear lack of systematic empirical evaluation efforts to assess the effectiveness of large scale comprehensive uses of technology to support language learning. This, of course, may be accounted for by the lack of large-scale implementation efforts. However, it may also be the result of an overall emphasis on the process rather than result of using technology in language learning. As mentioned earlier in this paper, there was a shift among researchers of language education in the mid 1980s from product-oriented research to more process-oriented research which focuses on understanding how students learn instead of what and how much they learn. For example, many studies on CMC have been about the nature and patterns of student participation in online interactions rather than how much their language skills have improved. While such research is necessary and important, we cannot ignore the practical question of how and in what ways technology uses are effective in improving language learning.

NOTE

This study was supported by a contract from the United States Department of Education as part of its E-language initiative. An earlier version of this paper was submitted to the US Department of Education as a concept paper. The author wishes to thank Dr. Alan Ginsburg, Dr. Susan Sclafani, and Adrianna de Kanter for their support and insights. Views expressed in this paper do not necessarily reflect those of the US Department of Education.

REFERENCES

Adair-Hauk, B., Willingham-McLain, L., & Youngs, B. E. (2000). Evaluating the integration of technology and second language learning. CALICO Journal, 17 (2), 269-305.

Al-Seghayer, K. (2001). The effect of multimedia annotation modes on L2 vocabulary acquisition: A comparative study. Language Learning & Technology, 5 (1), 202-232. Retrieved from http://llt.msu.edu/vol5num1/alseghayer/default.html

Bacon, S., & Finnemann, M. (1990). A study of the attitudes, motives, and strategies of university foreign language students and their disposition to authentic oral and written input. The Modern Language Journal, 74, 459-473.

Beauvois, M. H. (1997). Computer-mediated communication(CMC): Technology for improving speaking and writing. In R. M. Terry (Ed.), Technology enhanced language learning (pp. 165-184). Lincolnwood, IL: The National Textbook Company.

Bernstein, J. (1997). Automatic spoken language assessment by telephone (Tech. Rep. No. 5-97). Menlo Park, CA: Entropic, Inc.

Bernstein, J., Cohen, M., Murveit, H., Rtischev, D., & Weintraub, M. (1990). Automatic evaluation and training in English pronunciation. Paper presented at the International Conference on Spoken Language Processing (ICSLP), Kobe, Japan.

Bernstein, J., Najmi, A., & Ehsani, F. (1999). Subarashii: Encounters in Japanese spoken language education. CALICO Journal, 16 (3), 361-384.

Burston, J. (2001). Exploiting the potential of a computer-based grammar checker in conjunction with self-monitoring strategies with advanced level students of French. CALICO Journal, 18 (3), 499-515.

Cahill, D., & Catanzaro, D. (1997). Teaching first-year Spanish online. CALICO Journal, 14 (2-4), 97-114.

Cavanaugh, C. S. (2001). The effectiveness of interactive distance education technologies in K-12 learning: A meta-analysis. International Journal of Educational Telecommunications, 7 (1), 73-88.

Chao, C.-C. (1999). Theory and research: New emphases of assessment in the language learning classroom. In E. Hanson-Smith (Ed.), CALL Environments: Research, practice, and critical issues (pp. 243-256). Alexandria, VA: TESOL.

Chapelle, C. A. (1997). CALL in the year 2000: Still in search of research paradigms. Language Learning & Technology, 1 (1), 19-43. Retrieved from http://llt.msu. edu/vol1num1/chapelle/default.html

Chapelle, C. A. (2001). Computer applications in second language acquisition: Foundations for teaching, testing, and research. Cambridge: Cambridge University Press.

Chun, D. M., & Plass, J. L. (1997). Research on text comprehension in multimedia environments. Language Learning & Technology, 1 (1), 60-81. Retrieved from http://llt.msu. edu/vol1num1/chun_plass/default.html

Coniam, D. (1998). The use of speech recognition software as an English language oral assessment instrument: An exploratory study. CALICO Journal, 15 (4), 7-24.

Cuban, L. (2001). Oversold and underused: Computers in schools 1980-2000. Cambridge, MA: Harvard University Press.

Dalby, J., & Kewley-Port, D. (1999). Explicit pronunciation training using automatic speech recognition technology. CALICO Journal, 16 (3), 425-445.

Egan, K. B. (1999). Speaking: A critical skill and a challenge. CALICO Journal, 16 (3), 277-293.

Egbert, J., Chao, C.-C., & Hanson-Smith, E. (1999). Computer-enhanced language learning environment: An overview. In E. Hanson-Smith (Ed.), CALL environments: research, practice and critical issues (pp. 1-16). Alexandria, VA: TESOL.

Ehsani, F., & Knodt, E. (1998). Speech technology in computer-aided language learning: Strengths and limitations of a new CALL paradigm. Language Learning & Technology, 2 (1), 45-60. Retrieved from http://llt.msu.edu/vol2num1/article3/index.html

Eskenazi, M. (1999). Using a computer in foreign language pronunciation training: What advantages? CALICO Journal, 16 (3), 447-469.

Glass, G. (1976). Primary, secondary and meta-analysis of research. Educational Researcher, 5, 3-8.

Glass, G. (1977). Integrating findings: The meta-analysis of research. Review of Research in Education, 5, 351-379.

Gonzalez-Bueno, M. (1998). The effects of electronic mail on Spanish L2 discourse. Language Learning & Technology, 1 (2), 55-70. Retrieved from http://llt.msu.edu/vol1num2/article3/default.html

Green, A., & Youngs, B. E. (2001). Using the web in elementary French and German courses: Quantitative and qualitative study results. CALICO Journal, 19 (1), 89-123.

Hanson-Smith, E. (1999). Classroom practice: Using multimedia for input and interaction in CALL environments. In E. Hanson-Smith (Ed.), CALL environments: research, practice, and critical issues (pp. 189-215). Alexandria, VA: TESOL.

Harless, W. G., Zier, M. A., & Duncan, R. C. (1999). Virtual dialogues with native speakers: The evaluation of an interactive multimedia method. CALICO Journal, 16 (3), 313-337.

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando: Academic Press.

Herron, C., Cole, S. P., Corrie, C., & Dubreil, S. (1999). The effectiveness of a video-based curriculum in teaching culture. The Modern Language Journal, 83 (4), 518-533.

Herron, C., Dubreil, S., Cole, S. P., & Corrie, C. (2000). Using instructional video to teach culture to beginning foreign language students. CALICO Journal, 17 (3), 395-430.

Holland, V. M., Kaplan, J. D., & Sabol, M. A. (1999). Preliminary tests of language learning in a speech-interactive graphics microworld. CALICO Journal, 16 (3), 339-359.

Jacobs, G., & Rodgers, C. (1999). Treacherous allies: Foreign language grammar checkers. CALICO Journal, 16 (4), 509-529.

Johnson, B. (1999). Theory and research: Audience, language use, and language learning. In E. Hanson-Smith (Ed.), CALL environments: Research, practice, and critical issues (pp. 55-64). Alexandria, VA: TESOL.

Kelm, O. R. (1998). The use of electronic mail in foreign language classes. In K. Arens (Ed.), Language learning online (pp. 141-154). Austin, TX: The Daedalus Group Inc.

Kitajima, R., & Lyman-Hager, M. A. (1998). Theory-driven use of digital video in foreign language instruction. CALICO Journal, 16 (1), 37-48.

Krashen, S. (1985). The input hypothesis: Issues and implications. London: Longman.

Labrie, G. (2000). A French vocabulary tutor for the web. CALICO Journal, 17 (3), 475-499.

Lafford, P. A., & Lafford, B. A. (1997). Learning language and culture with the Internet. In R. M. Terry (Ed.), Technology-enhanced language learning. Lincolnwood, IL: The National Textbook Company.

LaRocca, S. A., Morgan, J. J., & Bellinger, S. M. (1999). On the path to 2X learning: Exploring the possibilities of advanced speech recognition. CALICO Journal, 16 (3), 295-309.

Lee, L. (1998). Going beyond classroom learning: Acquiring cultural knowledge via online newspapers and intercultural exchanges via online chatrooms. CALICO Journal, 16 (2), 101-120.

Levy, M. (1997). Computer-assisted language learning: Context and conceptualization. Oxford: Oxford University Press.

Lou, Y., Abrami, P. C., & d'Apollonia, S. (2001). Small group and individual learning with technology: A meta-analysis. Review of Educational Research, 71 (3), 449-521.

Lyman-Hager, M. A. (2000). Bridging the language-literature gap: Introducing literature electronically to the undergraduate language student. CALICO Journal, 17 (3), 431-452.

Lyons, L. C. (1995a). Meta-analysis: Methods of accumulating results across research domains [web]. Retrieved December 20, 2001, from http://www.mnsinc.com/solomon

Lyons, L. C. (1995b). The meta-analysis calculator : A SuperCard project for conducting a meta-analysis based on Hunter, Schmidt, and Jackson (1982) and Hunter and Schmidt (1990) (Version 1.0). Manassas, VA: Author.

Mostow, J., & Aist, G. (1999). Giving help and praise in a reading tutor with imperfect listening—Because automated speech recognition means never being able to say you're certain. CALICO Journal, 16 (3), 407-424.

Muyskens, J. A. (1998). New ways of learning and teaching: Focus on technology and foreign language education. Boston: Heinle & Heinle.

Nagata, N. (1993). Intelligent computer feedback for second language acquisition. The Modern Language Journal, 77, 330-339.

Nutta, J. (1998). Is computer-based grammar instruction as effective as teacher-directed grammar instruction for teaching L2 structures? CALICO Journal, 16 (1), 49-61.

Pellettieri, J. (2000). Negotiation in cyberspace: The role of chatting in the development of grammatical competence. In R. Kern (Ed.), Network-based language teaching: Concepts and practice (pp. 59-86). Cambridge: Cambridge University Press.

Pennington, M. C. (1996). The power of the computer in language education. In M. C. Pennington (Ed.), The power of CALL (pp. 1-14). Houston, TX: Athelstan.

President's Committee of Advisors on Science and Technology (Panel on Educational Technology). (1997). Report to the president on the use of technology to strengthen K-12 education in the United States. Washington, DC: President's Committee of Advisors on Science and Technology.

Rosenthal, R. (1973). The Pygmalion effect lives. Psychology Today, 7 (4), 56-60.

Salaberry, M. R. (2001). The use of technology for second language learning and teaching: A retrospective. The Modern Language Journal, 85 (1), 39-56.

Schultz, J. M. (2000). Computers and collaborative writing in the foreign language curriculum. In R. Kern (Ed.), Network-based language teaching: Concepts and practice (pp. 121-150). Cambridge: Cambridge University Press.

Shea, P. (2000). Leveling the playing field: A study of captioned interactive video for second language learning. Journal of Educational Computing Research, 22 (3), 243-263.

Sinyor, R. (1997). An analysis of student behavior and error sources in an Italian CALL context. CALICO Journal, 14 (2-4), 35-50.

Thorton, P., & Dudley, A. (1996). The CALL environment: An alternative to the language lab. CAELL Journal, 7 (4), 29-34.

Wachowicz, K. A., & Scott, B. (1999). Software that listens: It's not a question of whether, it's a question of how. CALICO Journal, 16 (3), 253-276.

Warschauer, M. (1998). Researching technology in TESOL: Determinist, instrumental, and critical approaches. TESOL Quarterly, 32 (4), 757-761.

Warschauer, M., & Kern, R. (2000). Network-based language teaching. Cambridge, UK: Cambridge University Press.

Weyers, J. R. (1999). The effects of authentic video on communicative competence. The Modern Language Journal, 83 (3), 339-349.

Zhao, Y. (1997). The effects of listener's control of speech rate on second language comprehension. Applied Linguistics, 18 (1), 49-68.

AUTHOR'S BIODATA

Yong Zhao is an associate professor of educational technology at Michigan State University, where he also directs the Center for Teaching and Technology. His research interests include computer-assisted language learning and online learning.

AUTHOR'S ADDRESS

Yong Zhao

115 Erickson

College of Education

Michigan State University

East Lansing, MI 48864

Phone: 517/353-4325

Fax: 517/432-4797

Email: zhaoyo@msu.edu

6. An Empirical Investigation of Whether Authentic Web Sites Facilitate Intermediate-level French Language Students� Ability To Learn Culture

SEBASTIEN DUBREIL
University of Notre Dame
CAROL HERRON
Emory University
STEVEN P. COLE
Research Design Associates, Inc.

Abstract:
This investigation examined the effects of authentic Francophone web sites on cultural learning at the intermediate level. Fifty-four students �surfed� eight web sites. A pre-posttest design assessed long-term gains in culture and the learning of practices, products, and perspectives from exposure to a curriculum with Internet activities. Eight posttask tests measured the students� ability to retain information and to make inferences. A questionnaire assessed perceptions of learning. Results indicated a significant gain in cultural knowledge with posttest scores significantly higher than pretest scores. Students learned significantly more products than practices over the semester. On the short-answer and free-recall portions of the posttask tests, students� ability to make inferences or retain information did not improve significantly in either an advance organizer (AO) or a non-AO condition. The students believed that the web sites featured significantly more cultural products than practices. Results support using the Internet to teach culture and its potential to emphasize cultural products.

KEYWORDS

Internet, Cultural Learning, Intermediate-level Foreign Language Students, Experimental Research

INTRODUCTION

The long-standing endorsement by foreign language (FL) educators to include culture in the curriculum became a reality in the last two decades with the

establishment of nationally recognized standards for cultural acquisition (Standards for Foreign Language Learning, 1996, 1999). In a proficiency-oriented perspective, where students learn a language to understand and to perform it, learning a foreign culture can be seen as “the process of acquiring the culture-specific and culture-general knowledge, skills, and attitudes required for effective communication and interaction with individuals from other cultures. It is a dynamic, developmental, and ongoing process which engages the learner cognitively, behaviorally, and affectively” (Paige, Jorstad, Siaya, Klein, & Colby, 2000, p. 50).

While these investigators acknowledge the importance of behavioral and affective aspects of learning about a different culture included in the above definition, the current investigation focused on the cognitive objective of whether FL Internet activities on the World Wide Web facilitated the learning of cultural knowledge and understanding for intermediate-level FL students. The orientation of the current study was also culture-specific, that is, the learning of French or Francophone culture. In the present study, the operational definition of culture-specific knowledge included cultural understanding, as well as the accumulation of specific pieces of information. We distinguished between the retention of cultural knowledge and cultural understanding on the premise that cultural knowledge denotes factual accumulation (e.g., French people tend to buy bread from a local bakery on a regular basis), whereas cultural understanding engages the student in reflective thinking as well (e.g., what is the role of bread in daily life; what kind of social interaction happens around the event of buying bread; and what is bread's relationship to time).

The researchers also framed the current study on the three-dimensional definition of culture in the Standards (1996, 1999). These guidelines for teaching culture suggest a classroom focus on practices, products, and perspectives. According to the most recently published Standards, practices consist of daily life interactions; products refer to “formal culture,” including artifacts (e.g., monuments); perspectives comprise meanings, attitudes, values, and ideas (see Standards, 1999, pp. 47-48). The notion of perspective is included in this investigation as students identified cultural practices and products and demonstrated their understanding of the differing perspectives (meanings) underlying these practices and products.

In spite of a current trend to include web-based activities in instructional materials, the profession knows very little about whether the Internet actually facilitates cultural learning. Contrary to the belief that Internet activities enrich the FL classroom, one could argue that the Internet might in fact not foster cultural learning. Lively, Harper, and Williams (1998) pointed out that the “very essence of the input text being imbued with native culture is what makes accessing the language in authentic documents so difficult for students” (p. 82). They urged teachers using authentic texts to intervene, or “mediate” the activity, with support materials for new vocabulary, grammar, and cultural information embedded in the text. Another possible disadvantage of a hypermedia or web environment could be cognitive overload (Chun & Plass, 2000). In this respect, Just and Carpenter (1992) emphasized that, according to informational processing theory, humans are limited in their attention and processing capacity, that is, an individual's

capacity to process information is finite rather than open ended. Consequently, if students using a FL web site focus their attention on surfing the Internet and decoding the linguistic medium, these actions could place a high cognitive demand on their working memory. As a result, their capacity to process cultural elements in the web pages could be impeded.

In view of these unresolved issues, the purpose of this study was to assess whether or not web site activities could achieve cognitive cultural goals in the intermediate-level FL classroom. It looked at the extent to which web resources presented students with a rich visual context in which to interpret and learn about cultural practices and products. Due to the cognitive potential of informational overload, the study also assessed the effectiveness of an advance organizer (AO) to enhance students' comprehension and retention of culture on the web site.

REVIEW OF LITERATURE

A review of literature suggested that the existing FL research on cultural learning in a technologically enhanced language-learning (TELL) setting has concentrated more on video than on web-based activities (cf. Adair-Hauck, Willingham-McLain, & Youngs, 2000; Bush, 2000; Herron, Corrie, Cole, & Dubreil, 1999; Herron, Dubreil, Cole, & Corrie, 2000, 2002; Kitajima & Lyman-Hager, 1998). In general, findings from these studies support the use of video to teach culture.

Two of the above studies (Herron et al., 1999, 2000) reported on the kind of culture learned: little “c” (patterns of daily behavior) and big “C” (achievements, institutions). The researchers found that elementary-level French students, who viewed FL narrative videos over the course of a semester, knew significantly more little “c” culture on pretest and posttest measures. For FR101 students, but not for FR102 students, the gain in little “c” knowledge was significantly larger than the gain in big “C” knowledge over time.

A third video/culture study cited above by those same researchers (Herron et al., 2002) added to the chain of TELL research by: (a) focusing on journalistic video, (b) including intermediate level students in the sample, (c) incorporating an assessment of the effect of an AO on students' cognitive ability to make inferences and to recall information about culture embedded in video, and (d) examining the learning of cultural knowledge and fostering of cultural understanding from authentic videos with short-answer and free recall testing. With respect to the main effect of video on the cultural knowledge of FR201 students over time, posttest scores (regardless of culture type) were statistically significantly higher than pretest scores. Therefore, results indicated that intermediate-level, college French students did in fact improve their French and Francophone cultural knowledge over the course of a semester after exposure to a curriculum with an accompanying journalistic-style video. Regarding the distinction between practices and products scores on the pre-posttest, the magnitude of the gains in cultural practices scores was not different from the products gains over the course of the semester. These results did not support the findings of previous research (Herron, Corrie, Cole, & Dubreil, 1999; Herron et al., 2000) which had found that beginning students viewing narrative video had statistically significantly higher little “c” scores

on the pretest and on the posttest and that, for FR101 students, there were greater gains in little “c” knowledge over the course of the semester. The researchers suggested that this discrepancy in findings could be attributed to the nature of the videos. For FR101 and 102, the narrative videos told an episodic story, whereas for FR201, the journalistic videos could have stressed information over behavior. For the intermediate-level students, the administration of an AO prior to video viewing did not yield a significant difference in cultural understanding between the AO and non-AO conditions on either the short-answer test items or free recall. However, the students did recall significantly more cultural practices than cultural products on the open-ended, free recall portion of the postvideo viewing tests.

As to the evaluation of the Internet as an effective source of authentic documents to teach culture in the classroom, experimental research is scarce. To some extent, the effectiveness of the Internet to teach language skills has been documented (see Lee, 1997), but the impact of Internet tools to teach culture remains largely unexplored. In spite of the lack of convincing evidence, researchers advocate the use of the Internet as a potentially effective source to teach culture because of its ability to combine different types of media (Omaggio Hadley, 2001; Warschauer, 1999; Warschauer & Kern, 2000). Walz (1998) presented various activities that can be organized using the Internet, following the guidelines set by the National Standards. However, he did not evaluate learning through these various activities.

Lee (1997) designed a study in which her students had to choose a cultural topic that they wanted to study, find the online resources they needed in order to accomplish what they wanted to do, and actually produce and present a portfolio of their work to their classmates and instructor. The evaluation determined the usefulness of the Internet tools that students had used and the degree of enjoyment they attained while doing the project. There was no assessment of the kind of culture learning that actually occurred in the process.

A second study addressed the use of the Internet in the FL classroom. Gaspar (1998) designed a study using a similar format to Lee's. Students were engaged in McKenzie's (1995) iterative research cycle (cited in Gaspar, 1998, p. 72): questioning (formulating a research question), planning (developing a strategy), gathering (data, information), sifting (assessing the information), synthesizing (making sense of the information), and evaluating (drafts and finished product). Her evaluation of the activity was conducted through students' reports and comments. Students found the Internet helpful and enjoyed doing the project. A critical need remains to assess the kind of learning that takes place from student exposure to the Internet in terms of cultural information and understanding of that information.

SIGNIFICANCE OF CURRENT STUDY

The purpose of this investigation was to investigate the effectiveness of the Internet to enhance intermediate-level French students' cultural knowledge as well as their understanding of Francophone culture. More specifically, the study continued the investigators' research into how different kinds of culture are learned

in different instructional environments. To achieve this goal, the investigators addressed the following six research questions:

1. Did intermediate French students' overall knowledge of French culture improve over the course of a semester when exposed to a curriculum with an Internet component?

2. Did intermediate French students retain more cultural practices or cultural products over the course of a semester when exposed to a curriculum with an Internet component?

3. Did AOs to the web sites enhance students' ability to make cultural inferences on short-answer tests administered immediately after exposure to the authentic web sites?

4. Did AOs to the web sites improve students' retention or comprehension of cultural information on tests of free recall administered immediately after exposure to the authentic web sites?

5. Did intermediate French students retain more cultural practices or cultural products on tests of free recall administered immediately after exposure to the Internet web sites?

6. What were intermediate French students' perceptions of how much cultural practices and cultural products were presented in the web sites and how much of that cultural information was learned?

Due to the lack of research regarding the effectiveness of the Internet to teach culture, no hypotheses were formulated for the present study.

METHODOLOGY

Participants

Fifty-four students enrolled in four sections of a 15-week, third-semester French course (FR201) at a medium-sized, private university participated in this study. The study took place during the Fall 2001 semester. Four different teachers taught the four classes (groups) of FR201. All four teachers were nonnative speakers of French. Chi-square analysis revealed no statistically significant differences between groups by gender χ² (3, N = 54) = 1.21, p = .75 or by university classification (first-year students vs. upper classmen) χ² (3, N = 54) = 7.33, p = .06. There were no differences between groups in the number of previous years of formal instruction in French F(3, 53) = .85, p = .47 (see Table 1).

General Classroom Procedures

Classes took place on the same days for all four sections (Monday, Tuesday, Wednesday, and Friday). Each session lasted 50 minutes. The instructional program for all four sections was Bravo! (Muyskens, Harlow, Vialet, & Brière, 1998), a book-based course supplemented by a videotape and a workbook with aural and written activities.

Table 1

Sample Student Characteristics by Group

	Groups
Characteristics	A	B	C	D
Gender
Number of Females Number of Males	9 3	11 2	8 4	12 5
University classification
% First-year Students % Second-year Students % Third-year Students % Fourth-year Students	67 25 8 0	62 38 0 0	75 17 8 0	30 58 0 13
Years Experience^a
Mean Standard Deviation	1.92 (0.50)	1.90 (0.63)	2.52 (0.86)	2.25 (0.61)

^aOne semester of college French was treated as equivalent to 1 year of high school experience.

Heinle and Heinle, the publishers of the textbook, maintain a web site (bravo.heinle.com) through the University of California at Santa Barbara. This web site proposes Internet activities based on authentic web sites from the Francophone world. Each book chapter has an Internet activity based on a set of web sites reflecting the themes, aspects of culture, and functions of language presented in the text, even though, like the textbook's accompanying video, they do not constitute a continuation of the textbook. The sites offer new cultural information and do not repeat cultural information in the book.

In FR201, students completed 9 of the 10 chapters in the textbook. All classes followed the same semester-long syllabus that allotted an average of four days per chapter. Eight days were set aside for the purpose of the Internet activities. On the Internet days, class activities in all four sections were similar. The class hour began with a brief conversational warm up, immediately followed by the study; that is, students were exposed to an AO to the target web site or not, depending on experimental condition. Next, all students visited the assigned web site for 30 minutes. Students were each provided with an individual computer so that they did not have to share one with another student. Students then took the posttask test (10 minutes). Afterwards, instructors collected the tests and did not discuss either the test or the web site with students. Class was dismissed. The teachers did not discuss any of the targeted web sites until after the administration of the posttest at the end of the semester.

Target Web Sites

The study included eight target web sites taken from the pool of web sites offered through the Bravo! homepage. These sites, covering a range of cultural topics, were chosen based on their availability and functionality. We took the liberty to modify the order in which they appeared in the book since one of the first sites on which students were going to work was the Institute of the Arab World, located in Paris, France. According to the original sequence, this activity was scheduled to happen shortly after the events of September 11, 2001. Since the fresh memory of the events could influence students' answers to questions, this web activity was moved to a future date.

On the assessment class day, each student received a sheet of paper with the assigned URL of the day as well as instructions regarding navigation of the site. To limit how far away from the homepage students were allowed to go, they had to stay within two or three clicks of the given homepage. Each of the four FR201 classes worked on the same web site on the same day. The first of the eight web sites was randomly assigned to either the AO condition or the non-AO condition for two French classes. That web site was then assigned to the opposite condition for the other two sections. Table 2 illustrates the counterbalancing of treatment across groups. Assignment of web sites to conditions alternated for the four classes. When summed across participants and web sites, the results are counterbalanced: each class had a total of four web sites in each condition. This counterbalanced design also compensated for the fact that random assignment of students to the four classes was not possible as classes were preconstituted by the registrar.

Table 2

Counterbalancing of Treatments Across Groups

	AO condition (AO + Internet)	Non-AO condition (Internet Only)
Internet Activity #1	Group A and Group B	Group C and Group D
Internet Activity #2	Group C and Group D	Group A and Group B
Internet Activity #3	Group A and Group B	Group C and Group D
Internet Activity #4	Group C and Group D	Group A and Group B
Internet Activity #5	Group A and Group B	Group C and Group D
Internet Activity #6	Group C and Group D	Group A and Group B
Internet Activity #7	Group A and Group B	Group C and Group D
Internet Activity #8	Group C and Group D	Group A and Group B

Teaching Procedure

All four sections followed the same procedure; we provided the instructors with

the material for the activity (AO or not, instruction sheets, posttask tests). All groups conducted the activity on the same day. It was the first time any of the students in the four sections saw the web site. In two of the four sections, the teachers provided no introductory remarks about the web site. For these sections, the students surfed the web site for 30 minutes, within the previously described restrictions, without taking notes. In the other two sections, prior to presenting the web site, the teacher administered a four-sentence AO in the form of a transparency set on an overhead projector. The sentences were in French. The teacher read aloud the four sentences once, while the students listened and read silently. Then the students surfed the web site for 30 minutes, again within the previously described restrictions, without taking notes. In all four groups, the posttask test (short answer and free recall) was administered by the class instructor immediately after the 30-minute period.

A sample AO follows.

Le site Internet pour ce chapitre présente:

1. des informations générales sur Bruxelles

2. des informations sur les activités de tourisme et loisirs à Bruxelles

3. une description du rôle de Bruxelles dans l'Union européenne

4. une description des institutions de l'Union européenne

[The web site for this chapter presents:

1. general information on Brussels

2. information regarding free-time and touristic activities in Brussels

3. a description of Brussels' role within the European Union

4. a description of the institutions of the European Union]

Testing Procedures: Pretest and Posttest

In order to permit assessment of the main effect of the Internet on cultural knowledge over time, all students in the four FR201 classes took a multiple-choice pretest at the beginning of the semester, prior to visiting the first of the target web sites. At the end of the semester, after the last Internet activity, they took a test containing the same items (posttest). The teachers informed the students that neither the pretest nor the posttest counted toward their final grades in the class. Of the 33 items on the pre-/posttest, 13 related to knowledge of cultural practices, 13 to knowledge of cultural products, and 7 grammatical items served as distracters for the purpose of reducing students' anticipation of being tested on culture. The pre-/posttest was based on cultural information contained in the target web sites. Given that the students were not exposed to any of the target web sites prior to taking the pretest, they had to answer the pretest questions solely on the basis of prior knowledge. When they took the posttest, they had been exposed to the target web sites and, therefore, to the cultural information covered.

Below are examples of pre-/posttest cultural items. (The correct answer is starred. Practices/products are indicated in parentheses).

1. What is the profession of Sonia Rykiel? (products)

a. A fashion designer*

b. An architect

c. A journalist

d. A politician

2. What kind of sports activities could a visitor to Guadeloupe not do? (practices)

a. Snorkeling

b. Mountain climbing*

c. Canyoning

d. Riding in a 4-wheeler

The design of this pre-/posttest followed that of the pre-/posttest for the video study (Herron et al., 2002). Each item consisted of a stem and four possible responses. Each test item was worth 0 or 1 point. Incorrect answers received 0 points; correct answers, 1 point. The pre-/posttest score of each participant was computed by adding the correct responses and dividing by the total number of questions. Beside the total score, the “practices” pretest/posttest scores and the “products” pretest/posttest scores were calculated by the same process.

Testing Procedures: Posttask Tests (Short Answer and Free Recall)

In order to permit assessment of their ability to recall information and make inferences from the web sites, students in all four classes took the same comprehension test immediately after visiting the web site for the day. This written test had two distinct formats: short answer and free recall. Short-answer items were designed and administered first to students to assess their understanding of the meaning (perspectives) of cultural knowledge in the web sites. The free recall portion that followed allowed students, with no guidelines, to inform researchers of what they retained from the web sites, be it facts or their interpretations of them.

For the short-answer section, students answered two questions involving the formulation of inferences based on analysis of information in the web site. No short-answer test item asked for information that had been included in the pretest or in the AO. The researchers constructed the short-answer portion of the posttask tests to assess students' ability to reflect on and analyze cultural information presented in the web site. This format allowed the researchers to assess the students' understanding of the relationship between cultural practices and products on the one hand and cultural perspectives on the other. For example, for the Internet activity #1, the students visited a web site presenting Sonia Rykiel, a fashion designer, and her work in various fashion domains. On the posttask test, the students were to explain what indications they could gather that Sonia Rykiel was a versatile designer and how they could tell whether or not the collection that was featured on the web site was up to date. After the students finished the two short-answer items stressing cultural understanding, teachers in all four sections told

them to turn the page to the free-recall test segment. The students then had three minutes to write everything that they remembered from the web site without further specification of the elements on which they should concentrate. The teachers distributed the posttask test immediately after the class finished surfing the web site. They asked students to record their answers on the test sheets. All questions were in English, and students wrote their answers in English so as not to confound their cultural understanding with their skills in writing in the FL. Each class had a total of eight posttask tests. Examples of an Internet activity and posttask test are below.

Sheet #1

Internet Activity #1

Instructions:

You will use the web site below as the “homepage” for this activity. You will be given 30 minutes to read information from this web site on your own.

NOTE: When you click on the links that this homepage offers, you will be taken to another page from this site. You can use as many links as you wish but do not go more then three clicks away from the home page.

Homepage: www.soniarykiel.fr/version3/fr_auto.htm

Sheet #2

Internet Activity #1—Posttask Questionnaire

1. Why is Sonia Rykiel considered to be a versatile designer?

2. How does one know, from looking at the Sonia Rykiel web site homepage, that it is up to date?

(Students write what else they remember about the web site on the back of the answer sheet.)

A list of criteria was established for each short-answer inference question. An answer received 0, 1, or 2 points according to the level of adequacy with respect to the list of criteria. Each question received a separate score that was computed by dividing the total number of points awarded by the total number of points possible. For example, a participant who received 1 point for each question received a score of 1/2 (i.e., .50) for each question and 2/4 (.50) for the total posttask test. A student who was awarded 2 points for each question received a score of 4/4 (i.e., 1.00). Each posttask test was scored blindly by two judges in order to establish an unbiased estimate of interrater reliability. Both judges scored all posttask viewing tests. The interjudge reliability for the short-answer questions was 90%. Disagreements were identified and discussed, and the relevant tests were re-scored accordingly.

The material on the back of the posttask tests was scored along lines suggested by various researchers (e.g., Bernhardt, 1983; Bernhardt & James, 1987) who recommended grading free recall in an objective fashion by counting and awarding 1 point to any information (in this case, cultural) listed by students that legitimately

appeared in the web site. This system has the advantage of giving students equal credit for every item they remembered as opposed to penalizing them for missing what the teacher deemed important.

Scoring the free-recall tests followed a specific sequence. First, the investigators calculated the total number of cultural responses recalled by the students per web site. This initial classification of all responses per web site into “cultural” or “not cultural” was cross-checked by a second investigator. Interrater agreement was 91%. Items that were not cultural were discarded from further analysis. For example, investigators did not count the response: “Sonia Rykiel has a daughter.” They did count the response: “Sonia Rykiel's daughter, Nathalie, is now the director of design for Sonia Rykiel.” Once these cultural elements were identified, the researchers categorized the elements into cultural practices and cultural products. This classification was cross-checked with a rate of agreement of 90%. When disagreement occurred, the investigators discussed and reached consensus concerning how to count the particular response.

Second, each student had two sets of two scores. One set represented how much a student freely recalled in the AO versus non-AO condition. The other set represented how much a student freely recalled in terms of cultural practices and products. All individual scores were calculated by dividing the number of cultural responses that a student freely recalled by the total number of cultural responses freely recalled by all students per web site. For example, for Internet Activity #4, students freely recalled and listed a total of 30 cultural responses. The first student freely recalled four of those responses and, therefore, received a score of 4/30 or .13 for that web site. That score was entered as an AO or non-AO score depending on whether the student had viewed the web site in the AO or non-AO condition. Next, the responses of students were coded as cultural practices or cultural products. A similar proportion was calculated. For example, of the four responses listed by the first student for Internet Activity #4, three of the freely recalled items were categorized under cultural products and one item under cultural practices. The products score was computed as 3 divided by 30 (total cultural responses freely recalled by all students for Internet Activity #4) or .10. The practices score was computed in the same fashion as 1 divided by 30 or .03. Final computations consisted of calculating mean scores for each student over all target web sites in each condition (AO vs. non-AO) and for each type of culture (practices vs. products).

Testing Procedures: Culture Questionnaire

At the conclusion of the semester, students completed a cultural questionnaire. The questionnaire began with a definition of cultural practices and products designed to help answer the items. This questionnaire included open-ended questions about students' perceptions of the kind of culture they thought was presented on the web sites and how much culture they thought they had actually learned from the web sites. It also included items on how comfortable they felt working with the web sites and how much they enjoyed the activities. The students answered questions

regarding previous experiences in France or previous exposure to French (and, if so, under what circumstances). They were asked if they had ever used web sites in French class before and under what conditions. Also, they were asked if they surfed the internet and if they went to French or Francophone web sites.

RESULTS

Before analyzing data pertaining to the research questions, the investigators calculated item difficulties for the pre-/posttest and the posttask tests. For the pre-/posttest, item difficulty was calculated for questions regarding cultural practices and products separately. Researchers ran t tests for independent samples in order to reveal possible differences between mean difficulty scores for the pre-/posttest items (13 items for each type of culture). There was no significant difference between the mean difficulty for cultural practices (M = .43, SD = 0.23) and cultural products (M = .31, SD = 0.13) items, t(24) = 1.63, p = .12. The mean for practices was slightly higher than the mean for product but the difference was not statistically significant and there was no risk of either floor or ceiling effect. Analysis of each item revealed that no item had a difficulty higher than .90 or lower than .10. Therefore the investigators kept all items in the pre-/posttest scores for all further analysis. Researchers also assessed item difficulty on the posttask tests measuring cultural understanding. No item had a difficulty higher than .90 or lower than .10. Therefore, all items were retained on the posttask tests for further analysis.

To assess whether students processed cultural information (and therefore whether changes in overall cultural knowledge over the course of the semester occurred) and to assess possible differences in long-term learning of cultural practices and products, investigators subjected pertinent data to a 2 (pre, post) X 2 (type of culture) repeated measures analysis of variance (ANOVA). Data for 3 of the original 54 participants were removed from the analysis because they had missed either the pretest or the posttest.

With respect to the main effect of the Internet on cultural knowledge over time (Research Question 1), posttest scores (regardless of culture type) were statistically significantly higher than pretest scores F (1, 50) = 75.64, p < .001. The effect size for this analysis was η² = .60. The main effect of culture type was statistically significant (F (1, 50) = 13.49, p < .001) for both types of culture, and the culture x time interaction (Research Question 2) was also statistically significant (F (1, 50) = 10.21, p < .01) largely due to the increase in cultural products scores. The effect sizes for these last two analyses were respectively .21 and .17. This means that students learned significantly more cultural products than cultural practices over the course of the semester. Table 3 presents pre-posttest means and standard deviations.

To explore further the relationship between pretest and posttest performance, we computed Pearson product-moment correlations between pretest and posttest performances for the total scores and for each type of cultural information (practices and products). No statistically significant correlations were found between performance on the pre-/posttest and other variables (e.g., previous experience with French).

Table 3

Pre- and Posttest Means (and Standard Deviations) Overall and by Type of Culture

	Pretest	Posttest
Practices	.42 (0.11)	.51 (0.12)	.47 (0.11)
Products	.31 (0.14)	.49 (0.16)	.40 (0.15)
Total	.37 (0.08)	.50 (0.09)***

***Posttest scores higher than pretest scores, p < .001

To assess the effectiveness of an AO on students' ability to make inferences about culture (Research Question 3), a t test for paired samples was used for the 53 participants who had complete data. A paired t test takes advantage of the fact that each student performed in both the AO and non-AO conditions. For the short-answer portion of the posttask tests, the total AO mean score (M = .58, SD = 0.13) was not significantly different from the total non-AO mean score (M = .61, SD = 0.15), t(52) = 1.02, p = .32, effect size = .17.

Regarding the effect of AOs on students' ability to recall freely cultural information from the web sites (Research Question 4), again a t test for paired samples was employed to assess differences in mean proportion correct scores between the AO and the non-AO conditions. For the free-recall portion of the posttask tests, the total AO mean score (M = .14, SD = 0.07) was not significantly different from the total non-AO mean score (M = .13, SD = 0.06), t(52) = 0.96, p = .34, effect size = .13.

For Research Question 5 concerning the kind of culture freely recalled after viewing each web site, investigators conducted a t test for paired samples. The mean score for cultural practices (M = .07, SD = 0.04) was not significantly higher than for cultural products (M = .07, SD = 0.07), t(52) = 0.90, p = .37, effect size = .12. Table 4 presents the mean scores for practices and products for all subjects on the free-recall portion of the posttask web site tests.

Table 4

Free Recall Mean Proportion Correct (and Standard Deviations) by Type of Culture for Posttask Tests

	Type of Culture
	Practices	Products
Posttask Free Recall Tests	.07 (0.04)	.07 (0.07)

p = .37

To determine the association between performance on the posttask tests (short answers and free recall) and other variables (e.g., previous experience with French, travel to France), we used Pearson product-moment correlations. No statistically

significant correlations were found between posttask performance and other variables.

Regarding student perceptions (Research Question 6), 47 of the 54 students completed the cultural questionnaire. Of these students, 68% thought that a lot or vast number of cultural products appeared in the web sites, and 45% thought that a lot or a vast number of cultural practices appeared. Moreover, 28% of the students believed that they learned a lot or a vast amount about cultural products throughout the semester, 26% of the students believed that they learned a lot or a vast amount about cultural practices. To test differences between type of culture presented and type of culture learned, t tests for paired samples were conducted. Students believed that more cultural products (M = 3.72, SD = 0.62) were presented than cultural practices (M = 3.38, SD = 0.74), t(46) = 3.07, p < .01, effect size = .93. However, they did not believe that they learned more about cultural products (M = 3.17, SD = 0.60) than cultural practices (M = 3.0, SD = 0.78), t(46) = 1.59, p = .12, effect size = .17.

Part of the information collected from students pertained to use of the Internet in a FL classroom, their enjoyment of the Internet activities, their level of comfort with computer use, previous exposure to French, previous travel to France, and previous exposure to a foreign culture. Of the 47 students that provided information on these questions, 16 (34%) reported having been previously exposed to a French web site on the Internet (either through a teacher-controlled class activity or on their own). Thirty-one students (66%) had never seen a French web site prior to this study (neither in nor outside the FL classroom). Thirty-seven students (79%) reported having enjoyed the Internet activities and all 47 students (100%) reported being comfortable or very comfortable with using the Internet. All students had received between one and three years of French instruction prior to this study, and 15 (32%) had also received instruction in another foreign language. Eleven students (23%) reported having been exposed to a foreign culture in a significant manner before (born abroad or having a foreign parent). Finally, 19 students (40%) reported having visited France prior to this study, most of them (with the exception of 3 who spent 6-8 weeks there on a summer study abroad program) for a week or less. From the correlational analysis, it seems that none of these variables had an impact on performances on the various tests.

DISCUSSION

Before drawing conclusions from this study, we wish to stress the constraints and limitations of the research. We did not take into account teachers' views regarding the goals of foreign language education, or their perspective on the teaching of culture as well as the degree of prioritization of culture in the classroom. We did not compare the teaching of culture via the internet with other possible ways of presenting culture (e.g., reading magazines). Furthermore, the lack of experimental research on the effectiveness of the Internet to teach culture in the FL classroom prevented us from making directional or “educated” hypotheses regarding the outcome of this investigation. Finally, it should be remembered that all comparisons between this study and other previous relevant studies were put forth as

suggestions rather than findings. Even though the investigators addressed the same six research questions in the previously mentioned study on intermediate-level students' cultural acquisition from journalistic videos (Herron et al., 2002), and although the basic instructional program remained the same, Bravo! (Muyskens et al., 1998), a direct comparison between the current Internet study and the previous video experiment could not be made because the students, teachers, and the technological tools assessed differed. A similar design, however, allowed for interesting speculations on how much and what kind of culture students gained from each medium and if it only allowed for hypotheses, these hypotheses did yield possible directions for future research. Keeping these limitations in mind, we suggest the following conclusions in response to the research questions.

Question 1: Did intermediate French students' overall knowledge of French culture improve over the course of a semester in a curriculum with an Internet component?

Results of the current study indicate that intermediate-level college French students did improve their French and Francophone cultural knowledge over the course of a semester in a curriculum with an Internet component. Since none of the demographic variables were significantly correlated with performance on the posttest, these variables did not help in providing explanations for the improvement in cultural knowledge. These findings encourage teachers to enhance the cultural aspect of intermediate-level FL college programs by using a curriculum that includes an Internet component. It is important to remember that in our study, the Internet activities took place in the classroom on individual computers (one per student), with constraints such as how far and how long each student could “surf.” While teachers remained in the classroom during the Internet activity, they did not interrupt or intervene in the students' work. It is our contention, based on experience and previous research, that cultural gains could have been even higher than those observed if students had received more guidance. When preparing Internet activities using authentic web sites, teachers could plan lexical and cultural explanations to facilitate navigation and comprehension so as to be able to give even more control to the student in the learning process and to teach within a more learner-centered, task-based approach. In addition to learning culture over the course of the semester, students indicated that they had enjoyed visiting the web sites and that the Internet was a learning tool with which they were very comfortable.

Question 2: Did intermediate French students retain more cultural practices or more cultural products over the course of a semester in a curriculum with an Internet component?

With respect to the distinction between practices and products scores on the pre-/posttest, the magnitude of the gain in cultural products scores was significantly higher than the gain in cultural practices over the course of the semester. These results were not consistent with previous research on the learning of culture through

video. Researchers (Herron, Corrie, Cole, & Dubreil, 1999; Herron et al., 2000) had found that beginning students had statistically significantly higher little “c” scores than big “C” scores on pretests and on posttests and that for the FR 101 students there were greater gains in little “c” than in big “C” knowledge over the course of the semester. Even with intermediate students viewing video, although not significantly so, the gains in cultural practices were larger than the gains in cultural products (Herron et al., 2002). These conflicting results could be related to the following hypothesis. After viewing all the video in the FR101, 102, and 201 projects and the Internet sites in the FR201 study, we posit that the Internet seems to foster a different type of cultural learning than videos. Images on video appeared to highlight cultural behavior (i.e., practices) while authentic materials on the web sites appeared to highlight more static cultural images and artifacts (i.e., products). It is possible that the apparent absence of performance of cultural behaviors on the web sites (as opposed to videos) in favor of static images leads students to focus more on discrete facts and information than patterns of daily living and habits talked about or alluded to on the web sites. In the video component, even though journalistic in nature, students still got to see people behave in an authentic cultural setting. This was not the case with the web site where cultural practices had to be inferred from the information presented on the web site as opposed to the video where students could witness these practices.

Question 3: Did AOs to the web sites enhance students' ability to draw correct cultural inferences in short-answer tests administered immediately after exposure to the authentic web sites?

Regarding short-term retention, the administration of an AO prior to visiting the web site did not reveal a significant difference in cultural understanding between the AO and non-AO conditions. This finding is consistent with previous research on video indicating that an AO that describes major points in an upcoming video did not appear to facilitate cultural understanding and reflection for intermediate-level students (Herron et al., 2002). However, a close look at the mean proportion of correct responses on the Internet short-answer inferential tests revealed that, on average, students answered approximately three of every five inferential items correctly on the Internet tests. Student performance did not appear to be overly handicapped by short-answer items that required them to reflect on the perspectives underlying the cultural practices and products to which they were exposed. This reflection entails a higher level of thinking than knowledge of cultural facts. For example, Internet Activity #5 dealt with a web site that offers information about cinema, film releases, box office, films reviews and trailers, television programs, and so forth. When asked to comment on how this web site suggested cinema was marketed in France, several students described how French people seem to focus on the director of the film whereas Americans' focus lies more on the star system and on the actors that play in a particular film. This point denoted not only attention to details but also demonstrated the ability to understand cultural perspectives. Correlational analyses disclosed no association between student performance on the inferential items of the posttask tests and other variables.

Question 4: Did AOs to the web sites improve students' retention or comprehension of cultural information on tests of free recall administered immediately after exposure to the authentic web sites?

The administration of an AO prior to visiting the web site did not enhance the intermediate FL students' free recall of culture embedded in the web sites. It should be emphasized, however, that, whereas the AO treatment with intermediate-level FL students in the current study (descriptive statements summarizing the web site's topics) and in the previous Herron et al. (2002) study (descriptive statements summarizing the video's topics) did not seem to enhance cultural reflection, a different AO treatment could benefit comprehension. This issue should be examined in future research.

Question 5: Did intermediate students retain more information about cultural practices or cultural products on tests of free recall administered immediately after exposure to the authentic web sites?

Students recalled an equivalent amount of cultural practices and cultural products on the open-ended, free-recall posttask tests. This result did not support the initial finding for Research Question 1 that students learned significantly more products than practices from pretest to posttest. Perhaps on the free recall students were less influenced than on the posttest by the manner in which culture is presented on the Internet, which is what we alluded to in our observation that the Internet appeared to present culture statically (more product-oriented) rather than dynamically (more practice-oriented). In future research, a detailed content analysis of each web site for practices and products, prior to its use, might shed some light on why students' performance on free recall yielded these results. Even without an extensive analysis of the content of the authentic Internet materials, it appears that the cultural practices in the web sites were described or suggested rather than demonstrated in actions. This observation could help to explain why the intermediate students, when asked to recall freely the most salient aspects of videos, favored cultural behaviors and patterns of daily life over products (Herron et al., 2002). It is possible that the Internet fosters a different kind of learning and that in the absence of representation of cultural behaviors, students' focus might shift onto more discrete types of information, such as names, historical events, objects, and so on, typically falling in the category of cultural products.

Question 6: What were intermediate French students' perceptions of how much cultural practices and cultural products were presented in the web sites and how much of that cultural information was learned?

Analysis of the data indicated that the students believed that significantly more cultural products than practices had been presented. Similarly, although the trend was not statistically significant, students tended to believe that they had learned more about cultural products than cultural practices. These findings from the student opinion poll in favor of products are in line with the results above that students performed significantly better on cultural products than practices items on

the posttest and tended to favor cultural products on tests of free recall. This finding, however, is in contradiction with findings on the previous intermediate-level study measuring cultural acquisition from video (Herron et al., 2002) and previous research on the effectiveness of video to teach culture to beginning French students (Herron, Corrie, Cole, & Dubreil, 1999; Herron et al., 2000). In these studies, students believed that they had learned more about cultural practices than cultural products from videos. Although the exact reason for these differences remains elusive, these results suggest that teachers should remember students' perceptions and preferences when choosing a medium to prepare for culture activities. Students tend to perceive video as a learning tool for its propensity to present cultural practices and the Internet for its ability to present cultural products. In this respect, it is possible that video could be a more effective pedagogical tool to introduce cultural practices in the classroom, whereas the Internet tends to be more effective to present students with cultural products.

CONCLUSION

While previous experimental studies (Herron, Corrie, Cole, & Dubreil, 1999; Herron et al., 2000; Herron et al., 2002) suggested that video fosters the acquisition of cultural practices over cultural products, this study provided evidence that the Internet may constitute a privileged medium for the teaching and learning of cultural products. These results have to remain within the scope of using the Internet as a source of cultural information within the classroom as opposed to a networked community. In the latter case, through interaction with natives of the target culture, students may engage in inquiries that lead them to formulate hypotheses about cultural practices in the target culture. This possibility was not within the scope of the present endeavor, but more research is clearly needed in this area. We hope that the current study will continue to foster a chain of classroom research on how best to teach culture at all levels and how best to use the techniques and technologies that we have at our disposal, a sort of cultural “teachnology.”

REFERENCES

Adair-Hauck B., Willingham-McLain, L., & Youngs, B. E. (2000). Evaluating the integration of technology and second language learning. CALICO Journal, 17, 269-306.

Bernhardt, E. B. (1983). Testing foreign language reading comprehension: The immediate recall protocol. Die Unterrichtspraxis, 16, 27-33.

Bernhardt, E. B., & James, C. J. (1987). The teaching and testing of comprehension in foreign language learning. In D. W. Birckbichler (Ed.), Proficiency, policy, and professionalism in foreign language education (pp. 65-81). Lincolnwood, IL: National Textbook Company.

Bush, M. D. (2000). Digital versatile disc (DVD): The new medium for interactive video. CALICO Journal, 17, 453-474.

Chun, D. M., & Plass, J. L. (2000). Networked multimedia environments for second language acquisition. In M. Warschauer & R. Kern (Eds.), Networked-based language teaching: Concepts and practice (pp. 151-170). Cambridge: Cambridge University Press.

Gaspar, C. (1998). Situating French language teaching and learning in the age of the Internet. French Review, 72, 69-80.

Herron, C., Corrie, C., Cole, S. P., & Dubreil, S. (1999). The effectiveness of a video-based curriculum in teaching culture. Modern Language Journal, 83, 518-533.

Herron, C., Dubreil, S., Cole, S. P., & Corrie, C. (2000). Using instructional video to teach culture to beginning foreign language students. CALICO Journal, 17, 395-429.

Herron, C., Dubreil, S., Cole , S. P., Corrie, C. (2002). A classroom investigation: Can video improve intermediate-level French language students ability to learn about a foreign culture? Modern Language Journal, 86, 36-53.

Just, M., & Carpenter, P. A. (1992). A capacity theory of comprehension: Individual differences in working memory. Psychological Review, 99, 122-149.

Kitajima, R., & Lyman-Hager, M. A. (1998). Theory-driven use of digital video in foreign language instruction. CALICO Journal, 16, 37-48.

Lee, L. (1997). Using Internet tools as an enhancement of C2 teaching and learning. Foreign Language Annals, 30, 410-427.

Lively, M. G., Harper, J., & Williams, J. K. (1998). Mediating language with teacher talk: Bringing speech to ideas. In J. Harper, M. Lively, & M. Williams (Eds.), The coming of age of the profession (pp. 81-96). Boston: Heinle & Heinle.

Muyskens, J. A., Harlow, L. L., Vialet, M., & Brière, J.-F. (1998). Bravo! (3rd ed.). Boston: Heinle & Heinle.

Omaggio Hadley, A. (2001). Teaching language in context (3rd. ed.). Boston: Heinle & Heinle.

Paige, R. M., Jorstad, H. L., Siaya, L., Klein, F., & Colby, J. (2000). Culture learning in language education: A review of the literature. In R. M. Paige, D. L. Lange, & Y. A. Yershova (Eds.), Culture as the core: Integrating culture into the language curriculum (pp. 47-113). Minneapolis, MN: Center for Advanced Research on Language Acquisition.

Standards for foreign language learning: Preparing for the 21^st century. (1996). Yonkers, NY: Standards for Foreign Language Learning Project.

Standards for foreign language learning in the 21^st century. (1999). Yonkers, NY: National Standards in Foreign Language Education Project.

Walz, J. (1998). Meeting standards for foreign language learning with World Wide Web activities. Foreign Language Annals, 31, 103-114.

Warschauer, M. (1999). Electronic literacies: Language, culture, and power in online education. Mahwah, NJ: Lawrence Erlbaum Associates.

Warschauer, M., & Kern, R. (Eds.). (2000). Networked-based language teaching: Concepts and practices. Cambridge: Cambridge University Press.

ACKNOWLEDGMENTS

We would like to thank Dr. Cathleen Corrie and Dr. Nicole Mills for their invaluable assistance in collating data for our investigation on using technology at the intermediate-level.

AUTHORS' BIODATA

Sébastien Dubreil is an Assistant Professor of French and the Director of the French Language Program at the University of Notre Dame. He is a fellow of the Nanovic Institute for European Studies. His research focuses on culture learning. He is especially interested in assessing the effectiveness of multimedia technologies in the teaching/learning of culture in the FL classroom. He has also investigated the redefinition of the roles of the learner and the teacher in TELL environments. He teaches and works on French and Francophone cinema. His articles appear in such publications as The Modern Language Journal and the CALICO Journal.

Carol Herron is Professor of French at Emory University. She is currently the Director of the French Language Program and the Emory Summer Study Abroad Program in Paris, France. She also directs the joint doctoral program in French/Educational Studies. She is the recipient of the Arthur M. Blank/National Endowment for the Humanities Distinguished Teaching Chair in French. Her research focuses on foreign language acquisition and more specifically on how technology can facilitate language learning. Her articles appear in such publications as The Modern Language Journal, French Review, Foreign Language Annals, and Studies in Second Language Acquisition.

Steven P. Cole is Director of Research at Research Design Associates, Inc., Yorktown Hts., NY, and Adjunct Professor in the Department of Psychology at Emory University. He earned a Masters Degree in Developmental Psychology from Columbia University and a Ph.D. in Human Experimental Psychology from Emory University. His research interests include perceptual and memory processes, and research design and statistics. His articles appear in such journals as The Modern Language Journal, French Review, Journal of Social Psychology, and Perceptual and Motor Skills.

AUTHORS' ADDRESSES

Sébastien Dubreil

Department of Romance Languages and Literatures

343 O'Shaughnessy Hall

University of Notre Dame

Notre Dame, Indiana 46556

Phone: 574/631-6540

Fax: 574/631-3493

Email: sdubreil@nd.edu

Carol Herron

Department of French and Italian

Emory University

Atlanta, GA 30322

Phone: 404/727-7944

Fax: 404/727-4579

Email: cherron@emory.edu

Steven P. Cole

Research Design Associates, Inc.

1315 Baptist Church Road

Yorktown Heights, NY 10598

Phone: 914/962-0610

Fax: 914/962-4716

Email: spc@cloud9.net

7. Retrofitted Materials for WebCT: Guidelines for Authors, Web Designers, and Users

TONY HOUSTON
University of Missouri-Rolla

Abstract:
With the advent of web-based programs that automate the scoring of exercises, materials are now being developed from the outset with the capabilities of the web in mind. Materials developed without consideration of the web's capabilities, on the other hand, must be retrofitted. Texts that are retrofitted to the web have the advantage of being pedagogically, not technologically, driven. With adequate attention paid to the pedagogy of adapting such texts to the web, these texts can compete on their own merits with products that are designed from the outset for use on the web. This paper discusses some of the complications of working with retrofitted materials and presents some solutions to common problems.

KEYWORDS

Web, Retrofitting Materials, WebCT, Pedagogy

INTRODUCTION

A frequently heard mantra in the teaching profession is that pedagogy, not technology, should drive our decisions. With the advent of web-based programs that automate the scoring of exercises, materials developers are now bearing in mind the capabilities of the web from the outset. Materials developed without consideration of the web's capabilities, on the other hand, must be retrofitted. As Feustle (2001) observes, the question of whether retrofitted electronic versions of workbooks are superior to their paper counterparts remains to be seen. In order to tap into the potential of the web, he advises that commercial publishers consider the possibilities and limitations of the web when designing materials so that these materials do not have to be retrofitted later. Although retrofitting can yield an awkwardly adapted product, it has the advantage of making materials available for the web that are pedagogically, not technologically, motivated.

Web-based course management programs can liberate instructors from the tedious task of grading student workbooks, saving valuable instructor time better

657

spent on research, curriculum development, and lesson planning. Although there are many such programs available, WebCT and Blackboard have been the leading choices. Deciding which program to use is usually in the hands of administrators, not instructors. For those instructors who have been using WebCT, the present paper offers guidelines on the use of the quiz module, especially in conjunction with retrofitted materials. It also offers suggestions for ways in which authors and the designers who retrofit their materials to the web can work together to improve materials.

Two important advantages of web-based programs and materials are to liberate instructors from tedious grading so that their talents can be applied to developing more effective ways of teaching and to free up class time that might otherwise be spent on reviewing homework. Despite these advantages, there a number of limitations in such programs. In this paper, I present a number of technical challenges to retrofitting texts to the web and describe what adapters and adopters of these materials can do to overcome some of the limitations of the WebCT quiz tool. The examples that follow are primarily from a customized version of Feustle's (2004) WebCT adaptation of ¿Sabías que …?. The customization involved modifications that are described in the sections that follow. Although the examples come primarily from one text in one course management tool, many of the challenges apply to other texts and to Blackboard as well.

TECHNICAL CHALLENGES

Technical Challenge #1: Free Expression Questions

The question formats useful for language instruction are matching, multiple choice, short answer, and paragraph. Grading is automated with all of these except the paragraph format. Feustle (2001) observes that course management tools such as WebCT and Blackboard are ill suited to deal with free expression in the target language. Although the grading of these questions cannot be automated, Feustle points out that typed assignments are easier to read than hand-written ones. Another advantage is that once the score is assigned, it transfers automatically to the online grade book. The disadvantage to having free expression questions online relates back to the issue of instructor time. The tedium of correcting free written expression online can be prohibitive. This problem is exacerbated when lower level instruction is relegated to teaching staff whose university-provided computer equipment is slow or obsolete. To grade free expression questions, instructor must click on each submission, wait for it to load, read it, enter a grade, and submit the assignment. It may actually be more efficient to grade such exercises on paper.

To customize the publisher's WebCT product for use at Saint Louis University, we took assignments that require instructor intervention off line and copied them to an organizer page. Because free expression activities in ¿Sabías que …? tend to be labeled para entregar 'to hand in,' we used that same label for the organizer page. On the organizer page are links to the activities in Word. If the purpose of using an online workbook is to save instructor time, not using the WebCT

658

paragraph format is an option worth considering. Instructors who prefer to grade such exercises on paper can still require learners to type their responses, although learners may find the expense of printing daily homework objectionable. We have retained these activities in the electronic workbook for instructors who prefer online grading.

Adopters working with retrofitted materials who are interested in generating more automated material may wish to convert some paragraph questions to short answer, matching or multiple choice. To change the format of a question, it is necessary to generate a new question in the desired format. Using the cut and paste function, it is possible to transfer the direction lines to the new question without losing the hypertext that encodes any formatting or links to sound files. It is crucial to transfer any associated images, shown in the image dialogue box, to the new question. Figure 1 shows a link to a sample paragraph question.

Figure 1

Sample Paragraph Question

When converting a paragraph question to an automated format, it may be necessary to select the setting "Release the score once the quiz has been submitted." In the WebCT products we used, the designer selected the setting "Release the score once the quiz has been submitted and all the questions have been graded" with paragraph questions. The designer has also programmed the submission message "Your instructor will have to grade this quiz." It may be necessary to delete any such submission messages that are no longer relevant when automating quizzes.

We allowed two attempts for each activity and accepted the average of the two scores. Students see the correct answer after their first attempt, so most manage

659

to score 100% on the second attempt. By accepting the average score, we encouraged learners to make a genuine attempt the first time rather than guessing the first time just to trigger the answer prompts. Because paragraph questions require instructor intervention, we allowed only one attempt for quizzes containing paragraph questions. Figure 2 shows the link to question settings.

Figure 2

Quiz Settings

It is sometimes possible to replace paragraph questions with automated question formats. The following activity was rendered as a paragraph question in the original WebCT product. It asks learners to state their names and the names of their instructor and another classmate.

Lección preliminar

¿Quién eres?

ACTIVIDAD A Nuevos amigos

You have probably spent the first day or two of Spanish class getting to know people in the class. Can you remember some names? Answer the following questions by filling in the appropriate names.

1. ¿Cómo te llamas?

2. ¿Cómo se llama tu profesor(a) de español?

3. ¿Cómo se llama otra (another) persona en la clase?

By modifying the direction lines to read "Answer the following questions in complete sentences," it is possible to render this paragraph question in a short answer format. By using the "answer contains" setting rather than "answer equals,"

660

the designer can program the answer key to check for whether the target form appears in the answer. Such a modification, however, is out of sync with the intent of this activity. This activity was clearly intended to check the learner's knowledge of names and was designed to be evaluated by the instructor. Instructors still have the option of overriding a score that does not reflect a thoughtful response, but, once the grading is automated, they will almost certainly opt not to do so. Although modifying such activities would generate more automated material, care should be taken not to subvert the author's intent when converting paragraph questions to short answer.

Technical Challenge #2: Gratuitous Production

It may not always be advisable to convert paragraph questions to short answer just for the purpose of generating more automated material. When the goal is also to eliminate gratuitous production, however, it may be worthwhile to convert some paragraph questions to multiple choice or matching. We found that the following paragraph question worked well as multiple choice.

Lección 1

Durante la semana

Actividad B La rutina del presidente

Below are a number of activities that the President of the United States might do in a given week. Finish each with the phrase that indicates the frequency with which he performs each activity.

MODELO: Se levanta temprano frecuentemente.

Duerme ocho horas raras veces.

In the modified activity, we provided the following expressions of frequency as alternatives. Because the question asked for the learner's opinion, full credit was awarded for any one of the following expressions of frequency:

a. siempre

b. frecuentemente

c. a veces

d. raras veces

e. nunca

661

Although, again, care should be taken not to subvert the author's intent when adapting free expression questions to automated formats, reducing gratuitous production allows learners to convey the same information in a format that does not require instructor intervention to grade. In the case of ¿Sabías que …?, the modified activity above is similar to other opinion questions found throughout the text. Rendering a free expression question as a free expression question is clearly the most faithful representation of the original text. In the absence of guidelines for designers with respect to reasonable parameters for creative freedom, these decisions will fall to adopters.

Technical Challenge #3: Questions that Depend on World Knowledge

We converted the following activity, like the one above, from paragraph to multiple choice to automate the grading and to reduce gratuitous production. For this activity we found it necessary to modify the direction lines to eliminate the words "with a written response." Rather than writing out their responses, students select either Sí or No. The activity below demonstrates the problem of questions that depend on learners' real-world knowledge.

Lección preliminar

Más sobre las clases

Actividad G ¿En tu universidad …?

Based on what you know about your university, answer each question that you hear on the audio program with a written response.

The audio script reads as follows:

1. ¿Hay clases de química orgánica?

2. ¿Hay cafetería?

3. ¿Hay clases de economía?

4. ¿Hay clases de japonés?

5. ¿Hay clases de geografía?

In the activity above, we adjusted the answer key to award full credit only for the correct answer but accepted the higher score out of two attempts so that there would be no consequences for an incorrect first attempt. The information, based on knowledge of the campus, will vary from one institution to another. Adopters will have to customize activities that depend on campus-specific knowledge to their own institutions. For purposes of scoring, adopters will have to consider whether they want to make students responsible for knowing or researching the information.

Technical Challenge #4: Absolute Precision Demands

Another limitation of Blackboard and WebCT is that short answer questions require absolute precision of the student. As Feustle (2001) explains, technologically savvy teachers have become accustomed to programs such as BASIC, PILOT,

662

Hypercard, and ToolBook that deal better with free expression than course management tools such as WebCT and Blackboard. With these course management tools, the designer must anticipate multiple correct or partially correct responses in order to develop a product that is tolerant of minor errors.

One of the tenets of communicative language teaching is that formal accuracy "in the beginning stages should be neither required nor expected" (Savignon, 1997, p. 29). Although the precision demands of WebCT and Blackboard are out of step with the tenets of communicative language teaching, the knowledgeable adopter can overcome this limitation in a number of ways. One partial solution is to reduce gratuitous production in short-answer questions as shown below. The original WebCT product rendered the following activity as a series of short answer questions although students' task was to make a binary choice. We modified it using the multiple-choice question format.

Lección preliminar

Más sobre las clases

ACTIVIDAD A ¿Oscar el optimista o Pedro el pesimista?

Below is a list of statements. Decide if each is made by either Oscar (O) the optimist or Pedro (P) the pessimist.

¿O o P?

The above activity is a short-answer question that we converted to multiple choice. We converted the following activity from short answer to matching. In the modified version, students were instructed to "match the name to the number of credit hours."

Lección preliminar

Más sobre las clases

ACTIVIDAD D Horarios

Paso 1. Listen as each person says his or her name and how many credit hours he or she is taking. Write down the information below.

Besides reducing gratuitous production, another strategy for overcoming Web-CT's absolute precision demands is to favor the "answer contains" over the "answer equals" setting and to limit the text in the answer key to target forms. Using this technique, the designer/user can reduce the incidence of points taken off for extraneous errors and extra spaces in the student responses. By providing the full

663

text of the correct response in the general feedback box, the designer/user allows students to self-check any extraneous errors that the answer key permits. Figure 3 demonstrates this technique. The activity is from the customized version of ¿Sabías que …? (Lección dos: ¿A quién le gusta?, ACTIVIDAD I Isabel y sus amigos).

Figure 3

Short Answer Feedback

Technical Challenge #5: Long Fill-in Passages

Short answer questions in Blackboard and WebCT do not allow in-line placement of answer boxes within texts, although the technology to allow in-line placement exists (Mallard, developed at the University of Illinois). The Blackboard and WebCT systems organize text answer boxes for short-answer questions vertically down the page. As a result, the questions often scroll off the screen before learners can enter all of the answers. One solution to this problem is to put the text on a separate web document linked to the quiz. It is necessary to code the link such that the document opens in a separate browser window.

Technical Challenge #6: Affective Questions

One feature of ¿Sabías que …? that Feustle (2001) observes with respect to its adaptation to WebCT is "this text's penchant for questions for which either every answer is correct or there is no correct answer at all" (p. 843). In the scholarship on structured input, such activities are called affective activities. Referential activities, in contrast, have one correct answer and the choice indicates having processed a form correctly. In structured input activities, "grammatical form contains meaning and learners must attend to form in order to complete the task" (Lee & VanPatten, 2003, p. 143). Farley (2005, p. 95) offers the following activity as an example of referential structured input.

664

Activity A Referential SI Activity Items: Object marker a

This activity requires that learners pay attention to the formal features of the language in order to provide a correct interpretation. In affective activities, the correct answer depends on the learner. Farley (2005, p. 95) offers the following example:

Activity B Referential SI Activity Items: Object marker a

Cierto Falso A la generación X no la comprenden los viejos.

Cierto Falso A los viejos no los respetan los jóvenes.

As Feustle (2001) notes, questions for which there is no incorrect response make poor use of WebCT's scoring capabilities. Newer materials, such as the Sol y viento workbook (VanPatten, Leeser, Keeting, & Houston, 2005) are designed with the web in mind and include only questions with right or wrong answers. The inclusion of activities with affective questions, however, does not mean that ¿Sabías que …? is ill suited for adaptation to the web. When scoring an activity in which the correct response depends on learners, WebCT does exactly what the instructor would do when grading a paper submission--to check to see whether the work has been completed and to trust that the learner has responded thoughtfully. In the original version of the WebCT activity above, each possible answer was allocated 100% of the total points possible for the questions, but learners may choose only one answer.

The value of structured input activities is well established in scholarship. Although the research has established a direct relationship between language acquisition and structured referential input only, Farley (2005) recommends that affective structured input activities be used in conjunction with referential ones. He notes that affective structured input activities allow learners to apply the target forms in a way that is personally meaningful.

Affective activities in the WebCT manual for ¿Sabías que …? are most often rendered as multiple-choice questions. Feustle (2001) uses the following activity as an example. It asks the learner to make a logical judgment.

Lección nueve

Las bebidas

Actividad B ¿Lógico o absurdo?

¿Te parece lógico beber café a las once de la noche? Escucha las situaciones en el programa auditivo y indica si las decisiones que se toman son lógicas o absurdas (¡en tu opinión!)

665

Although the activity above consists of a series of opinion questions, the audio script below demonstrates that there is always a more plausible choice.

Although there are no incorrect choices in the above activity, we generated a feedback response for each incoherent choice to simulate an instructor's likely reaction. For example, if learners chose "absurdo" for the first item, the feedback response said "Absurd? But herbal tea doesn't have caffeine!" We also used the general feedback function to provide a written repetition of the audio script so that learners could review their answers. Figure 4 demonstrates the use of the feedback with this multiple-choice question.

Figure 4

Multiple-choice Feedback

666

While the above activity in its original form does not utilize WebCT's capacity to score right or wrong answers, the modified version does discriminate between coherent and incoherent responses. It was WebCT's interactive feedback capabilities that were underutilized in the original activity. The WebCT product treated these questions as affective ones with no incorrect answer, as does the print version of the manual. The question settings allocated full credit to coherent and incoherent responses alike. We treated the questions as referential. Because the instructions call for an opinion, we adjusted the settings to accept the higher score out of two attempts so that there would be no consequences for an incoherent response on the first attempt.

To overlook the pedagogical value of affective activities merely because they do not utilize WebCT's capacity to score right or wrong answers would be to allow technology, not pedagogy, to drive our decisions. As demonstrated above, affective activities whose answers depend on logical judgments make good use of WebCT's functionality if feedback to incoherent responses is developed.

Technical Challenge #7: Affective Check-all-that-apply Questions

Although affective questions for which any one answer is correct need not necessarily be regarded as a problem, those exercises for which any number or none of the choices is correct do require modification in WebCT. Whenever there is an affective check-all-that-apply question, it should be adapted to WebCT as a forced-choice question. Online survey research suggests that forced-choice questions yield more authentic responses than check-all-that-apply questions. Order of presentation effects have been found that favor the items presented first (Dillman, 2000; Israel & Taylor, 2003; Sudman & Bradburn, 1982).

To adapt a check-all-that-apply question to forced choice, there must be an applies/does not apply choice for each item or some other exhaustive range of mutually exclusive choices. The following activity demonstrates the problem.

Lección 4

Mis relaciones con la familia

Actividad B Me …

Paso 1 Select a relative of yours (padre, madre, hijo) or a set of relatives (padres, madres, hijos, abuelos). Indicate which of the following apply. Remember that me is an object pronoun, not a subject!

667

The above activity was problematic in the original WebCT product. The first six items were rendered as a check-all-that-apply question with multiple responses enabled in the settings. In WebCT, each alternative answer must be allocated a percentage of the total value of the question. The original version of the activity allowed learners to earn 0-600% depending on the number of responses selected. We modified the activity by generating six separate questions, each of which required a Sí or No response. We allocated full credit to either response, but learners could select only one response per question.

The following activity represents the problem of a check-all-that-apply question with choices that are not mutually exclusive. The activity asks students to identify whether each of eight statements applies to university students or retired people. In a student's judgment, neither or both may also be appropriate answers.

Lección 2

Actividades para el fin de semana

Actividad H Los estudiantes universitarios frente a los jubilados

What do you think is true for the two groups of people below? Mark each statement accordingly.

In paper form, the student can check either Los estudiantes universitarios 'university students,' Los jubilados 'retirees,' check both, or leave the item blank to indicate that neither is a logical choice. The original WebCT product allowed learners to earn 0-200% depending on the number of answers selected. We modified

668

the question to include the words "Choose the best response for each statement" to the direction lines and changed the "multiple answers" setting to allow "one answer." Alternatively, we could have expanded the choices to include "neither" and "both," but this hardly seemed necessary given that there was always a case to be made for a more plausible choice.

The above activity was adapted simply by directing students to choose the best response. In the following activity, it was necessary to expand the choices because they were not exhaustive.

Lección 6

Épocas anteriores

ACTIVIDAD B Contrastes

How has your world changed? Which of the following were true for you as a

child but aren't true now? Which were true both then and now?

*comerse = to eat up (not a true reflexive)

669

In this activity, the answer depends entirely on learners' experience. There is no judgment to be made about the most plausible response. It is also possible that learners would respond falso de niño y falso hoy 'false as a child and false today' or falso de niño y cierto hoy 'false as a child and true today.' By adding these two options to the modified exercise, we generated a set of mutually exclusive options that exhausted all possibilities so that one and only one answer is possible for each item.

Technical Challenge #8: Referential Check-all-that-apply Questions

Assigning percentages to responses is fairly straightforward when there is one correct response. When there are multiple correct responses to a question, each response must be allocated a percentage of the total. The following exercise is such an example.

Lección 6

Épocas anteriores

Actividad A Alternativas

Paso 2 Match the verb phrases with a logical conclusion (a, b, c, d).

CALL (Call Assisted Language Learning)

Pages

Jumat, 24 Desember 2010

CALL related to literature 2

5. Recent Developments in Technology and Language Learning: A Literature Review and Meta-analysis

6. An Empirical Investigation of Whether Authentic Web Sites Facilitate Intermediate-level French Language Students� Ability To Learn Culture

7. Retrofitted Materials for WebCT: Guidelines for Authors, Web Designers, and Users

Tidak ada komentar:

Posting Komentar

Sample Page