Institutional Context and the Development of Critical Thinking: A Research Note

Linda Serra Hagedorn, Ernest T. Pascarella, Marcia Edison, John Braxton, Amaury Nora, and Patrick T. Terenzini

Tables

It has long been a central aim of American postsecondary education to foster students' critical thinking skills. There are various definitions of critical thinking, but there seems to be a consensus that a constituent set of cognitive skills involves some or all of the following: making correct inferences from data, identifying central issues or assumptions in an argument, deducing conclusions from information or data provided, interpreting whether [End Page 265] conclusions are warranted on the basis of data given, and evaluating the validity of an argument (Brabeck & Wood, 1990; Furedy & Furedy, 1985; McMillan, 1987; Pascarella & Terenzini, 1991). One needs only to peruse recent college catalogs or bulletins to see "critical thinking" or a closely related term employed as one of the desired outcomes of an undergraduate education at a wide range of institutions. Moreover, as McMillan (1987) points out, two influential national reports published in the mid-1980s, Involvement in Learning (National Institute, 1984) and Integrity in the College Curriculum (Association of American Colleges, 1985) have underscored the enhancement of critical thinking as one of the essential impacts of an undergraduate education. More recently the salience of critical thinking as an outcome of undergraduate education was also made explicit by the National Education Goals Panel (1991).

There is a substantial body of evidence to suggest that exposure to postsecondary education in general has a positive impact on the development of critical thinking (e.g., Astin, 1993; McMillan, 1987; Pascarella, 1989; Pascarella, Bohr, Nora & Terenzini, 1996; Steele, 1986; Whitla, 1978). Less certain, however, is the extent to which different institutional contexts influence students' critical thinking skills. The sparse evidence available on this issue is inconsistent. Employing a cross-sectional design and student self-reports, respectively, Winter, McClelland, and Stewart (1981) and Pace (1974) presented evidence to suggest that students made greater gains in critical thinking at selective liberal arts colleges than at less selective institutions. (Institutional "selectivity" in both studies was defined in terms of average institutional SAT or ACT score.) However, when Pascarella (1989) effected statistical controls for precollege critical thinking and other confounding influences (controls not possible in the evidence reported by Winter et al. or Pace), he found that the partial correlations between institutional selectivity (based on average ACT score) and end-of-first-year critical thinking were small and not statistically significant. In other words, when important individual student precollege characteristics were taken into account, the average academic ability of an institution's student body had only a trivial influence on the development of critical thinking.

In addition to being sparse, the existing evidence on the effects of institutional context on critical thinking has methodological problems. The cross sectional design of Winter, McClelland & Stewart (1981) and the self-reported nature of Pace's (1974) findings make it difficult, if not impossible, to separate the impact of institutional context from that of potential confounding influences. For example, students attending selective liberal arts colleges may report greater institutional effects on their growth in critical thinking in part because they were more open to the influence of college initially than their peers at less selective institutions. While Pascarella's (1989) investigation is longitudinal and controls for a wide range of student back [End Page 266] ground traits, it has an extremely small sample--47 students, 30 of whom attended 18 different institutions. Thus, his study may simply have lacked the statistical power to detect significant institutional context effects on critical thinking.

Despite the equivocal nature of the existing evidence, knowledge about institutional effects on critical thinking may be of more than just theoretical interest. With dramatic increases in the costs of higher education, students and parents might ask what additional benefits one gets from attending an expensive, "high quality" institution. "Institutional quality" is most typically, though not totally, defined in terms of the academic selectivity (i.e., the average SAT or ACT score) of a school's entering class (Astin, 1993; Knox, Lindsey, & Kolb, 1993; Pascarella & Terenzini, 1991). A body of evidence suggests that a degree from an academically selective institution can confer at least modest career and economic advantages (James, Alsalam, Conaty, & To, 1989; Kingston & Lewis, 1990; Pascarella & Terenzini, 1991; Rumberger & Thomas, 1991; Useem & Karabel, 1986). However, it is not equally clear that there is an analogous advantage in such general cognitive skills as critical thinking that comes from attending an institution where one's peers have a high level of intellectual development. Recent evidence underscores the salience of peers as a source of influence on student growth and change during college (Astin, 1993, Baxter Magolda, 1992; Kuh, 1993; Pascarella, Edison, Nora, Hagedorn, & Terenzini, 1996). Consequently, it seems reasonable to hypothesize that the peer context of an institution would have a nontrivial influence on dimensions of student cognitive growth.

The present study sought to add to our knowledge of the effects of institutional context on critical thinking through analyses of the first- and third-year follow-ups of the National Study of Student Learning. Specifically, the study had two purposes. First it sought to estimate the unique or net effect of institutional context on an individual student's critical thinking after the first and third years of college. Institutional context was defined as the average critical-thinking level of an institution's incoming first-year class. In short, does it matter if a student attends an institution where peers tend to have a high or low level of critical thinking skills? Second, the study sought to determine if the effect of institutional context on critical thinking differed in magnitude for students with different characteristics (e.g., differences in precollege critical thinking level, academic motivation, ethnicity, gender, family social origins).

Method

Institutional Sample

The sample consisted of incoming first-year students at 18 four-year and 5 two-year colleges and universities located in 16 states throughout the country. [End Page 267] Institutions were chosen from the National Center on Education Statistics Integrated Postsecondary Education Data System (IPEDS) data to represent differences in colleges and universities nationwide on a variety of characteristics including institutional type and control (e.g., private and public research universities, private liberal arts colleges, public and private comprehensive universities, two-year colleges, historically Black colleges), size, location, commuter versus residential character, and the ethnic distribution of the undergraduate student body. Our sampling technique produced a sample of institutions with a wide range of selectivity. For example, we included some of the most selective institutions in the country as well as some that had an essentially open-admissions policy. Thus, the results of our sampling technique was a student population from 23 schools that approximated the national population of undergraduates by ethnicity and gender.

Student Sample and Instruments

The individuals in the sample were students participating in the National Study of Student Learning (NSSL), a large longitudinal investigation of the factors that influence learning and cognitive development in college. The initial sample was selected randomly from the incoming first-year class at each participating institution. We told the students in the sample that they would be participating in a national longitudinal study of student learning and that they would receive a cash stipend for their participation in each data collection. They were also informed that any information they might provide would be kept confidential and never become part of their institutional records.

We conducted an initial data collection in the fall of 1992, with 3,840 students from the 23 institutions participating. This first data collection lasted approximately three hours and students were paid a stipend of $25 for their participation. Again, we reminded students that all responses would be kept in the strictest confidence and that they should make an honest effort on tests and give candid responses to all questionnaire items. The data collected included a precollege survey that gathered information on student demographic characteristics and background, as well as aspirations and expectations of college, while a series of items assessed their orientation toward learning. Participants also completed Form 88A of the Collegiate Assessment of Academic Proficiency (CAAP), developed by the American College Testing Program (ACT) specifically to assess selected general skills typically acquired by students during the first two years of college (ACT, 1989). The total CAAP consists of five 40-minute, multiple-choice test modules, one of which, critical thinking, is the focus of this study. (Participants also completed the CAAP reading comprehension and mathematics modules during the initial data collection.) [End Page 268]

The CAAP critical thinking test is a 32-item instrument that measures the ability to clarify, analyze, evaluate, and extend arguments. The test consists of four passages that are designed to be representative of the kinds of issues commonly encountered in a postsecondary curriculum. A passage typically presents a series of subarguments that support a more general conclusion. Each passage presents one or more arguments and uses a variety of formats, including case studies, debates, dialogues, overlapping positions, statistical arguments, experimental results, or editorials. Each passage is accompanied by a set of multiple choice items. The KR-20 reliability coefficients for the critical thinking test ranged from .81 to .82 (ACT, 1989). In pilot testing various instruments for use in the National Study of Student Learning on a sample of 30 college students, the critical thinking test of the CAAP was found to correlate .75 with the total score on the Watson-Glaser Critical Thinking Appraisal. The Watson-Glaser Critical Thinking Appraisal is, by far, the most commonly employed objective measure of critical thinking (McMillan, 1987).

The first follow-up testing of the sample took place in the spring of 1993. This data collection required about three and a half hours and included an extensive set of questionnaire measures of the students' freshman-year experience, and critical thinking, reading comprehension, and mathematics modules of Form 88B of the Collegiate Assessment of Academic Proficiency. Students received a second stipend of $35. Of the original sample of 3,840 students who participated in the fall 1992 testing, 2,685 participated in the spring 1993 data collection, for a follow-up response rate of 69.9%.

Given the high response rate, it was not particularly surprising that the sample was reasonably representative of the population from which it was drawn. However, to adjust for potential response bias by gender, ethnicity, and institution, we developed a sample weighting algorithm. Specifically, within each individual institution, we weighted participants in the follow-up data collection up to the institution's population by gender (male or female) and ethnicity (white, African American, Hispanic, other). Thus, for example, if institution A had 100 African American men in its first-year class and 25 African American men in the sample, each African American male in the sample was given a sample weight of 4.00. An analogous weight was computed for participants falling within each gender times ethnicity cell within each institution. The effect of applying sample weights in this manner was to adjust, not only for response bias by gender and ethnicity, but also for response bias (i.e., differential response rate) by institution.

A second follow-up testing of the NSSL sample took place in the spring of 1994 in which students completed the writing skills and science reasoning modules of the CAAP. Of the 2,685 students who participated in the first follow-up (spring 1993), 1,761 participated in the second follow-up [End Page 269] (spring 1994), for a response rate of 65.6%. We then developed a second weighting algorithm, analogous to that applied in the first year, that adjusted for sample nonresponse bias by gender, ethnicity, and institution.

The third follow-up testing of the sample took place in the spring of 1995. Since there were so few two-year college students still identifiable in the sample after three academic years, the third follow-up testing was conducted exclusively at the 18 four-year schools in the NSSL sample. This data collection required about two and a half hours and included the critical thinking and reading comprehension modules of Form B of the Collegiate Assessment of Academic Proficiency and an extensive set of questionnaire measures of students' third-year experience of college. Students received a final stipend of $45. Of the sample of 1,613 four-year college students who had participated in the spring 1994 data collection (the second follow-up), 1,054 participated in the spring 1995 data collection, for a third follow-up response rate of 65.3%. We developed a third weighting algorithm, analogous to those applied in the first and second years of the NSSL, that once again adjusted for sample nonresponse bias by gender, ethnicity, and institution.

Variables and Analytical Model

The dependent variables in the study were end-of-first-year (spring 1993) and end-of-third-year (spring 1995) scores on the CAAP critical thinking test. The independent variable of interest was institutional context. We operationally estimated this variable by the average precollege (fall 1992) CAAP critical thinking score for the sample of incoming students at each of the 23 institutions in the study--that is, i.e., we assigned each student in the sample the mean of his or her institution on the fall 1992 CAAP critical thinking test. Operationally defined in this manner, institutional context is quite consistent with a line of social-psychological inquiry known as contextual analysis (Pintrich, Marx, & Boyle, 1993). Contextual analysis attempts to estimate the net influence of an aggregate-level variable on an individual-level variable (Erbring & Young, 1980; Firebaugh, 1980).

Not surprisingly, the estimated average critical-thinking level of each institution was highly correlated with student-body selectivity. Since some of the institutions in the NSSL sample were essentially open admissions, average ACT or SAT scores of incoming students were unavailable. However, for those schools requiring either of the standardized tests, the correlation between estimated average critical-thinking level and the average ACT score (or SAT converted to the ACT) was .903.

We expected that the simple correlations between institutional context and end-of-first- and -third-year critical thinking would be spuriously inflated by the presence of other causal influences. Consequently, to obtain a more accurate estimate of the net impact of institutional context on critical [End Page 270] thinking, we included as many of these other causal influences as possible in the analytical model, drawing them from the existing literature about factors that independently influence learning and cognitive development during college (e.g., Astin, 1968, 1977, 1993; Kuh, 1993; Pascarella, Bohr, Nora, & Terenzini, 1996; Pascarella & Terenzini, 1991; Pascarella et al., 1993; Terenzini, Springer, Pascarella, & Nora, 1995). The other causal influences, or control variables, in the model were: precollege critical thinking, precollege academic motivation, gender, ethnicity, age, family social origins, work responsibilities, total credit hours taken, place of residence, hours per week spent studying, and coursework taken in five general areas: social sciences, mathematics, technical/preprofessional, arts and humanities, and natural sciences and engineering. (See the appendix for operational definitions of all variables.)

We conducted the analyses in three stages. In the first stage, we determined the net total impact of institutional context on critical thinking (i.e., spring 1993 and spring 1995 CAAP critical thinking scores) by using a reduced-form equation approach (Alwin & Hauser, 1975). Thus, using ordinary least squares, we regressed end-of-first- and -third-year critical thinking scores on institutional context and judged only the seven control variables to be causally antecedent to, or coincident with, institutional context (i.e., precollege critical thinking, precollege academic motivation, age, ethnicity, gender, family social origins, and work responsibilities).

In the second stage of the analyses, we estimated the net direct effect of institutional context on critical thinking. To accomplish this, we regressed end-of-first- and end-of-third-year critical thinking on all 15 control variables plus institutional context.

In the third stage of the analyses, we tested for the presence of conditional effects (Pedhazur, 1982). We computed a series of cross-product terms between the institutional context variable and each of the 15 control variables, then added them to the direct effects regression model employed in the second stage of the analyses. A statistically significant increase in explained variance in critical thinking attributable to the set of cross-product terms, above and beyond the direct effects equation, indicated that the net effects on critical thinking of institutional context varied in magnitude for students at different levels on the other variables in the prediction model.

We analyzed four different samples predicting end-of-first-year critical thinking: (a) a sample which aggregated students attending two- and four-year colleges (usable data available on 2,566 students); (b) a sample composed only of two-year college students (usable data available on 280 students); (c) a sample composed of all four-year college students who participated in the first follow-up data collection in the spring of 1993 (usable data available on 2,286 students); and (d) a sample composed of all four-year college students who participated in both the first follow-up data collection [End Page 271] in spring 1993 and the third follow-up in spring 1995 (usable data available on 1,036 students). In predicting end-of-third-year critical thinking, the sample consisted of all four-year college students participating in both the first and third NSSL follow-ups (usable data available on 1,036 students).

We conducted parallel analyses on the weighted and unweighted samples with essentially the same results. The remainder of this paper, however, reports the results from the weighted sample, adjusted to actual sample size to obtain correct standard errors. Because of the large sample sizes for the combined and four-year college samples (2,566, 2,286, and 1,036) we used a critical alpha level of .01 in all combined sample and four-year college analyses. We used a critical alpha level of .05 for all two-year college analyses because of the smaller sample size (280).

Results

End-of-First-Year Critical Thinking

Table 1 presents the means and standard deviations of the variables used in the analyses for each of the study models. The first four columns of numbers in Table 2 present the estimated total and direct effects of institutional context on end-of-first-year critical thinking. As shown in Part A of Table 2, in the presence of controls for causally antecedent or coincident influences, institutional context had a significant, net, positive total effect on end-of-first-year critical thinking irrespective of which sample was considered. The same was true for the direct effects of institutional context, as shown in Part B of Table 2. In the presence of controls for all other variables in the model, institutional context had a significant, net positive direct effect on end-of-first-year critical thinking in each of the four samples analyzed. Such findings support the concept that attending an institution where one's peers tend to have a highly developed level of critical thinking has a positive influence on the development of one's own critical thinking skills during the first year of college.

End-of-Third-Year Critical Thinking

According to our findings, the significant effect of institutional context on first-year critical thinking was not found in the prediction of critical thinking level after the third year of college. As shown in the last column of Table 2, when the influence of the control variables was taken into account, both the total and direct effects of institutional context on end-of-third-year critical thinking become small and statistically nonsignificant. However, the evidence of constriction of variance, due to sample differences between the first and third years, should be noted. The sample of students [End Page 272] [Begin Page 276] from four-year institutions who participated in the third year of the study consisted of those who both persisted to the junior year and also chose to continue to participate in the study.

Magnitude of Effects

In terms of variance explained, the magnitude of the net influence of institutional context on critical thinking appears quite modest. As shown in Table 2 the variance in first-year critical thinking attributable to institutional context ranged from 0.6% to 2.1% for the total effect, and from 0.6% to 2.7% for the direct effect. The variance in third-year critical thinking attributable to institutional context was only 0.1% (nonsignificant) for both total and direct effects.

Another perspective comes from considering the metric regression coefficient (b) as a measure of the net or unique effect of institutional context on individual student critical thinking. The metric or "b" regression coefficient in Table 2 can be considered as the amount of increase in individual level critical thinking uniquely attributable to a one point increase in the average critical thinking of the students at each institution. Thus, considering the direct effects, a one point increase in average student body critical thinking among the combined sample of two- and four-year colleges was associated with an increase of .333 of a point in individual level first-year critical thinking among two- and four-year college students. Average incoming student critical thinking level had a range of 14.91 points between the highest and lowest institutions in the combined sample of 23 schools. Multiplying .333 by 14.91 yielded 4.97. Thus, net of other influences, students attending the institution with the highest average student body critical thinking level answered about 5 more questions correctly out of 32 than students attending the institution with the lowest average student body critical thinking level. The institution with the highest average student body critical thinking was one of the most selective and prestigious private liberal arts colleges in the country. It is typically ranked in the top 5 or 10 liberal arts colleges nationwide by U.S. News and World Report. The institution with the second highest student body critical thinking was a highly selective private research university that U.S. News and World Report typically ranks in the top 10 or 15 universities nationally. The institution with the lowest average student body critical thinking level was an open-admissions, four-year institution, and the next lowest was an open-admissions community college.

Calculating the corresponding effects for the other analyses of first-year critical thinking yielded similar results. The net advantage in individual level critical thinking attributable to attending the two-year college with the highest, compared to the lowest, average student body critical thinking level was [End Page 276] an additional 5.66 questions out of 32 answered correctly. In the four-year college samples, the corresponding institutional advantage was an additional 3.73 of 32 questions and 4.12 of 32 questions respectively for four-year sample I and four-year sample II. Not surprisingly, since institutional context had only a small and nonsignificant effect on individual critical thinking skills in the third year of college, the advantage in terms of additional critical thinking questions answered correctly was also quite small. Net of other influences, students attending the four-year school with the highest average student body critical thinking level answered about one (1.24) more question out of 32 correctly than students attending the four-year school with the lowest average student body critical thinking level.

Conditional Effects

The third stage of the data analyses sought to determine if the institutional context effects on first- and third-year critical thinking were general or conditional. Adding the sets of cross-product terms to the general or main-effects model was associated with small (less than .3%) and nonsignificant R² increases in individual level critical thinking in four of the five samples analyzed. Only for the two-year college sample was the increase in explained variance associated with the cross-product terms even marginally significant (R² increase = 3.04%, p < .054). We judged this result sufficiently close to statistical significance with a small sample (N = 280) to warrant examination of individual conditional effects. Only one of the cross-product terms, institutional context times individual precollege critical thinking, was associated with a statistically significant increase in explained variance (R² increase = .83%, t = 2.65, p < .01). To determine the nature of the conditional effect, we divided individual precollege critical thinking at the mean and recomputed the direct effects of institutional context on first-year critical thinking separately for two-year college students in the lower and upper half of the precollege critical thinking distribution. The resultant comparison of regression weights for the two samples suggested that institutional context had a compensatory effect for students at two-year institutions. That is, the net positive effect of institutional context on individual critical thinking was moderately stronger for students in the lower half of the precollege distribution of critical thinking (b = .944, beta = .565) than for students in the upper half (b = .704, beta = .364). Thus, the benefit of attending a two-year college where one's peers have a relatively advanced level of critical thinking was somewhat more pronounced for students who started postsecondary education with relatively low levels of critical thinking skills. For their counterparts who entered college with a relatively high level of critical thinking skills, institutional context appears less important. [End Page 277]

Additional Analyses

Because of the wide range of institutions in the sample, we were concerned that a ceiling effect on the CAAP critical thinking test might artificially suppress the effects of institutional context on individual level critical thinking. To test for the presence of a ceiling effect, we repeated the entire set of analyses after dropping from the sample those students who initially scored above a 29 on the fall 1992 precollege administration of the CAAP. Our reasoning here was that, to achieve a maximum score on the follow-up testings of the CAAP (32), a student would now have to improve at least 3 points over time. Such an improvement was about 1.25 times the average student improvement from entrance in college to the end of the third year of college at any four-year institution in the sample. The results of these analyses were almost exactly the same as those yielded by the full sample. Thus, it would not appear that the presence of a ceiling effect in the data had a marked impact on our findings. Indeed, the selective liberal arts college with the highest average level of incoming student body critical thinking in fall 1992 also had the largest, positive, net impact of any institution at the end of either the first- or third-year on individual-level critical thinking. This finding tended to hold across all analyses, irrespective of whether the analyses were based on the full sample or the sample that dropped students initial scoring above 29.

Summary and Discussion

This study sought to determine if the average level of student body critical thinking at an institution (i.e., institutional context) influenced the development of an individual student's critical thinking skills. Based on analyses of data from 23 highly diverse two- and four-year colleges, the answer would appear to be a qualified yes. In the presence of controls for a battery of 15 potentially confounding influences, including individual precollege critical thinking level, the average level of student body critical thinking at an institution had statistically significant, positive effects on the critical thinking skills of individual students at the end of the first year of college. The positive net impact of institutional context on first-year critical thinking held across analyses of several different samples: a combined sample of 23 two- and four-year colleges; a subsample of 5 two-year colleges; and a subsample of 18 four-year colleges.

This conclusion, however, needs to be qualified in at least two important ways. First, considering the marked diversity of institutions in the sample, the magnitude of the net effects of institutional context on critical thinking are quite modest in magnitude. As with previous evidence on the impact of where one attends college (Pascarella & Terenzini, 1991), our findings do not support the public or academic folklore that marked differences in putative [End Page 278] institutional selectivity or "quality" are matched by similar differences in institutional impacts. Rather, over time, it may be that the variation of student achievement within institutions is greater than that between institutions. About 97-99% of the differences in first-year critical thinking were due to factors other than the average level of student body critical thinking skills at the school attended.¹ (Of course, part of that 97-99% is due to measurement error, but our regression models typically explained in excess of 60% of the variance in individual level critical thinking.)

Another perspective on the modest magnitude of the effect of institutional context can be gained when one considers the numbers of questions answered correctly by students at the top institutions contrasted with students at the bottom institutions in the sample. Net of other factors, the advantage uniquely attributable to attending the institution with the highest (versus the lowest) student body critical thinking level ranged only between 3.7 more and 5.7 more questions answered correctly on a 32 question test. Upon entrance to college, students attending the top institution in the sample answered, on average, 15 more questions correctly out of 32 than students attending the bottom institution.

A second important qualification is that we found the small, positive effect of institutional context on individual critical thinking only in the first year. By the end of the third year of college, attending an institution with a high (rather than a low) level of average student body critical thinking skills had only a trivial and nonsignificant influence on individual student critical thinking.² In fact, at the end of the third year of college, institutional context accounted for only 0.1% of the differences in individual student [End Page 279] critical thinking; and the advantage uniquely attributable to attending the four-year school with the highest (versus the lowest) student body critical thinking level was about one more question answered correctly on a 32-question test.

For the four-year schools in the study, the small positive effects of institutional context on critical thinking were general rather than conditional. That is, they were essentially the same in magnitude irrespective of where students stood on the 15 other variables in the prediction model (e.g., precollege critical thinking, precollege academic motivation, age, gender, ethnicity, family social origins, work responsibilities, etc.). The same was true for the two-year colleges with one major exception. The greatest benefits that derived from attending a community college where the student body had a relatively high level of critical thinking skills accrued to those individual students who began postsecondary education with a relatively low level of critical thinking. For those community college students who began postsecondary education with a relatively high level of critical thinking skills, the influence of institutional context on critical thinking become less important. In short, on this one important dimension, the institutional context of a community college influenced cognitive growth in a compensatory manner.

Institutional quality in undergraduate education in the United States is often defined in terms of the academic selectivity of a school's student body; and selectivity, in turn, is usually defined by average incoming student body scores on standardized tests (e.g., Astin, 1993; Pascarella & Terenzini, 1991). One need only examine the annual ranking of undergraduate institutions by U.S. News and World Report to see a strong association between where an institution ranks and the tested academic ability of its entering undergraduates. On the surface it would appear that students receive a better education at more selective institutions. Furthermore, given what we know about the socializing potential of one's peers, there is a great deal of logical appeal to the concept that the intellectual adroitness of one's classmates will have an important impact on the development of individual cognitive capabilities. It also seems logical that the cognitive skills of a college's undergraduate students should significantly influence the cognitive level and rigor at which classroom instruction can occur, although the evidence for this hypothesis is somewhat mixed (Braxton, 1993; Braxton & Nordvall, 1985).

Our findings support the concept that a measure of institutional context highly related to institutional selectivity (the average critical-thinking level of entering students) modestly influences at least one important dimension of student cognitive growth. However, such influence apparently diminishes as students progress through college. By the third year of postsecondary education, the influence of institutional context becomes small and nonsignificant. [End Page 280]

Several explanations may account for the waning of the positive cognitive influence of institutional context after the initial year of college. First, students may experience "contextual shock" most acutely during the first year of college. Although individuals entering highly selective institutions tend to be intellectually quite capable, they probably have not experienced the challenging peer environment created by such a heavy concentration of academically talented students in their secondary schools. Because of the intellectual agility of their peers, students may soon discover that they must more fully engage themselves in higher-order cognitive processes and activities, including critical thinking, if they are to meet the academic, and perhaps even the social, norms of the institution. As students adjust to their peer environment and experience some academic success during the first year of college, they are likely to do so, at least in part, by developing a new repertoire of intellectual skills and attitudes (Heath, 1968). Once these skills and attitudes acquired during the first year of college are internalized, they may need to be applied and refined but not fundamentally relearned. Similarly, differences in college life between the first and third years must be considered. By the third year in college many students have chosen an affinity group in which the majority of their interactions occur. Thus, students may center their interactions with individuals very similar to themselves rather than with the general student body. Self-imposed isolation (especially evident in some fraternities, sororities, or athletic groups) may effectively diminish the effect of institutional context. Thus, the cognitive influence of institutional context discernible after one year of college may essentially disappear by the end of the third year.

A second, and related, possibility is that the socializing impact of one's peers may have its most pronounced cognitive impact during the first year of college. In institutions with highly able students, peer acceptance may require that one be able to participate in discussions where the ability to think critically is taken for granted. Such discussions with peers may occur both in class and out. However, this peer influence may be obviated by the third year of college because, by then, such discussions become largely routine to the individual student.

A fourth possibility is that, although "contextual shock" and peer socialization may largely account for the effect of institutional context on critical thinking during the first year of college, they are insufficient in and of themselves to sustain the development of critical thinking. As Rodgers (1980) and Stage (1991) have suggested, further growth in critical thinking may require curricular and instructional processes that challenge and support students throughout their college careers. Academic folklore leads us to believe that such curricular and instructional processes are more prevalent in selective institutions. However, instead of concentrating on the selectivity of the institution, future research should examine the effect of discipline [End Page 281] differences, course-taking patterns, class size, and the amount and quality of student-faculty interactions. Our pattern of findings suggests the possibility that systematic differences in curricular and instructional processes may not exist across institutions of varying academic selectivity.

Limitations

This investigation has several limitations that should be kept in mind when interpreting the findings. First, although the overall sample is multi-institutional and consists of a broad range of two- and four-year institutions from 16 different states across the country, the fact that the analyses were limited to 23 institutions means that we cannot necessarily generalize the results to all two- and four-year institutions.

Similarly, although we attempted, in the initial sampling design and subsequent sample weighting, to make the sample as representative as possible at each institution, the time commitment and work required of each student participant undoubtedly led to some self-selection. We cannot be sure that those who were willing to participate in the study responded in the same way as those students would have who were invited to participate but declined. Weighed against this possibility, however, is the fact that we found no significant conditional effects involving such factors as gender, ethnicity, age, socioeconomic status, academic motivation, or, for the four-year sample, precollege critical thinking. Thus, even if the sample had some bias on these factors, it did not appear to have an appreciable influence on the study results.

Third, while we looked at one important measure of cognitive growth in college (the ability to think critically), this area is certainly not the only way in which students develop intellectually during college. Alternative conceptualizations or approaches to the assessment of cognitive development might have produced findings different from those yielded by this investigation.

Appendix

Definitions of Variables

Precollege Critical Thinking: An individual's precollege (fall, 1992) score on the CAAP critical thinking test.

Precollege Academic Motivation: An eight-item, Likert-type scale (5 = "strongly agree" to 1 = "strongly disagree") with an internal consistency reliability of .65. The scale items were based on existing research on academic motivation. Examples of constituent items are: "I am willing to work hard in a course to learn the material, even if it won't lead to a higher grade," "When I do well on a test it is usually because I was well prepared, not because the test was easy," "In high school I frequently did more reading in a class than was required simply because it interested me," and "In high school I frequently talked to my teachers outside of class about ideas presented during class."

Female: 1 = female, 0 = male.

Non-White: 1 = non-white, 0 = white.

Age: A continuous variable calculated by subtracting year of birth from 1992.

Family Social Origins: A combination of standardized measures of mother's and father's level of formal education and combined family income.

Total Credit Hours Completed: Cumulative number of credit hours completed.

Hours Worked Per Week: Combination of average number of hours of on- and off-campus work per week during the school year, coded 1 = none, to 9 = more than 35.

On-Campus Residence: 1 = lived on-campus, 0 = lived off campus.

Hours Per Week Spent Studying: single-item, 6-point self-report of average hours per week spent studying where 1 = none and 6 = more than 20 hours.

Social Sciences Courses Taken: Number of college courses taken in anthropology, audiology/speech pathology, child and family studies, communications, economics, geography, history, political science, psychology, sociology, or social work.

Mathematics Courses Taken: Number of college courses taken in prealgebra, algebra, calculus, statistics, computer science, geometry, matrix algebra, accounting, or business math.

Technical/Preprofessional Courses Taken: Number of college courses taken in drawing, drafting, architectural design, criminology, education, agriculture, business, physical therapy, pharmacy, physical education, nursing, or computer programming.

Arts and Humanities Courses Taken: Number of college courses taken in art history, art appreciation, studio art, dance, theater, music appreciation, music performance, composition or writing, English literature, foreign language, humanities, philosophy, linguistics, classics, or religious studies.

Natural Sciences and Engineering Courses Taken: Number of college courses taken in astronomy, botany, biology, chemistry, physics, geology, zoology, microbiology, and engineering.

Institutional Context: Estimated by the average precollege (fall 1992) CAAP critical thinking score for the sample of incoming students at each of the 23 institutions. Each individual student in the sample was then given the mean of his or her institution on the 1992 CAAP critical thinking test.

Dependent Variables: End-of-first-year or end-of-third-year individual scores on the CAAP critical thinking test.

Linda Serra Hagedorn is Assistant Professor and Senior Research Associate at the Center for Higher Education Policy Analysis, School of Education, University of Southern California. Ernest T. Pascarella is Mary Louise Petersen Professor of Higher Education at the University of Iowa. Marcia Edison is a Postdoctoral Fellow in Medical Education, University of Illinois at Chicago. John Braxton is Professor of Higher Education at Vanderbilt University, Nashville, Tennessee. Amaury Nora is Professor of Higher Education at the University of Houston. Patrick T. Terenzini is Professor of Higher Education and Senior Scientist at the Center for the Study of Higher Education, The Pennsylvania State University, State College. This study was conducted as part of the National Study of Student Learning (NSSL) at the University of Illinois at Chicago. NSSL is supported by Grant No. R117G1037 from the U.S. Department of Education to the National Center on Postsecondary Teaching, Learning, and Assessment. Address inquiries to Linda Serra Hagedorn, Assistant Professor, Center for Higher Education Policy Analysis, School of Education, University of Southern California, Waite Phillips Hall 701E, Los Angeles, CA 90089-0031; telephone (213) 740-6772; fax: 213-740-3889; e-mail: LSH@usc.edu

References

ACT. American College Testing Program. (1989). Report on the technical characteristics of CAAP: Pilot Year 1: 1988-89. Iowa City, IA: Author.

Alwin, D., & Hauser, R. (1975). The decomposition of effects in path analyses. American Sociological Review, 40, 37-47.

Association of American Colleges (1985). Integrity in the college curriculum: A report to the academic community. Washington, DC: Association of American Colleges.

Astin, A. (1968). Undergraduate achievement and institutional "excellence." Science, 161, 661-668.

Astin, A. (1977). Four critical years. San Francisco: Jossey-Bass

Astin, A. (1993). What matters in college. San Francisco: Jossey-Bass.

Baxter Magolda, M. (1992). Knowing and reasoning in college. San Francisco: Jossey-Bass.

Brabeck, M., & Wood, P. (1990). Cross-sectional and longitudinal evidence for differences between well-structured and ill-structured problem solving abilities. In M. Commons, C. Armon, L. Kohlberg, F. Richards, T. Grotzer, and J. Sinnott (Eds.), Adult development 2: Models and methods in the study of adolescent and adult thought. New York: Praeger.

Braxton, J. (1993). Selectivity and rigor in research universities. Journal of Higher Education, 64, 657-675.

Braxton, J., & Nordvall, R. (1985). Selective liberal arts colleges: Higher quality as well as higher prestige? Journal of Higher Education, 56, 538-554.

Erbring, L., & Young, A. (1980). Individuals and social structure: Contextual effects as endogenous feedback. In E. Borgatta & D. Jackson (Eds.), Aggregate data analysis and interpretation (pp. 25-59). Beverly Hills, CA: Sage.

Firebaugh, G. (1980). Assessing group effects: A comparison of two methods. In E. Borgatta & D. Jackson (Eds.), Aggregate data analysis and interpretation (pp. 13-24). Beverly Hills, CA: Sage.

Furedy, C., & Furedy, J. (1985). Critical thinking: Toward research and dialogue. In J. Donald and A. Sullivan (Eds.), Using research to improve teaching (pp. 51-69). San Francisco: Jossey-Bass.

Heath, D. (1968). Growing up in college. San Francisco: Jossey-Bass.

James, E., Alsalam, N., Conaty, J., & To, D-L. (1989). College quality and future earnings: Where should you send your child to college? American Economic Review, 79, 247-252.

Kingston, P., & Lewis, L. (1990). The high-status track: Studies of elite schools and stratification. Albany, NY: SUNY Press.

Kuh, G. (1993). In their own works: What students learn outside the classroom. American Educational Research Journal, 30, 277-304.

Knox, W., Lindsay, P., & Kolb, M. (1993). Does college make a difference? Westport, CT: Greenwood Press.

McMillan, J. (1987). Enhancing college students' critical thinking: A review of studies. Research in Higher Education, 26(1), 3-29.

National Education Goals Panel. (1991). Executive summary: The National Education Goals Report, 1991: Building a nation of learners. Washington, DC: Author.

National Institute of Education Study Group (1984). Involvement in learning: Realizing the potential of American higher education. Washington, DC: National Institute of Education, U.S. Department of Education.

Pace, C. (1974). The demise of diversity? A comparative profile of eight types of institutions. Berkeley, CA: Carnegie Commission on Higher Education.

Pascarella, E. (1989). The development of critical thinking: Does college make a difference? Journal of College Student Development, 30(1), 19-26.

Pascarella, E., Bohr, L., Nora, A., & Terenzini, P. (1996). Is differential exposure to college linked to the development of critical thinking? Research in Higher Education, 37(2), 159-174.

Pascarella, E. T., Edison, M., Nora, A., Hagedorn, L. S., & Terenzini, P. T. (1996). Influences on students' openness to diversity and challenge in the first year of college. Journal of Higher Education, 67(2), 174-195.

Pascarella, E. T., & Terenzini, P. T. (1991). How college affects students: Findings and insights from twenty years of research. San Francisco: Jossey-Bass.

Pascarella, E., et al. (1993). Cognitive impacts of living on campus versus commuting to college. Journal of College Student Development, 34, 216-220.

Pedhazur, E. (1982). Multiple regression in behavioral research: Explanation and Prediction (2d ed.). New York: Holt, Rinehart & Winston.

Pintrich, P. R., Mars, R. W., & Boyle, R. A. (1993). Beyond cold conceptual change: The role of motivational beliefs and classroom contextual factors in the process of conceptual change. Review of Educational Research, 63(2), 167-199.

Rodgers, R. (1980). Themes underlying student development. In D. Creamer (Ed.), Student development in higher education: Theories, practices, and future directions (pp. 10-95). Cincinnati, OH: ACPA Media.

Rumberger, R., & Thomas, S. (1991). The economic returns to college major, quality, and performance: A multilevel analysis of recent graduates. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.

Stage, F. (1991). Common elements of theory: A framework for college student development. Journal of College Student Development, 32(1), 56-61.

Steele, J. (1986). Assessing reasoning and communication skills of postsecondary students. Paper presented at the annual meeting of the American Educational Research Association, 1986, San Francisco.

Terenzini, P. T., Springer, L., Pascarella, E. T., & Nora, A. (1995). Influences affecting the development of students' critical thinking skills. Research in Higher Education, 36(1), 23-39.

Useem, M., & Karabel, J. (1986). Pathways to top corporate management. American Sociological Review, 51, 184-200.

Whitla, D. (1978). Value added: Measuring the impact of undergraduate education. Cambridge, MA: Office of Instructional Research and Evaluation, Harvard University.

Winter, D., McClelland D., & Stewart, A. (1981). A new case for the liberal arts: Assessing institutional goals and student development. San Francisco: Jossey-Bass.

[an error occurred while processing this directive]