Wednesday, July 5, 2017

The 2015 NAEP

You can follow me on Twitter @dbressoud.

On May 5, 2017, the presidents and executive directors of the member societies of CBMS received a report from Samantha Burg and Stephen Provasnik at the U.S. Department of Education’s National Center for Education Statistics (NCES) on the results in mathematics from the 2015 studies by the National Assessment of Educational Progress (NAEP) and Trends in International Mathematics and Science Study (TIMSS). The full PowerPoint of their presentation, covering both the 2015 NAEP and 2015 TIMSS, can be accessed at

Both assessments are conducted for students at grades 4, 8 and 12. NAEP is a federally mandated assessment of student achievement in the U.S. and is conducted every other year. TIMSS provides an international comparison and is run every four years for ages equivalent to grades 4 and 8. The 12th grade TIMSS is restricted to advanced mathematics students (in the U.S. those who have taken a course like AP Calculus). It was administered in 2015 for the first time in the U.S. since 1995.

The scores since 1990 for the 4 th and 8 th grade NAEP and since 2005 for the 12th grade are shown in Figures 1, 2, and 3. The distinguishing features for grades 4 and 8 are the strong growth from 1990 until 2007 and relative stagnation since then, with a small but statistically significant drop (except for the 90th percentile in grade 4) between 2013 and 2015. The 12th grade scores also show a drop since 2013 that is statistically significant at and below the 50th percentile.

This drop is a cause for concern, but not yet alarm. NCES is eagerly anticipating the 2017 NAEP results to see whether the downturn was simply a blip in what is essentially a stable state or the start of something more troubling.

Figure 1: NAEP scores for grade 4.
Source: Burg & Provasnik, 2017.

Figure 2: NAEP scores for grade 8. Source: Burg & Provasnik, 2017.

Figure 3: NAEP scores for grade 12.
Source: Burg & Provasnik, 2017.

An obvious question is whether the Common Core State Standards in Mathematics (CCSS-M) have had any effect on student scores. One hypothesis is that changing the curriculum has introduced enough confusion and uncertainty among teachers that it is having a visibly negative effect. Another hypothesis, which has some supporting evidence, is that the choices of topics for assessment may no longer be completely aligned with what is being taught.

The largest drops at Grade 4 were in the subject areas of Geometry and Data Analysis (Table 1). The NAEP Validity Studies (NVS) panel (Daro, Hughes, & Stancavage, 2015) found some misalignment between the NAEP questions and the CCSS-M curriculum. They found that 32% of the Data Analysis questions were either not covered in CCSS-M or were covered after grade 4. In Geometry, 18%, of the NAEP questions were covered after grade 4 in CCSS-M. In the other direction, only 57% of CCSS-M standards for Operations and Algebraic Thinking by grade 4 were covered by NAEP questions.

Table 1: Changes in NAEP scores, 2013 to 2015, by subscale topics.
Source: Burg & Provasnik, 2017

For grade 8, the misalignment occurs in both directions within Data Analysis. In the 8 th grade NAEP, 17% of the Data Analysis questions had not yet been covered in CCSS-M, and 59% of what is specified for statistics and probability by grade 8 in CCSS-M was not assessed by NAEP. For grade 12, there was a uniform 2-point drop across all subscales.

These observations raise interesting questions about the construction of future NAEP instruments. Because of the need for comparability from one test administration to the next, the distribution of topics has not changed. While CCSS-M is not the national curriculum that was once envisioned, the fact is that almost all states have aligned their standards with its expectations. NAEP may need to change to reflect the reality of what is taught by grades 4 and 8.

The breakdowns by race/ethnicity and gender for the overall mathematics scores in grades 4 and 8 (Table 2) show comparable increases from 1990 to 2015, and comparable declines since 2013. Black students in grade 4 saw the greatest gains since 1990, but at a score of 224 they are still well below the national average.

Table 2: Changes in NAEP Math scores for grades 4 and 8 by race/ethnicity and gender.
 Source: Burg & Provasnik, 2017.

At grade 12, the strongest gains since 2005 have been for Asian and Hispanic students (Table 3, Pacific Islanders are such a small proportion of Asian/Pacific Islander that it is not clear how their scores have changed, and the doubling of the percentage identifying as Two or More Races makes it difficult to compare the 2005 and 2015 scores). An interesting insight lies in the shift in the demographics of 12 th grade students. In ten years, the percentage of White students dropped from 66% to 55%, while the percentage of Hispanic 12 th graders rose from 13% to 22%.

Table 3: Changes in NAEP Math scores for grade 12 by race/ethnicity and gender.
 Source: Burg & Provasnik, 2017.

Next month I will be looking at the changing demographics of bachelor’s degrees earned in engineering, the mathematical sciences, and the physical sciences. In mathematics, the decline in the percentage of degrees in mathematics going to White students has been in line with the decline in their overall percentage at that age group, from 72.4% in 2005 to 59.6% in 2015 (NCES, 2005–2015). Some of this has been made up by a significant increase in mathematics degrees going to Hispanic students, from 5.7% to 8.9%, but the percentage of bachelor’s degrees in mathematics earned by Black students decreased from 6.1% to 4.7% over this decade, while Asian students remained essentially stable, 10.2% to 10.6%. Most of the shift has gone to non- resident aliens who accounted for 5.0% of the mathematics degrees in 2005, but 12.9% in 2015.

Burg, S. & Provasnik, S. (2017). NAEP and TIMSS Mathematics 2015. Presentation to the Conference Board of the Mathematical Sciences, May 5, 2017. Available at

Daro, P., Hughes, G.B., & Stancavage, F. (2015). Study of the alignment of the 2015 NAEP mathematics items at grades 4 and 8 to the Common Core State Standards (CCSS) for Mathematics. NAEP Validity Studies Panel report. Washington, DC: American Institutes for Research. Available at Alignment-NAEP-Mathematics- Items-common- core-Nov- 2015.pdf

National Center for Education Statistics (NCES). (2005–2015). Digest of Education Statistics. Available at

In compliance with new standards from the U.S. Office of Management and Budget for collecting and reporting data on race/ethnicity, additional information was collected beginning in 2011 so that results could be reported separately for Asian students, Native Hawaiian/Other Pacific Islander students, and students identifying with two or more races. In earlier assessment years, results for Asian and Native Hawaiian/Other Pacific Islander students were combined into a single Asian/Pacific Islander category.

As of 2011, all of the students participating in NAEP are identified as one of the following seven racial/ethnic categories:
  • White 
  • Black (includes African American) 
  • Hispanic (includes Latino) 
  • Asian 
  • Native Hawaiian/Other Pacific Islander 
  • American Indian/Alaska Native 
  • Two or more races
When comparing the results for racial/ethnic groups from 2013 to earlier assessment years, results for Asian and Native Hawaiian/Other Pacific Islander students were combined into a single Asian/Pacific Islander category for all previous assessment years.

Thursday, June 1, 2017

Re-imagining the Calculus Curriculum, II

You can follow me on Twitter @dbressoud.

Last month, in "Re-imagining the Calculus Curriculum," I, I introduced Project DIRACC (Developing and Investigating a Rigorous Approach to Conceptual Calculus), developed by Pat Thompson, Mark Ashbrook, and Fabio Milner at Arizona State University. References to the theory underpinning this approach are given at the end of this column. This month’s column will expand on some details of this curriculum.

One of the first common student misconceptions that Project DIRACC tackles is that variables are simply stand-ins for unknown quantities. The authors begin the meat of his course in Chapter 3 with an explanation of the distinction between variable, constant, and parameter, pointing out how context-specific the designations as either variable or parameter can be. One of the distinctive features of this project is the thoughtful use of technology, in this case enabling students to play with the effect of varying a variable with a variety of choices of parameter (see

This leads to relationships between variables (how volume varies with height), and then functions as a special class of relationships between variables, one in which “any value of one variable determines exactly one value of the other.” The point is that the f in f (x) has meaning. It is the name of the relationship. This enables the authors to tackle the misconception that f (x) is simply a lengthy way of expressing the variable y.

While acknowledging that f(x) can represent a second variable, they emphasize that it is shorthand for “the value of the relationship f when applied to a value of x.” This point is driven home by an example of the usefulness of functional notation. If d(x) relates a moment in time, x measured in years, to the distance between the Earth and the Moon at that time, then d(x) – d(x–5) enables us to express the change in distance over the five years before time x, while d(x+5) – d(x) expresses the change in distance over the succeeding five years.

The authors also make the important distinction between functions defined conceptually—the distance between Earth and Moon at a given time—and those defined computationally, such as V(u) = u(13.76 – 2u)(16.42 – 2u). They then proceed to devote considerable effort to describing the structure of functions as they are built from sums, products, quotients, compositions, and inverses. This includes clarifying the distinction between the independent variable and the argument of a function. Thus for f (x/3 + 5) the independent variable is x, but the function argument is x/3 + 5, an important step toward understanding composition of functions.

While function structure should be part of precalculus, the importance of including this material has been revealed in exploring student difficulties with differentiation. Given a complicated computational rule that defines a function, students often have difficulty parsing this rule and thus determining the choice and order of the techniques of differentiation they need to use.

Rates of change are now introduced in Chapter 4. The authors distinguish between ∆x, the parameter that describes the length of a small subinterval of the domain, and the changes in x and y represented by the differentials dx and dy. These are variables that within the given subinterval are always connected by a linear relationship.

A nice illustration of how this works is given with a photograph of a truck traveling through an intersection (Figure 1).

Figure 1. A photo of truck taken with a shutter setting of 1/1000 sec.
Taken at a shutter speed of 1/1000th of a second, it appears to freeze the truck. But if you zoom in on the tail light (Figure 2, see Section 4.3 for a video of the zoom), the streaks reveal that the truck was moving.

Figure 2. A closer look at the truck's tail light shows small streaks.
The truck moved slightly while the camera's shutter was open.

One can even estimate the length of the streaks to approximate the velocity of the truck. Over 1/1000th of a second, it is doubtful that the truck’s velocity changed very much. The picture of the truck was taken at a “moment” in time, but that moment stretched over 0.001 seconds. The point is that this period of time is short enough that the truck’s velocity measured as change in distance over change in time is “essentially constant.” If y is position and x is time, then over this interval of length ∆x = 0.001 seconds, we can treat the variable dy as a constant times dx. It is this constant that is used to define the rate of change at a moment,

We say that a function has a rate of change at the moment x0 if, over a suitably small interval of its independent variable containing x0, the function’s value changes at essentially a constant rate with respect to its independent variable.

Significantly, even as the authors are defining the rate of change at a moment, they emphasize that “all motion, and hence all variation, is blurry.”

Note that there is no mention of limits, a means of defining the derivative that is often more confusing than enlightening (see the 2014 Launchings columns from July, August, and September).

After further discussion and exploration of rate of change functions, the authors now move in Chapter 5 to Accumulation Functions, building up total changes from rates of change that are essentially constant on very small intervals. These give rise to what are anachronistically referred to as left-hand Riemann sums. Students use technology to explore the increasing accuracy as ∆x gets smaller. The effect of the choice of starting value is noted, and the definite integral with a variable upper limit now appears. It is important that the first time students see a definite integral it has a variable upper limit.

In Chapter 6, the inverse problem, going from knowledge of an exact expression of the accumulation function to the discovery of the corresponding rate of change function, is now explored, leading to the Fundamental Theorem of Integral Calculus in the form: The derivative with respect to x of the definite integral from a to x of a rate of change function is equal to that rate of change function evaluated at x. Techniques and applications of differentiation follow as the semester concludes.

The great strength and promise of this approach is that the traditional content of the first semester of calculus is only slightly tweaked, especially since it is increasingly common for university Calculus I courses to avoid or significantly downplay limits. But the curriculum has been totally reshaped to address common student difficulties and misconceptions. This route into calculus has the added advantage—though perhaps a disadvantage in the eyes of some students—that those who have been through a procedurally oriented course are unlikely to recognize this as an accelerated repetition of what they have already studied. It will challenge them to rethink what they believe calculus to be.


Thompson, P.W. and Silverman, J. (2008). The concept of accumulation in calculus. In M.P. Carlson & C. Rasmussen (Eds.), Making the connection: Research and teaching in undergraduate mathematics (MAA Notes Vol. 73, pp. 43–52). Washington, DC: Mathematical Association of America.

Thompson, P.W., Byerley, C. and Hatfield, N. (2013). A conceptual approach to calculus made possible by technology. Computers in the Schools. 30:124–147.

Thompson, P.W. and Dreyfus, T. (2016). A coherent approach to the Fundamental Theorem of Calculus using differentials. In R. Göller. R. Biehler & R. Hochsmuth (Eds.), Proceedings of the Conference on Didactics of Mathematics in Higher Education as a Scientific Discipline (pp. 355–359 ) Hannover, Germany: KHDM.

Monday, May 1, 2017

Re-imagining the Calculus Curriculum, I

You can now follow me on Twitter @dbressoud.

I was recently asked about calculus instruction: Which is easier, reforming pedagogy or curriculum? The answer is easy: pedagogy. This is not to say that it is easy to change how we teach this course, but it is far easier than trying to change what we teach. The order and emphasis of topics that emerged in the 1950s has proven extremely hard to shift.

Not that we have not tried. During the Calculus Reform movement of the late 1980s and early 1990s, NSF encouraged curricular innovation. Several of these efforts adopted an emphasis on modeling dynamical systems, introducing calculus via differential equations and developing the tools of calculus in service to this vision. This provides wonderful motivation, and this approach survives in a few pockets. It is how we teach calculus at Macalester, and the U.S. Military Academy at West Point has successfully used this route into calculus for over a quarter century. But despite its appeal, this curriculum necessitates modifying the entire year of single variable calculus, raising problems for institutions that must accommodate students who are transferring in or out. It also is a difficult sell to those who worry about “coverage” since it requires devoting considerable time to topics—modeling with differential equations, functions of several variables, and partial differential equations—that receive little or no attention in the traditional course.

This month, I want to talk about a promising curricular innovation that Pat Thompson at Arizona State University has been developing in collaboration with Fabio Milner and Mark Ashbrook, Project DIRACC (Developing and Investigating a Rigorous Approach to Conceptual Calculus). It has the advantage that it fits more easily into what is expected from each semester. It has been under development since 2010 and is slated to be the curriculum used for all Calculus I sections for mathematics or science majors at ASU beginning in fall, 2018.

Thompson began with research into the misconceptions that students carry into calculus and that impede their ability to understand it. I quote these common misconceptions from his website (
  • Calculus, like the school mathematics, is about rules and procedures. Students think that calculus is difficult primarily because there are so many rules and procedures. 
  • Variables do not vary. Therefore rate of change is not about change. 
  • Integrals are areas under a curve. Students wonder, "How can an area represent a distance or an amount of work?" 
  • Average rate of change has little to do with rate of change. It is about the direction of a line that passes through two points on a graph. 
  • A tangent is a line that "just touches" a curve. 
  • Derivative is a slope of a tangent. The net result is that, in students' understandings, derivatives are not about rates of change.
The second bullet point is particularly common and problematic. In an expression such as
f(x) = x3 – 3x + 2,

many students see the expression f(x) as nothing more than a lengthy way of writing the dependent variable, and functions are seen as static objects that prescribe how to turn the input x into the output f(x). With this mindset, differentiation and integration are nothing more than arcane rules for turning one static object into another.

Choosing to define integrals as areas and derivatives as slopes, as is common in the standard curriculum, is equally problematic. It reinforces the notion that calculus is about computing values associated with geometric objects. To complicate matters, while area is a familiar concept, slope is far less real or meaningful to our students. Too many students never come to the realization that the real power of differentiation and integration arises from their interpretation as rate of change and as accumulation.

While the earliest uses of accumulation were for determining areas, those Hellenistic philosophers who mastered it also recognized its equal applicability to questions of volumes and moments. By the 14th century, European philosophers were applying techniques of accumulation to the problem of determining distance from knowledge of instantaneous velocity. None of these come easily to students who are fixated on integrals as areas.

Seeing the derivative as a slope is even more problematic, a static value of an obscure parameter. Differentiation arose from problems of interpolation for the purpose of approximating values of trigonometric functions in first millenium India, in understanding the sensitivity of one variable to changes in another in the work of Fermat and Descartes, and in relating rates of change as in Napier’s analysis of the logarithm and Newton’s Principia. Derivative as slope came quite late in the historical development of calculus precisely because its application to interesting questions is not intuitive.

These insights provide the starting point for Thompson’s reformation of Calculus I. His textbook, which is still a work in progress, can be accessed at See Thompson & Silverman (2008), Thompson et al. (2013), and Thompson & Dreyfus (2016) for additional background. I find it deliciously ironic that one of the first topics he tackles is the distinction among constants, parameters, and variables. If you look at the calculus textbooks of the late 18 th century through the middle of the 19 th century, this is exactly where they started. Somehow, we lost recognition of the importance of elevating this distinction for our students. Thompson goes on to spend considerable effort to clarify the role of a function as a bridge between two co-varying quantities. And then, he really breaks with tradition by first tackling integration, which he enters via problems in accumulation.

This accomplishes several desiderata. First, it ensures that students do not begin with an understanding of the integral as area, but as an accumulator. Second, it makes it much easier to recognize this accumulator as a function in its own right. Students struggle with recognizing the definite integral from a to the variable x as a function of x (see the section of last month’s column, Conceptual Understanding, that addresses Integration as Accumulation). Thompson begins by viewing the integrand as a rate of change function. The variable upper limit arises naturally. Third, and perhaps most important, it gives meaning to the Fundamental Theorem of Integral Calculus, that the derivative of an accumulator function is the rate of change function.

Differentiation can then be introduced in precisely the way Newton first understood it: Given a closed expression for the accumulator function, how can we find the corresponding rate of change function?

Next month, I will expound on exactly how Thompson introduces these steps, but for now I would like to conclude with a comparison of the two curricular innovations, that of Thompson and the approach described at the start of this column that emphasizes calculus as a tool for modeling dynamical systems. The latter does overcome the problem of student belief that the derivative is to be understood as the slope of the tangent. It brings to the fore the derivative as describing a rate of change. The problem is that it does nothing to clarify the role of the integral as an accumulator. In some sense, it makes it more difficult. As we teach calculus at Macalester, the integral is introduced purely as an anti-derivative, making it extremely difficult to give meaning to the Fundamental Theorem of Integral Calculus. I have to work very hard in the second semester to help students understand the integral as an accumulator and so justify that this theorem has meaning. In a very real sense, Thompson approach begins with this fundamental theorem.

Thompson, P.W. and Silverman, J. (2008). The concept of accumulation in calculus. In M.P. Carlson & C. Rasmussen (Eds.), Making the connection: Research and teaching in undergraduate mathematics (MAA Notes Vol. 73, pp. 43–52). Washington, DC: Mathematical Association of America.

Thompson, P.W., Byerley, C. and Hatfield, N. (2013). A conceptual approach to calculus made possible by technology. Computers in the Schools. 30:124–147.

Thompson, P.W. and Dreyfus, T. (2016). A coherent approach to the Fundamental Theorem of Calculus using differentials. In R. Göller. R. Biehler & R. Hochsmuth (Eds.), Proceedings of the Conference on Didactics of Mathematics in Higher Education as a Scientitific Discipline (pp. 355–359 ) Hannover, Germany: KHDM.

Saturday, April 1, 2017

Conceptual Understanding

You can now follow me on Twitter @dbressoud.

Continuing my series of summaries of articles that have appeared in the International Journal of Research in Undergraduate Mathematics Education (IJRUME), this month I want to briefly describe three studies that address issues of conceptual understanding. The first is a study out of Israel that probed student difficulties in understanding integration as accumulation (Swidan and Yerushalmy, 2016). The second is from France, exploring student difficulties with understanding the real number line as a continuum (Durand-Guerrier, 2016). The final paper, from England, explores a method of measuring conceptual understanding (Bisson, Gilmore, Inglis, and Jones, 2016).

Integration as Accumulation
To use the definite integral, students need to understand it as accumulation. In particular, the Fundamental Theorem of Integral Calculus rests on the recognition that the definite integral of a function f, when given a variable upper limit, is an accumulation function of a quantity for which f describes the rate of change. Pat Thompson (2013) has described the course he developed for Arizona State University that places this realization at the heart of the calculus curriculum.

We know that students have a difficult time understanding and working with a definite integral with a variable upper limit. The authors of the IJRUME paper suggest that much of the problem lies in the fact that when students are introduced to the definite integral as a limit of Riemann sums, they only consider the case when the upper and lower limits on the Riemann sum are fixed. The limit is thus a number, usually thought of as the area under a curve. Making the transition to the case where the upper limit is variable is thus non-intuitive.

The authors used software to explore student recognition of accumulation functions based on right-hand Riemann sums. They investigated student recognition of how the properties of these functions are shaped by the rate of change function. The experiment involved a graphing tool, Calculus UnLimited (CUL), in which students input a function and the software provides values of the corresponding accumulation function given by a right-hand sum with Δx = 0.5 (see Figure 1). Students could adjust the upper and lower limits, in jumps of 0.5. The software displays the rectangles corresponding to a right-hand sum. Students were not told that these were points on an accumulation function, merely that this was a function related to the initial function. They were encouraged to start with a lower limit of –3 and to explore the functions x2, x2 – 9, and then cubic polynomials, and to discover what they could about this second function. Students received no further prompts.

Figure 1: The CUL interface. Taken from Swidan and Yerushalmy (2016), page 33.

Thirteen pairs of Israeli 17-year olds participated in the study. They all had been studying derivatives and indefinite integrals, but none had yet encountered definite integrals. Each pair spent about an hour exploring this software. Their actions and remarks were video-taped and then analyzed.

One of the interesting observations was that the key to recognizing that the second function accumulates areas came from playing with the lower limit. Adjusting the upper limit simply adds or removes points, but adjusting the lower limit moves the plotted points up or down. Once students realized that the point corresponding to the lower limit is always zero, they were able to deduce that the y-value of the next point is the area of the first rectangle, and that succeeding points reflect values obtained by adding up the areas of the rectangles. Rectangles below the x-axis were shaded in a darker color, and students quickly picked up that they were subtracting values. Seven of the thirteen pairs of students went as far as remarking on how the concavity of the accumulation function is related to the behavior of the original function.

This work suggests that a Riemann sum with a variable upper limit is more intuitive than a definite integral with a variable upper limit. In addition, it appears that students can discover many of the essential properties of a discrete accumulation function if allowed the opportunity to experiment with it.

Understanding the Continuum
The second paper explores student difficulties with the properties of the real number line and describes an intervention that appears to have been useful in helping students understand the structure of the continuum. Mathematicians of the nineteenth century struggled to understand the essential differences between the continuum of all real numbers and dense subsets such as the set of rational numbers. It comes as no surprise that our students also struggle with these distinctions.

The author analyzes the transcripts from an intervention described by Pontille et al. (1996). It began with the following question: Given an increasing function, f, ( x < y implies f(x) ≤ f(y) ) from an ordered set S into itself, can we conclude that there will always exist an element s in S for which f(s) = s? The answer, of course, depends on the set. The intervention asks students to answer this question for four sets: a finite set of positive integers, the set of numbers with finite decimal expansions in [0,1], the set of rational numbers in [0,1], and the entire set [0,1]. In the original work, this question was posed to a class of lycée students in a scientific track. Over the course of an academic year, they periodically returned to this question, gradually building a refined understanding of the structure of the continuum. The author’s analysis of the transcripts from these classroom discussions is fascinating.

Durand-Guerrier then posed this same question to a group of students in a graduate teacher- training program. In both cases, students were able to answer the question in the affirmative for the finite set, using an inductive proof or reductio ad absurdum. Almost all then tried to apply this proof to the dense countable sets. Here they ran into the realization that there is no “next” number. The graduate students, given only an hour to work on this, did not get much further. The lycée students did come to doubt that it was always true for these sets. As they began to think about the “holes” these sets left, they were able to construct counter-examples.

The continuum provides the most difficulty. The lycée students were eventually able to prove that it is true in this instance, but only after being given the hint to consider the set of x in [0,1] for which f(x) > x and to draw on the property of the continuum that every bounded set has a least upper bound.

Measuring Conceptual Understanding
The last paper in this set addresses the problem of measuring conceptual understanding. We know that students can be proficient in answering procedural questions without the least understanding of what they are doing or why they are doing it. But measuring conceptual understanding is difficult. A meaningful assessment with limited possible answers, such as a concept inventory, requires a great deal of work to develop and validate. Open-ended questions can provide a better window into student thinking and understanding, but consistent application of scoring rubrics across multiple evaluators is hard to achieve.

The authors build a solution from the observation that it is far easier to compare the quality of the responses from two students than it is to compare one student’s response against a rubric. They therefore suggest asking a simple, very open-ended question, scored by ranking student responses, which is achieved by pairwise comparisons. As an example, to evaluate student understanding of the derivative, they provided the prompt,
Explain what a derivative is to someone who hasn’t encountered it before. Use diagrams, examples and writing to include everything you know about derivatives.
The 42 students in this study first read several examples of situations involving velocity and acceleration (presumably to prompt them to think of derivatives as rates of change rather than a collection of procedures) and were then given 20 minutes to write their responses to the prompt. 

Afterwards, 30 graduate students each judged 42 pairings. The authors found very high inter-rater reliability (r = .826 to .907). In fact, they found that comparative judgments appeared to do a better job of evaluating conceptual understanding than did Epstein’s Calculus Concept Inventory (Epstein, 2013).

Similar studies were undertaken to evaluate student understanding of p-values and 11- to 12-year- olds understanding of the use of letters in algebra. Again, there was very high inter-rater reliability, and in these cases there were high levels of agreement with established instruments.

This approach constitutes a very broad method of assessment, but it does enable the instructor to get some idea of what students are thinking and how they understand the concept at hand. It can be used even with large classes because it is not necessary to look at all possible pairs to get a meaningful ranking.

The three papers referenced here are very different in focus and goal, but I do see the common thread of searching for ways to encourage and assess student understanding. After all, that is what teaching and learning is really about.

Bisson, M.-J., Gilmore, C., Inglis, M., and Jones, I. (2016). Measuring conceptual understanding using comparative judgement. IJRUME. 2:141–164.

Durand-Guerrier, V. (2016). Conceptualization of the continuum, an educational challenge for undergraduate students. IJRUME. 2:338–361.

Epstein, J. (2013). The Calculus Concept Inventory - measurement of the effect of teaching methodology in mathematics. Notices of the American Mathematical Society, 60, 1018–27.

Pontille, M. C., Feurly-Reynaud, J., & Tisseron, C. (1996). Et pourtant, ils trouvent. Repères IREM, 24, 10–34.

Swidan, O. and Yerushalmy, M. (2016). Conceptual structure of the accumulation function in an interactive and multiple-linked representational environment. IJRUME. 2:30–58.

Thompson, P.W., Byerley, C., and Hatfiled, N. (2013). A Conceptual approach to calculus made possible by technology. Computers in the Schools. 30:124–147.

Wednesday, March 1, 2017

MAA Calculus Studies: Use of Local Data

You can now follow me on Twitter @dbressoud.

From our 2012 study, Characteristics of Successful Programs in College Calculus (NSF #0910240), the most successful departments had a practice of monitoring and reflecting on data from their courses. When we surveyed all departments with graduate programs in 2015 as part of Progress through Calculus (NSF #1430540), we asked about their access to and use of these data, what we are referring to as “local data.”

The first thing we learned is that a few departments report no access to data about their courses or what happens to their students. For almost half, access is not readily available (see Table 1). When we asked, “Which types of data does your department review on a regular basis to inform decisions about your undergraduate program?”, most departments review grade distributions and pay attention to end of term student course evaluations (Table 2). Between 40% and 50% of the surveyed departments correlate performance in subsequent courses with the grades they received in previous courses and look at how well placement procedures are being followed. Given how important it is to track persistence rates (see The Problem of Persistence, Launchings, January 2010), it is disappointing to see that only 41% of departments track these data. Regular communication with client disciplines is almost non-existent.

Table 1. Responses to the question, “Does your department have access to data 
to help inform decisions about your undergraduate program? PhD indicates 
departments that offer a PhD in Mathematics. MA indicates departments for 
which the highest degree offered in Mathematics is a Master’s.

Table 2. Responses to the question, “Which types of data does your department
 review on a regular basis to inform decisions about your undergraduate program?”

We also asked departments to describe the kinds of data they collect and regularly review. Several reported combining placement scores, persistence, and grades in subsequent courses to better understand the success of their program. Some of the other interesting uses of data included universities that
  • Built a model of “at-risk” students in Calculus I using admissions data from the past seven years. Using it, they report “developing a program to assist these students right at the beginning of Fall quarter, rather than target them after they start to perform poorly.” 
  • Surveyed calculus students to get a better understanding of their backgrounds and attitudes toward studying in groups. 
  • Collected regular information from business and industry employers of their majors. 
  • Measured correlation of grade in Calculus I with transfer status, year in college, gender, whether repeating Calculus I, and GPA. 
  • Used data from the university’s Core Learning Objectives and a uniform final exam to inform decisions about the course (including the ordering of topics, emphasis on material and time devoted to mastery of certain concepts, particularly in Calculus II). 
  • Reviewed the performance on exam problems to decide if a problem type is too hard, a problem type needs to be rephrased, or an idea needs to be revisited on a future exam.
The intelligent use of data to shape and monitor interventions is a central feature of the large- scale initiatives that are now underway. To mention just one, the AAU STEM Initiative (Association of American Universities, a consortium of 62 of the most prominent research universities in the U.S. and Canada) has established a Framework for sustainable institutional change. It can be found at (Figure 1).

Figure 1. AAU STEM Initiative Framework

The three levels of change are subdivided into topics, each of which links to programs at member universities that illustrate work on this aspect of the framework.

Cultural change encompasses
  1.  Aligning incentives with expectations of teaching excellence. 
  2.  Establishing strong measures of teaching excellence. 
  3.  Leadership commitment.
Scaffolding includes
  1.  Facilities. 
  2.  Technology. 
  3.  Data. 
  4. Faculty professional development.
Pedagogy is comprised of
  1.  Access. 
  2. Articulated learning goals. 
  3. Assessments. 
  4. Educational practices.
In addition, AAU is now finalizing a list of “Essential Questions” to ask about the institution, the college, the department, and the course, illustrating the types of data and information that should be collected and pointing to helpful resources. This report, which should be published by the time this column appears, will be accessible through the AAU STEM Initiative homepage at

Wednesday, February 1, 2017

MAA Calculus Study: PtC Survey Results

You can now follow me on Twitter @dbressoud.

In spring 2015 the MAA’s Progress through Calculus (PtC) grant (NSF#1430540) surveyed all U.S. Departments of Mathematics that offer a graduate degree in Mathematics to learn about departmental practices, priorities, and concerns with respect to their mainstream courses in precalculus through single variable calculus. I have reported on some of the results from this study in November, 2015. This month’s column describes a variety of data relative to mainstream Calculus I that were collected in that survey. The full report can be found under PtC Reports (link from

The survey was sent to the chairs of all departments of mathematics in the United States that offer a graduate degree in Mathematics (PhD or Master’s). We received responses from 134 of the 178 PhD-granting universities (75%) and 89 of the 152 Master’s-granting universities (59%).

Given how ineffective the standard precalculus course is known to be (see my Launchings column from October, 2014), we were particularly interested in efforts to teach precalculus topics concurrently with calculus. Accomplishing this through a stretched-out Calculus I is now fairly common (20 of 222 respondents use this approach to incorporate precalculus topics into Calculus I). Eleven universities have courses or options with extra hours to allow time on precalculus, and three offer precalculus courses designed to be taken concurrently with Calculus I. We also found 14 universities with an accelerated calculus specifically designed to meet the needs of students entering with AP® Calculus credit. Three universities have special lower credit courses that enable students who begin in a non-mainstream Calculus I to transition to mainstream calculus.

Table 1: Number of surveyed universities that reported using each
of the listed variations in single variable calculus classes.

Every five years, CBMS surveys departments of mathematics in the U.S. to get enrollment numbers, but those are only gathered for the fall term. In this survey, we were particularly interested in how these numbers vary over the full year, both academic and summer terms. While we only have results for a sample of universities, and no undergraduate colleges, the numbers are large enough, 150,000 in Precalculus, 200,000 in mainstream Calculus I, and 160,000 in subsequent mainstream single variable classes, to get a good idea of how these enrollments distribute over the year. For Precalculus, 57% of the enrollment occurs in the fall term. Fall term accounts for 60% of the Calculus I students. Not surprisingly, Calculus II is predominantly a second-term course (47%), but 40% of the students who take Calculus II do so in the fall. The distribution among the terms is complicated by the fact that some universities are on a quarter system, others on semesters. What I have labeled 2nd Term, is either spring semester or winter quarter. The 3rd Term refers to the spring quarter for those on a quarter system. Summer aggregates all summer terms. Figure 1 shows actual numbers from the universities that responded to give an idea of how enrollments drop off. For the purposes of the survey, “Precalculus” was defined as the last course before mainstream Calculus I. It is variously called Precalculus, College Algebra, College Algebra with Trigonometry, or Preparation for Calculus. Calculus II includes all mainstream single variable calculus courses that follow Calculus I. On a semester system, there is usually just one. On a quarter system, there usually are two such courses.

Figure 1: Distribution of enrollments by term among the 205 universities that respond to this question.
2nd term = spring semester or winter quarter. 3rd term = spring quarter.
Calculus II includes all mainstream single variable calculus classes that follow Calculus I.

The number of contact hours (including recitation sections) in Calculus I averaged 4.17 (SD = 0.77) at PhD-granting universities and 4.25 (SD = 0.64) at Masters-granting universities. The DFW rate in mainstream Calculus I was 21% (SD =12.2), at PhD-granting universities and 25% (SD = 13.7) at Masters-granting universities.

The next table (Table 2) reports the fraction of universities in which Calculus I is frequently taught by each type of instructor. For each category of instructor, the options were “Never,” “Rarely,” or “Frequently.”

Table 2: Percentage of universities for which each category of
instructor frequently teaches mainstream Calculus I.

Recitation sections were far more common at PhD-granting universities. All classes have recitation sections for 49% of the institutions, some classes at 6%, and there are no recitation sections at 45% of the universities. For Masters-granting universities, the percentages were 18% for all classes, 6% for some classes, and 76% for no classes.

We also found that active learning was much more common at Masters-granting universities than PhD-granting universities. Figures 2 and 3 record primary instructional format for mainstream Calculus I. “Some active learning” includes techniques such as use of clickers or think-pair-share. “Minimal lecture” includes Inquiry Based Learning and flipped classes. “Other” usually means too much variation to be able to identify a primary instructional format. We did find that 35% of the PhD-granting universities did report having at least some sections that were using active learning approaches.

Figure 2. Primary instructional format for regular classes
(not recitation sections) at 214 PhD-granting universities.
Figure 3. Primary instructional format for regular classes
(not recitation sections) at 109 Masters-granting universities.

At 73% of the PhD-granting universities and 74% of the Masters-granting universities that offer recitation sections, they are simply homework help, Q&A, and review. Recitation sections are built around active learning approaches 21% of the time at PhD-granting universities, 4% of the time at Masters-granting universities.

Table 3 reports which elements of mainstream Calculus I are common across all sections. We see much more uniformity at PhD-granting universities. In view of our findings from the earlier Characteristics of Successful Programs in College Calculus that coordination of course elements was one of the significant factors of successful calculus programs (see my Launchings column from January 2014), the results of this study suggest a great deal of room for improvement.

Table 3: Percentage of reporting universities that have these elements across all sections of mainstream Calculus I.

Another aspect of coordination that was characteristic of the most successful programs was the practice of regular meetings of the course instructors. As shown in Table 4, there is also a great deal of room for improvement here.

Table 4: Response to "When several instructors are teaching in the same term,
how often do they typically meet as a group to discuss the course?"

The situation at PhD-granting universities is disappointing. The primary means of instruction is still large lecture with few or no structured opportunities for students to reflect on what is being presented to them, supplemented by recitation sections in which graduate students simply go over homework and answer student questions. At the Masters-granting universities, where classes are smaller and there is more emphasis on teaching, there is little coordination, often resulting in highly variable instruction. But there is room for hope. While there is no previous study with comparable data, there appears to be good deal of experimentation. My own experience in visiting these predominantly large public universities is that they are aware that what they are doing is not working, and they are looking for ways to improve what happens in this critical sequence.

Sunday, January 1, 2017

IJRUME: Approximation in Calculus

You can now follow me on Twitter @dbressoud.

In an earlier column, "Beyond the Limit, III," I talked about how Michael Oehrtman and colleagues have been able to use approximation as a unifying theme for single variable calculus that helps students avoid many of the confusing aspects of the language of limits. I also pointed out that this is hardly a new idea, having been used by many textbook authors including Emil Artin in A Freshman Honors Course in Calculus and Analytic Geometry and Peter Lax and Maria Terrell in Calculus with Applications. The IJRUME research paper I wish to highlight this month, “A study of calculus instructors’ perceptions of approximation as a unifying thread of the first-year calculus” by Sofronas et al., looks at how common this approach actually is.

The authors address four research questions:

  1. Do calculus instructors perceive approximation to be important to student understanding of first-year calculus? 
  2. Do calculus instructors report emphasizing approximation as a central concept and-or unifying thread in the first-year calculus? 
  3. Which approximation ideas do calculus instructors believe are “worthwhile” to address in first-year calculus?  
  4. Are there any differences between demographic groups with respect to the approximation ideas they teach in first-year calculus courses? 
They surveyed calculus instructors at 182 colleges and universities, collecting 279 responses.

To the first two questions, 89% agreed that approximation is important, but only 51% considered it a central concept, and only 40% found that it provides a unifying thread (see Figure 1). For those who did consider it central and-or unifying, the reasons that they gave included: (a) it illuminates reasons for studying calculus, (b) most functions are not elementary and approximation is helpful in dealing with such functions, (c) approximation facilitates the understanding of fundamental concepts including limit, derivative, integral, and series, (d) linear approximations lie at the foundation of differential calculus, and (e) an emphasis on approximation resonates with the instructors personal interests in applied mathematics or numerical analysis.
Figure 1: Graph depicting participants’ perceptions of approximation (N=214).
 Source: Sofronas et al. 2015.

For those who did not consider approximation to be central or unifying, many stated that it is not sufficiently universal, only important in a few contexts such as motivating the definition of the derivative at a point or the value of a definite integral. Many stated other unifying threads such as limit or the study of change. Some objected to an emphasis on approximation because of its inevitable ties to the use of technology. There were also a large number of obstacles to the use of approximation that instructors identified. These included: (a) an overcrowded syllabus that left no room for the instructor to develop a unifying thread, (b) required adherence to a curriculum emphasizing procedural facility, (c) students with weak preparation who are not prepared to understand the subtleties of approximation arguments, (d) lack of access to technology, (e) lack of familiarity with how to use approximation ideas in developing calculus. I personally find these obstacles to be very sad, in particular the assumption on the part of many instructors that the only way to get through the required syllabus or to enable students to pass the course is to focus exclusively on memorizing procedures.

Jumping ahead to the fourth question, the authors found that the single factor that most highly correlated with emphasizing approximation as a central concept and-or unifying thread was having served on either a local or national calculus committee. Not surprisingly, this factor was also highly correlated with number of years teaching calculus, rank, being the recipient of a teaching award, and having published or presented on a calculus topic.

To the third research question, the combined list of topics gleaned from all of the responses truly spans first-year calculus: numerical limits, definition of limit, definition of the derivative, derivative values, tangent line approximations, differentials, error estimation, function change, functions roots and Newton’s method, linearization, integration, Riemann sums, Taylor polynomials and Taylor series, Newton’s second law, Einstein’s equation for force, L’Hospital’s rule, Euler’s method, and the approximation of irrational numbers. One unexpected outcome of the survey is that several of the respondents commented that answering this survey about their use of approximation in first-year calculus opened their eyes to the opportunity to use it as a unifying theme. As one respondent wrote,
I agree that approximation is an important concept AND after taking this survey I can see teaching calculus using approximation as the main theme. The rate of change theme offers many opportunities for real-life applications but I can see how using approximations from the beginning would offer other opportunities. It is an interesting idea, and I would love to incorporate more of this theme into my lessons.
For those who are interested in following up on the use of approximation as a unifying thread, this article also supplies a wealth of background information that includes a discussion of the different ways in which approximation can be used and the research evidence for its effectiveness as a guiding theme in developing student understanding of limits, derivatives, integrals, and series.


Artin, E. (1958). A Freshman Honors Course in Calculus and Analytic Geometry Taught at Princeton University. Buffalo, NY: Committee on the Undergraduate Program of the Mathematical Association of America

Lax, P. & Terrell, M.S. (2014). Calculus with Applications, Second Edition. New York, NY: Springer.

Sofronas, K.S., DeFranco, T.C., Swaminathan, H., Gorgievski, N., Vinsonhaler, C., Wiseman, B., Escolas, S. (2015). A study of calculus instructors’ perceptions of approximation as a unifying thread of the first-year calculus. Int. J. Res. Undergrad. Math. Ed. 1:386–412 DOI 10.1007/s40753-015- 0019-5