Readiness Test Analysis

This plots students' final course grades vs. their readiness test scores and does a linear regression.

Typically these data have lots of scatter and the quality of fit is poor, seeming to suggest that the readiness test doesn't tell us much. But that's overly pessimistic. Our goal isn't really to predict a student's final grade; rather, we're only trying to estimate the likelihood of their passing vs. failing, which is easier. The other tabs here try to assess the readiness test on that basis.

This shows readiness test scores grouped according to whether a student passed or failed the course. A boxplot shows test score quartiles for each group. One hopes that the medians differ by at least one quartile, as an indicator that the test has at least some ability to discriminate between students who end up passing and those who fail.

This shows, for each test question, the correlation (aka Pearson correlation coefficient, aka point-biserial correlation coefficient) of final course grade with a correct response to that question. Dark blue indicates a statistically significant positive correlation. Questions with low correlation are poor predictors of final grade; these are the questions to change or eliminate if you want to improve the test.

This shows a contingency table for each question, dividing students into four groups according to whether they got that question right/wrong and whether they passed/failed the course. Plots in bold indicate questions that are good pass/fail discriminators, in that there is a statistically significant relationship between pass/fail and right/wrong.

This shows, for each test question, the proportion of students who answered that question correctly.

The most useful/informative questions will have a correct response rate around 0.5; a question with a very high or low rate of correct responses will yield little information, since most responses to that question can be predicted in advance. In trying to improve a readiness test, it makes sense to make somewhat easier the problems with a low rate of correct responses, and to make somewhat harder the problems with a high rate of correct responses.

This shows the cross-correlation matrix for responses to individual questions. Questions have been re-ordered to highlight any clusters that might occur. If responses to two questions are highly correlated then those questions aren't really independent and you might as well eliminate one of them, especially if you're looking to make room for better questions.

These plots help interpret readiness test scores when advising students. The graph on the left plots, for each test score X, the estimated probability (with confidence intervals) that a student will fail the course, given that their readiness test score was less than X. Similarly, the graph on the right plots, for each test score Y, the estimated probability that a student will pass the course, given that their readiness test score was greater than Y.

If the readiness test does what it's supposed to then it should be possible, by interpreting the graphs above, to find thresholds X and Y such that you can give students reliable, evidence-based advice e.g. of the form, "If your test score was less than X then there is an 80% chance you will fail the course; if your test score was greater than Y then there is an 80% chance you will pass the course; if your test score falls between X and Y then your success in the course strongly depends on factors other than mere readiness."

This web app is to help analyze and improve the performance of "readiness tests" for e.g. 1st-year math courses, by comparing final course grades against readiness test scores and responses to individual questions on the test.

Several diagnostics are calculated in the various tabs. Use the controls on the left to select an existing data set or upload your own, and to set parameters for the analysis.

Data must be provided in a .csv file with the following specifications:

  • The separator character must be a comma.
  • Each line records the final course grade and readiness test question responses for one student.
  • The first line provides column headings which must include at least "Course", "Q1", "Q2", etc. Any additional columns will be ignored. Column headings are case-insensitive.
  • The "Course" column records each student's final numerical course grade (out of 100). A negative or non-numeric course grade will be interpreted as an incomplete or withdrawal; how these are treated in the analysis is determined by the controls on the left.
  • Columns "Q1", "Q2", etc. record each student's responses to individual questions on the readiness test, with correct responses indicated by a 1 and incorrect responses indicated by a 0 or missing value. There are no limits on the number of test questions.

The data set must be uploaded to the server that runs this app. If you use the app to upload your own, it will only be stored on the server for the duration of your session. If you want your data set available here permanently, feel free to contact me directly and I will be happy to upload it for you.

The app is implemented in the R language using the Shiny package.

Richard Taylor

Department of Mathematics and Statistics

Thompson Rivers University

January 2019