Reading your result

01Read the chart in steps

Standards written for single-case research set out how to read a chart by eye (Kratochwill et al., 2010). First, the before-period has to settle into a predictable pattern; otherwise there is nothing steady to compare with. Then each period is read on its own, then beside the next one, and last the whole run is checked for the same change showing up again.

Six features are read along the way:

• Level: the period’s average.
• Trend: whether readings drift up or down within the period.
• Spread: how much readings swing.
• How fast a change shows: the last three readings before a change against the first three after it.
• Overlap: how many after-readings fall within the range of the before-readings. Less overlap means a clearer change.
• Whether repeats agree: whether the same pattern comes back when the change is made again.

The standards count an effect as shown only when the same change appears three times, at three different points in time. Their authors call the number three a convention, without a formal basis (Kratochwill et al., 2010).

Repeats matter because one comparison can’t tell a difference that belongs to the person from ordinary ups and downs. In simulated trials, only repeating the comparison in the same person could tell the two apart (Senn, 2016).

02The same chart, read differently

Reading by eye is a judgement, and careful people differ. A review of 19 studies found that people rating the same single-case charts agreed about three times in four, and in some conditions agreement was low (Ninci et al., 2015).

Simple aids help. In one study, brief training in a structured rule for reading before-and-after charts raised correct readings from 55% to 94% in one group, and from 71% to 95% in another (Fisher et al., 2003). One easy aid is a flat line at the before-days’ average, carried on across the after-days, so each after-day can be read against what the before-days predict. The standards make the same comparison, between the pattern projected from one period and what actually followed (Kratochwill et al., 2010).

It also helps to decide in advance what will count. In one study, people who tested a food on themselves for 12 days disagreed about what difference would matter (Karkar et al., 2017). Some would act on a steady gap of one or two points. Others wanted the days with the food to sit near the top of the scale.

Single-case research shows readers the raw chart, so they can make their own reading (Kratochwill et al., 2010). A reporting checklist lists 26 items a single-case write-up should cover (Tate et al., 2016). The same habit suits a one-person test: keep the whole chart and the notes, and write down how you decided.

03“Inconclusive” is an answer

Unclear results are common, even in formal trials. In a review of one-person medical trials, 38% of the people whose later decisions were reported had ambiguous results (Gabler et al., 2011).

The standards list what stops a change from counting: an unsteady before-period, a lot of swing, a long delay, a lot of overlap, or repeats that don’t match (Kratochwill et al., 2010). “Inconclusive” is the honest word for that, and it doesn’t mean “no difference”. In the 12-day food study, many people took “no evidence” to mean the food made no difference to them, though the test was built to look for a change, not to rule one out (Karkar et al., 2017). Its authors added that a short test is more likely to miss a real but small change.

An inconclusive result is a reason to leave the question open, not to read more into the chart than it shows.

04What you expect can steer what you see

Beliefs shape how people read their own results. In a study of a self-experiment app, some people trusted results most when they matched what they already felt. Some found reasons to doubt results that went against their beliefs, such as too few days or a few unusual ones (Daskalova et al., 2021).

In the 12-day food study, one person set aside the one clear result she got. Another kept believing caffeine affected him, despite a result showing no evidence. The authors saw hints of confirmation bias (Karkar et al., 2017).

Writing down what you expect, and what change would count, before you start gives you something to check your reading against.

05What the studies found

• In 2010, a panel of seven experts wrote single-case standards for the US What Works Clearinghouse (Kratochwill et al., 2010). Reading by eye runs in four steps and judges six features. An effect counts as shown only after three demonstrations at three different times, a rule the panel calls professional convention.
• In a 2016 statistics paper, simulated trials showed that one comparison per person can’t reveal whether an apparent response is personal and repeatable, but repeating the comparison in the same people can (Senn, 2016). Senn concluded that the common belief in a strongly personal response to treatment lacks sound statistical evidence (Senn, 2016).
• A 2015 review pooled 19 studies (32 estimates) in which several people rated the same single-case charts (Ninci et al., 2015). Overall agreement was 0.76. Across conditions such as the raters’ experience and the use of visual aids, it ranged from low to adequate.
• In a 2003 paper, structured rules for reading before-and-after charts were tested on simulated data, then taught (Fisher et al., 2003). They raised far fewer false alarms than an older chart method, and caught real effects more often than two statistical tests. Brief training raised correct readings from 55% to 94% in 5 staff, and a 15-minute version from 71% to 95% in 87 people.
• In a 2017 study, 15 adults with irritable bowel syndrome each spent 12 days testing one item, such as caffeine or lactose, had or avoided at breakfast on randomly assigned days (Karkar et al., 2017). “No evidence” was the most common result; three people’s results showed possible evidence on a symptom, and one person’s showed strong evidence on one of her five symptoms. People’s own readings often differed from the computed result, in both directions.
• The 2016 SCRIBE statement, built from two online surveys and a two-day meeting of experts, lists 26 items that reports of single-case research should cover (Tate et al., 2016).
• A 2011 review covered 108 published one-person medical trials with 2,154 people (Gabler et al., 2011). About half chose the better option with a statistical cut-off, a quarter by comparing graphs, and a fifth by a cut-off for a clinically important difference. Of 488 people whose later decisions were reported, 54% decided in line with their result, 8% against it, and 38% had ambiguous results.
• In a 2021 study, 16 local adults who called themselves healthy used a self-experiment app for two weeks, and 16 students taught experimental design used a later version (Daskalova et al., 2021). Some people skipped the condition they thought less healthy, and some doubted results that went against their beliefs. Ten of the students changed what they measured after noticing other influences on it.

06Sources on this page

Each source is listed once, with links to its record. The reference list says what was read for each, and when it was checked.

1. Daskalova et al., 2021Daskalova N, Kyi E, Ouyang K, Borem A, Chen S, Park SH, et al. 2021. Self-E: Smartphone-Supported Guidance for Customizable Self-Experimentation. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–13.DOI 10.1145/3411764.3445100
2. Fisher et al., 2003Fisher WW, Kelley ME, Lomas JE. 2003. Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs. Journal of Applied Behavior Analysis 36(3):387–406.DOI 10.1901/jaba.2003.36-387 · PubMed 14596583 · PMC1284456
3. Gabler et al., 2011Gabler NB, Duan N, Vohra S, Kravitz RL. 2011. N-of-1 trials in the medical literature: a systematic review. Medical Care 49(8):761–768.DOI 10.1097/MLR.0b013e318215d90d · PubMed 21478771
4. Karkar et al., 2017Karkar R, Schroeder J, Epstein DA, Pina LR, Scofield J, Fogarty J, et al. 2017. TummyTrials: A Feasibility Study of Using Self-Experimentation to Detect Individualized Food Triggers. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), 6850–6863.DOI 10.1145/3025453.3025480 · PubMed 28516175 · PMC5432136
5. Kratochwill et al., 2010Kratochwill TR, Hitchcock J, Horner RH, Levin JR, Odom SL, Rindskopf DM, et al. 2010. Single-case designs technical documentation. What Works Clearinghouse, US Institute of Education Sciences. Version 1.0 (pilot), June 2010.ies.ed.gov
6. Ninci et al., 2015Ninci J, Vannest KJ, Willson V, Zhang N. 2015. Interrater agreement between visual analysts of single-case data: a meta-analysis. Behavior Modification 39(4):510–541.DOI 10.1177/0145445515581327 · PubMed 25878161
7. Senn, 2016Senn S. 2016. Mastering variation: variance components and personalised medicine. Statistics in Medicine 35(7):966–977.DOI 10.1002/sim.6739 · PubMed 26415869 · PMC5054923
8. Tate et al., 2016Tate RL, Perdices M, Rosenkoetter U, Shadish W, Vohra S, Barlow DH, et al. 2016. The Single-Case Reporting Guideline In BEhavioural Interventions (SCRIBE) 2016 Statement. Physical Therapy 96(7):e1–e10. Published at the same time in several journals; this is the Physical Therapy copy.DOI 10.2522/ptj.2016.96.7.e1 · PubMed 27371692