Checking 4,142 MS Clinical Records Against a Research Cohort

Electronic medical records hold an enormous amount of clinical information about multiple sclerosis (MS), and almost none of it was recorded with research in mind. The question that decides whether that information is usable is simple to state and hard to answer: is data pulled out of routine clinical notes good enough to trust? Damotte, Lizée and colleagues answered it for one large academic centre. They built an algorithm to find MS patients in the University of California San Francisco record system, which spans roughly 3.5 million patients and 390 million notes, de-identified the data, and extracted clinical measures from the free text. The distinctive step came next: for patients who were also enrolled in the centre's prospective research cohort, they compared what the clinical record said against what trained research staff had recorded. By the authors' account it is the first study to test data extracted from clinical records against a carefully curated MS research database, and they released the code openly so that other centres can run it.

Finding the Patients
The algorithm sorts patients into four tiers, from a Well-Defined MS Group down to three Probable groups, by combining diagnosis codes, codes for conditions that exclude MS, prescriptions for MS-specific treatments, and text matching in the notes. It identified 4,142 patients. To measure how many it would miss, the team checked it against 337 research participants known to have MS: the Well-Defined group alone captured 318 of them, a sensitivity of 94.4%, and all four groups together captured 323, or 95.9%. Clinicians then reviewed a sample of charts to measure how often a patient the algorithm labelled as MS truly had it, giving a positive predictive value of 95 to 97% for the Well-Defined group. The balance was chosen deliberately, favouring precision over completeness, on the reasoning that disease research is better served by a cohort whose members certainly have MS than by a larger one diluted with misclassified patients. The whole pipeline ran in two days of computing time.

Pulling Numbers Out of Free Text
Natural language processing across 37,001 visit notes produced 25,620 values for four measures: the Expanded Disability Status Scale (EDSS), the timed 25-foot walk, MS subtype, and year of onset. Manual review of 323 visits from 134 patients measured how accurate those extractions were. EDSS came out strongest, with performance of 97.6%, positive predictive value of 96.9% and sensitivity of 88.7%. The timed walk followed at 96.4%, 92.7% and 86.8%, and subtype at 92.9%, 91.7% and 73.9%. Year of onset proved hardest, at a sensitivity of 64.2% and positive predictive value of 76.5%, because clinicians describe when symptoms began in so many different ways. Reported per variable like this, the results tell anyone building on the work exactly which measures are ready to use and which still need development.

Clinical Values Match Research Values
For patients present in both datasets, agreement was high. EDSS reached an intraclass correlation of 0.87 (0.85–0.88; 245 patients, 861 paired values), the timed walk 0.79 (0.76–0.81; 217 patients, 684 pairs), and year of onset 0.87 (0.81–0.91; 90 patients). Subtype showed substantial agreement with a Cohen's kappa of 0.65. One small detail is revealing about how clinical data differs in texture: timed walk values with no decimal or one decimal were 5.3 and 3.4 times more common in the clinical record than in the research database, a sign of rounding in the clinic. The authors also disclose the one route by which agreement could be overstated, since a handful of clinic visits share dates and values with research visits: 37 of 915 visits with an EDSS value, 2 of 833 with a timed walk, and 28 of 873 with a subtype.

Where the Two Sources Differ, and Why That Is Useful to Know
The most practically important difference sits in the subtype table. Of 42 patients classed as secondary progressive in the research database, 23 carry the same label in the clinical record, while 12, or 28.6%, are still recorded as relapsing-remitting. The clinical record lags in registering conversion to progressive disease. Anyone planning to study progression from routine data needs to know that in advance, and this paper supplies the size of the effect. Two other points concern demographics: demographic labels in the record matched research ancestry codes for only 58.5% of the 337 research patients, and ethnicity was missing for about a quarter of the 4,142 record-derived patients. A small counting note for anyone citing the subtype table: its cells add up to 223 rather than the 234 given in the text, and recomputing the kappa from the 223 printed cells gives 0.648, matching the reported 0.65, so the cells are consistent and 234 is the figure to correct.

The Record-Derived Cohort Behaves Like MS
Agreement for 337 people shows the extraction works; reproducing known natural history shows the resulting cohort resembles the disease. Here the results are reassuring. The sex ratio came out at 2.6 women to every man, within the 2 to 3 range usually reported for MS. On that measure the clinical cohort actually looks more like the general MS population than the research cohort does, whose ratio from the same table is about 1.9. Disability followed the expected ordering across subtypes, with median EDSS of 1.5 for clinically isolated syndrome, 2 for relapsing-remitting and 6 for progressive disease (P < 0.001 for each comparison). Men had higher EDSS than women, a median of 3 against 2.5 (P < 0.001). EDSS rose with age at examination and with disease duration, both at P < 0.001.

Measuring the Bias, and What Comes Next
The authors examine the weakness most specific to clinical data rather than leaving it for readers to raise. Patients with moderate disability, EDSS between 3 and 5.5, generate the most clinic encounters and the most extracted values per year (P = 2×10⁻³ for each), probably reflecting the patient mix of an academic specialty clinic. That is a bias any centre building a cohort this way should measure in its own data, and the paper shows how. The dataset covers about four years, from 2011 to 2015, which the authors note is too short for long-term disability trajectories. The next steps they set out are concrete: pooling record-derived data across the University of California medical centres through a shared ontology, recording MS-specific measures in structured fields rather than free text so extraction becomes more reliable, and building large reference cohorts against which an individual patient's course can be compared. The de-identifying and extraction code is released under a BSD licence, which is the part of this work most likely to outlast it.

Disclaimer: This blog post is based on the cited research article and is intended for informational purposes only. It is not intended to provide medical advice. Please consult with a healthcare professional for any health concerns.

Reference:
Damotte, V., Lizée, A., Tremblay, M., Agrawal, A., Khankhanian, P., Santaniello, A., Gomez, R., Lincoln, R., Tang, W., Chen, T., Lee, N., Villoslada, P., Hollenbach, J. A., Bevan, C. D., Graves, J., Bove, R., Goodin, D. S., Green, A. J., Baranzini, S. E., Cree, B. A. C., Henry, R. G., Hauser, S. L., Gelfand, J. M., & Gourraud, P.-A. (2019). Harnessing electronic medical records to advance research on multiple sclerosis. Multiple Sclerosis Journal, 25(3), 408–418. https://doi.org/10.1177/1352458517747407