Discussion Board

Expand all | Collapse all

COVID and cognitive bias

  • 1.  COVID and cognitive bias

    Posted 04-26-2020 10:14

    Hello to the list.  Thought I would share with the list an essay on COVID and bias I co-authored with my colleague Paritosh Prasad MD.  All stay healthy!

    Best

    Art

     

    https://www.medpagetoday.com/infectiousdisease/covid19/86149

     

     

    Art Papier MD

    Associate Professor of Dermatology and Medical Informatics

    University of Rochester College of Medicine

     

    CEO

    VisualDx

     



  • 2.  RE: COVID and cognitive bias

    Posted 04-26-2020 11:16
    Multiple PEs are a hallmark of Covid.

    So maybe not so premature closure; maybe not base-rate neglect, either!

    tom benzoni





  • 3.  RE: COVID and cognitive bias

    Posted 04-26-2020 11:31

    Tom,

    Yes coagulopathy is common.  In this case the patient was COVID negative. But your point speaks to the complexity of the current situation!

    Best

    Art

     






  • 4.  RE: COVID and cognitive bias

    Posted 04-26-2020 19:17

    I did some quick calculations as to what the PPV and NPV of covid pcr tests would be if I varied the sens from 70 to 90% while holding the spec fixed at either 95 or 99% at different community prevalences (ranging from 1% to 25% as currently seen throughout the USA

    the results are below

    Dr Brush (or anyone else who is interested), could you help me calculate how these test characteristics would be affected if two different pcr tests were run on any given pt in these scenarios???   as some medical teams in my region are now starting to perform a rapid pcr POC test followed by a "long cycle" PCR test even though we dont know what the accuracies of either of these tests are ...    :(

    I dont know how to calculate the PPV and NPV of two consecutive tests but I imagine its a little complicated lol ,,,

    Thanks; stay safe everyone!!!



    Inline image






    Tom

    Thomas Westover MD
    Assoc Professor Maternal Fetal Medicine
    Cooper Medical School
    Camden NJ












  • 5.  RE: COVID and cognitive bias

    Posted 04-30-2020 14:53
    Tom, 

    Some writers would say that doing sequential tests (supposing that you know the Sens/Spec, or at least that and the Confidence Intervals. - not a safe assumption yet, I don't think) is "simply" to replace the "base rate" for the second test with the results of the first. 

    Quickly going from one scenario to the next:
         Say you used as test 1 in the upper table on a patient (90/95), who has fever, dyspnea and anosmia, which made you suspect a 15% likelihood.  The post test probability after a positive test would have gone up to 76%, which you would substitute for prevalence in the second test.

    If you did a second test, say the lower right in second table (70/99) you would start with a 76% pre test likelihood, any positivity would make the post test likelihood VERY high.  If the second test was negative, on the other hand, it would reduce the likelihood to around 50%. 

    Hope this helps.  Given a good enough spread sheet you could do all the calculations using the CIs which would likely be more helpful.

    Best,
    Ted
    --
    Edward B, J. Winslow, MD, MBA
    Home 847 256-2475; Mobile 847 508-1442
    edbjwinslow@gmail.com
    winslowmedical.com
    "...the more you know about the past, the better you are prepared for the future."
    Theodore Roosevelt, 26th President of United States of America






  • 6.  RE: COVID and cognitive bias

    Posted 04-30-2020 15:55

    Sequential testing is not going to be the solution to improving testing for COVID-19. I was thinking of this earlier in the week when the White House came out with a couple powerpoints on national COVID-19 testing, and one of the powerpoints had a figure suggesting that sequential antibody testing could improve predictive values. I was cringing as I read it.

     

    In lab testing in general, most of the limitations of sensitivity and specificity aren't due to random error, but rather to inherent chemical and biological properties of the patient and test.  With the COVID-19 PCR test, most of the sensitivity issues appear to be that swabs just don't pick up all that much virus from the surface of the nasopharynx. RT-PCR is great at amplifying RNA, but it has to have something to start with. If the PCR test is, say, 80% sensitive in symptomatic patients, running two different PCR tests at the same time on that same patient (regardless of whether it's the same sample) is unlikely to improve sensitivity more than running a single one.  You don't magically pick up 80% of the initially missed 20% (i.e. you don't bring sensitivity up to 96%) just because you do a second test.  The only time repeating *might* be helpful would be if the patient's viral load (again, the load on the surface of the nasopharynx) happens to be right at the limit of detection, where random effects could make a difference in the result.

     

    Same story with serology testing.  The main cause of false negatives is going to be sampling too soon after infection, and repeating the test with a different manufacturer's kit isn't going to do anything useful.

     

    False-positives aren't as much of a problem for either PCR or serology (unless you're talking about the rapid lateral flow devices which are mostly crap), but again, sequential testing isn't going to be particularly useful here.

     

    When does sequential testing make sense? Mainly when there's a fast and inexpensive test that you want to use most of the time, and thereby only have to rely on the expensive/slow (but more accurate) confirmatory test later. Sequential testing does *not* make sense unless the confirmatory test is more accurate, and even then there has to be a practical reason why you don't just start with the more accurate test to begin with.  

     

    --Brian Jackson

     






  • 7.  RE: COVID and cognitive bias

    Posted 05-01-2020 06:57


    Thanks for both your thoughts;

    I agree that running the same (or a different) pcr test immed after a neg test doesnt make sense unless the sens of the first test is dramatically lower than the second assay for some technical reason and/or costs are dramatically different... BUT repeating a negative amplification test result at some later point in time COULD be useful as you describe in the setting of sampling "too early" in the time course...

    I would suggest/agree that we have four separate issues to consider ; sampling adequacy, analytical accuracy, time course evolution issues (timing of sample collection) and cost

    classically we teach that, if you have "good and accurate" antigen and ab tests, you sample both the antigen and baseline ab during the early course of the disease and then repeat both (or maybe just the ab test) later in the time course; the antigen rises initially without any ab present, the antigen then peaks and declines followed by IGM (or IGA) rise and decline and then IGG rise and plateau (eg primary HSV, CMV, parvo etc)

    if you have inadequate sampling (wrong swab, swab got dried out or heated or wrong transport media, poor collection technique etc)  then that is a problem that is fixable by performing a repeat antigen test at a later point in time (but you never know in real life if the technique was poor or not)

    if you have antigen sampling "too early" (which you also never know in real life); then that is also fixed by repeat antigen sampling at a later point in time

    if the test itself is inaccurate for an analytical reason then repeat antigen sampling wont help as you describe, you then need serial observation of sero-conversion to establish the dx (assuming the classical model of seroconversion is relevant and the ab tests are accurate without excess cross reactivity)

    if we had a cheap test that was highly sens we would accept that as a reasonable gold standard even if it carried a mod high false pos rate...

    this is why we traditionally obtain antigen sample and baseline ab at the time of sx onset and then repeat the ab titer at a later point in time to evaluate for seroconversion

    the catch in this setting (at least in my mind) is that we dont know EITHER the comparative sens/spec of the different commercial antigen(pcr) assays (especially comparing rapid point of care isothermal pcr tests versus the workhorse pcr thermal cyclers) OR the comparative sens/spec (or cross reactivity) of the different ab assays at different points in time... so we are somewhat blind here

    it seems to me that serial sampling may be useful depending on the clinical circumstances (and the clinicians desire to obtain truth as compared to their ability to handle uncertainty)

    I hope my reasoning makes sense..
     
    Respectfully

    Tom Westover MD







  • 8.  RE: COVID and cognitive bias

    Posted 04-30-2020 17:01

    I didn't address the last part of the original question. If you're looking at a short-cycle PCR test (with presumed inferior sensitivity) followed by a second long-cycle PCR, then the main issue you're addressing is indeed sensitivity. The second test is essentially replacing the first test in an effort to try to catch the low-positives. So the effective sensitivity of the 2-test algorithm essentially equals the sensitivity of the second test.

     

    It's harder to comment on specificity, but that really is less of a problem if both tests are well-designed and well quality-controlled. The RT-PCR assay we're running on the Hologic platform (and I think the Roche is the same) detects two different sequences that are believed to be quite specific to this virus. If it's a poorly-designed test or a poorly-managed lab, then you could potentially get contamination issues (ref: Dartmouth pertussis pseudo-epidemic from 20 years ago) but that's different from a random effect.

     

    --Brian