THCB

Categories

Category: Uncategorized

Docs are ROCs: a simple fix for a “methodologically indefensible” practice in medical AI studies

By LUKE OAKDEN-RAYNER

Anyone who has read my blog or tweets before has probably seen that I have issues with some of the common methods used to analyse the performance of medical machine learning models. In particular, the most commonly reported metrics we use (sensitivity, specificity, F1, accuracy and so on) all systematically underestimate human performance in head to head comparisons against AI models.

This makes AI look better than it is, and may be partially responsible for the “implementation gap” that everyone is so concerned about.

I’ve just posted a preprint on arxiv titled “Docs are ROCs: A simple off-the-shelf approach for estimating average human performance in diagnostic studies” which provides what I think is a solid solution to this problem, and I thought I would explain in some detail here.

Disclaimer: not peer reviewed, content subject to change 

A (con)vexing problem

When we compare machine learning models to humans, we have a bit of a problem. Which humans?

In medical tasks, we typically take the doctor who currently does the task (for example, a radiologist identifying cancer on a CT scan) as proxy for the standard of clinical practice. But doctors aren’t a monolithic group who all give the same answers. Inter-reader variability typically ranges from 15% to 50%, depending on the task. Thus, we usually take as many doctors as we can find and then try to summarise their performance (this is called a multi-reader multicase study, MRMC for short).

Since the metrics we care most about in medicine are sensitivity and specificity, many papers have reported the averages of these values. In fact, a recent systematic review showed that over 70% of medical AI studies that compared humans to AI models reported these values. This makes a lot of sense. We want to know how the average doctor performs at the task, so the average performance on these metrics should be great, right?

Continue reading…

Bayer G4A Agents of Change: Watch the Panels, Meet the Co’s that Got Deals

By JESSICA DaMASSA, WTF HEALTH

Bayer G4A, the global life science company’s digital health innovation arm, held their splashy “Agents of Change” event last month to not only introduce their latest cohort of health tech partners, but to also demonstrate the pharma co’s commitment to digital transformation. The entire C-suite of Bayer’s Pharma division became a panel itself — marking the first time the full leadership team of a major pharmaceutical company appeared together to talk strategically about tech’s role in shaping the pharma business model of the future. 

The rest of the program’s agenda teased out G4A’s priorities: consumer health, health disparities, women’s health, and investing. Matthew and I both moderated “star-studded” panels with health tech greats: he tackled health tech investment, ridiculous valuations, and advice for startups with a powerhouse crew of investors, while I led my women’s health panel past the usual talk of period-tracking and into a real push for a paradigm shift in thinking about what actually constitutes women’s health data. Rounding out the program were fascinating discussions about health equity and access led by Indu Sabiaya, and a ‘who’s-interviewing-who-here’ fireside about patient-centered tech with OneDrop’s Jeff Dachis and DiabetesMine founder Amy Tenderich, both entrepreneurs with diabetes who have a lot to say about how most tech misses the mark when it comes to grappling with patient needs in everyday life.

And… if you’re curious about what Bayer G4A actually invested in and who they decided to sign partnership agreements with, check out my exclusive WTF Health interviews featuring these companies by way of the playlist below. 

Spoiler Alert: Not a single digital therapeutic. 

What else could there possibly be for a pharma co to invest in? Watch and see. But, so you know a bit about what you’ll be getting into:

  • Caria is women’s health startup focused on menopause 
  • Sweetch is using just-in-time-interventions linked to mobile data to help “outsmart” chronic diseases
  • ONCARE is a care plan content management platform that lets any healthcare provider upload a care pathway that a patient can then follow via an app on their phone
  • Decipher Biosciences is using genomic testing to disrupt the way prostate cancer is diagnosed and treated 
  • Elly is helping improve the quality of life for cancer patients and those with chronic disease by way of educational and motivational content delivered via voice technology

#Healthin2Point00, Episode 173 | Calm, Story Health, Centivo, Ro & Pear

It’s a wacky world today. Lynne Chou O’Keefe’s Devine Ventures has done 4 deals this week. Calm raises $75m it doesn’t seem to need, Story Health takes on navigation around specialty care, Centivo gets $34m to start a Collective Health rival, Ro moves into the home (literally) bringing diagnostics and Softbank money is back! The lucky recipient of their $80m is DTx company Pear Therapeutics about whom I have almost no opinion… Matthew Holt

CRAZY AMERICA: Health Insurance Covers Testing When You Are Well But Not When You Are Sick

By HANS DUVEFELT

Insurance is the wrong word for what we have here. Our private health insurance system’s prioritization of sometimes frivolous screenings but non-coverage for common illnesses and emergencies is a travesty and an insult to typical American middle class families.

State Medicaid insurance for the underemployed has minimal copays of just a few dollars for doctor visits and medications. From my vantage point as a physician, it is the best insurance a patient can have. They cover almost everything and it is clear to me how to apply for exceptions or follow their step care requirements. I cannot say that about most other insurers.

Most employed people have the kind of commercial health “insurance” that covers an annual physical and certain screening tests at no cost, but requires people to pay the first several thousand dollars of actual sick care expenses out of pocket. This is, in my opinion, insane. It causes delays and omissions in diagnosis and treatment.

A shining example of this bizarre arrangement is the screening colonoscopy. It is free as long as it is normal. If a patient has a polyp removed, which if unchecked could turn cancerous, future health care costs for treating colon cancer are eliminated. But the patient gets billed for the early cure.

Continue reading…

#Healthin2Point00, Episode 172 | MedArrive, PointClickCare, Consejo Sano & FOLX Health

Today on Health in 2 Point 00, we’ve got quite a diverse set of companies to cover. MedArrive, which is Dan Trigub’s company – the former CEO of Uber Health – raises $4.5 million. Jess also asks me about PointClickCare acquiring Collective Medical, connecting data from EMRs in the acute care space into their long term care solution (I interviewed both companies on THCB Spotlights here). Consejo Sano quietly raised $17.1 million for their patient engagement and communication solution for the Hispanic community, and FOLX Health raises $4.4 million to provide care to LGBTQIA+ folks. —Matthew Holt

#Healthin2Point00, Episode 171 | Massive deals & a massive IPO

Today on Health in 2 Point 00, we have a huge IPO coming up – JD Health, a Chinese company which has a valuation of $28 billion, could raise up to $4 billion in its upcoming IPO. On Episode 171, Jess asks me about Everlywell raising $175 million in a Series D for at-home testing, everyone’s favorite keto startup Virta Health raising $65 million bringing its valuation to a billion, Olive Health already putting their funding raise to good use by acquiring Verata Health, and dtx company Click Therapeutics raising $30 million. —Matthew Holt

#Healthin2Point00 Episode 170 | Olive, WithMe, Andor & more

Today on Health in 2 Point 00, Jess & I are together in Marin County before Jess sets off! On Episode 170, Jess asks me about Olive raising $225 million following a recent raise as well, WithMe Health closing a $20 million Series B, Andor Health getting an undisclosed amount in a Series A with an investment from M12, Microsoft’s venture fund, Upfront Healthcare raising $11.5 million in a Series B, and Voluntis – which is a publicly traded DTx company in France – raising $7 million. —Matthew Holt

RWJF Emergency Response Challenges Video

On November 19, 2020 Catalyst @ Health 2.0 hosted the finals of the RWJF Emergency Response Challenges, one for tools for the General Public and the other for the Health System. The promise of the tools that have been built as part of these challenges is immense in the battle against this COVID-19 pandemic and the ones yet to come. The finalists for the General Public challenge were:

Binformed Covidata– A clinically-driven comprehensive desktop + mobile infectious disease, epidemic + pandemic management tool targeting suppression and containment of diseases such as COVID-19. The presenter was veteran health IT expert Rick Peters.

CovidSMS– A text message-based platform providing city-specific information and resources to help low-income communities endure COVID-19. In contrast to Rick, CovidSMS’ team were undergraduates at Johns Hopkins led by Serena Wang

Fresh EBT by Propel– A technology tool for SNAP families to address food insecurity & economic vulnerability in times of crisis – highlighted by Michael Lewis on his Against the Rules podcast about coaching earlier this year. Stacey Taylor, head of partnerships for Propel presented their solutions for those in desperate need.

The finalists for the Health System challenge were:

PathCheck– A non profit just spun out of MIT. It has a raft of volunteers and well known advisors like John Brownstein and John Halamka among many others, and is already working with several states and countries. Pathcheck provides privacy first, free, open source solutions for public health to supplement manual contact tracing, visualize hot spots, and interface with citizen-facing privacy first apps. MIT Professor Ramesh Raskar was the presenter.

Qventus– A patient flow automation solution that applies AI / ML and behavioral science to help health systems create effective capacity, and reduce frontline burnout. Qventus is a great data analytics startup story. It’s raised over $45m and has lots of health system clients, and they have built a suite of new tools to help them with pandemic preparedness. Anthony Moorman, who won the best facial hair of the day award, showed the demo.

Tiatros – A mental health and social support platform that combines clinical expertise, peer communities and scalable technology to advance mental wellbeing and to sustain meaningful behavioral change. They’ve done a lot of work with soldiers with PTSD and as you’ll see entered this challenge to get their tools to another group of extremely stressed professionals–frontline health care workers. CEO Kimberlie Cerrone and COO Seth Norman jointly presented.

We also presented the Catalyst @ Health 2.0 Covid19 SourceDB between the two competitions. Please enjoy the video

Healthcare on the Edge

By KIM BELLARD

Perhaps you read about, or were directly impacted by, the massive, multi-hour Amazon Web Services (AWS) outage last week.  Ironically, AWS’s effort to add capacity triggered the outage, although apparently was not the root cause.  It’s no surprise that AWS sought to add capacity; it, like most cloud service vendors these days, has seen skyrocketing growth.  Even healthcare has jumped into the cloud in a big way.

But, as the outage reminds us, sometimes having core computing functions done in far-off data centers may not be always a great idea.  Still, we’re not about to go back to local mainframes or networked PCs.  The compromise may be edge computing. 

Definitions vary, and the concept is somewhat amorphous, but goal is to move as much computing to the “edge” of networks, primarily to reduce latency.  PwC predicts: “Now, with the rise of IoT, the centralised cloud is moving down and out, and edge computing is set to take on much of the grunt work.” 

As they describe it:

With edge, instead of pushing data to the cloud to be computed, processing is done by devices ‘at the edge’ of your network. The grunt work is done closer to the user, at an edge gateway server and then select or relevant data is sent to the cloud for storage (or back to your devices).

The oft-cited example is self-driving cars; you really don’t want the AI to wait a single millisecond longer than necessary to make a potentially life-saving decision.  An article in Nextgov pointed out:

Thus, a Tesla isn’t just a next-generation car; it’s an edge compute node. But even with Tesla, a relatively straightforward use case, building and deploying the edge node is just the beginning. In order to unlock the full promise of these technologies, an entire paradigm shift is required.

Continue reading…

Stewardship: We Worry More About the Environment than Our Own Bodies

By HANS DUVEFELT

Sooner, rather than later, we will be driving electric cars because of the environment. We use energy efficient light bulbs and recyclable packaging for the same reason. And there is a growing debate about the environmental impact of what kind of food we produce and consume. But I still don’t hear enough about the internal impact on our own bodies when we consider stewardship of natural resources.

Our bodies and our health are the most important resources we have, and yet the focus in our culture seems to be on our external environment.

Just like the consumption culture has ignored its effect on our planet in favor of customer convenience and business profits, it has ignored the effect it has had on the health of the human beings it set out to serve. And just as we now are fearing for the future of our planet, we ought to be more than a little bit concerned about the future of the human race.

But, just as we really can’t expect the corporate world to lead the environmental effort, unless we can engineer a way for them to see profit in doing that, we cannot expect it to lead any kind of effort to make the population healthier. That is something that has to start with the individual.

Continue reading…
assetto corsa mods