Theranos Presentation Mostly a Dud, Say Scientists

Forget "Edison." Let's talk "miniLab."

That was the message Elizabeth Holmes sent at her highly anticipated presentation Monday at the American Association for Clinical Chemistry's annual meeting in Philadelphia.

Speaking for roughly 55 minutes in front of an estimated 2,650 people -- most of them laboratory scientists -- Holmes introduced a new device that can fit on a desktop. The machine, called a miniLab, integrates several blood-testing methodologies and, using robots and the company's proprietary "nanotainer" collection tube, will be able to process small volumes of blood remotely and send data back to a centralized location.

Holmes presented internal company data that showed the system worked for a range of tests, including a new assay for Zika.

Dr. Jerry Yeo, however, director of clinical chemistry laboratories at the University of Chicago, seemed to speak for a number of scientists when he summed up his response this way:

"There was nothing revolutionary or disruptive behind the technology of the miniLab," Yeo said.

Yeo said the miniLab uses spectrophotometry and other measurement principles that are "well known in the field and are existing, current technologies employed by many diagnostic vendors."

While the internal performance data Holmes showed "looked impressive," he said, "ultimately the proof will require that these units be tested in independent clinical labs, so for now the jury is out, until we see some peer-reviewed publications from those sources."

Yeo also said the data Holmes provided were incomplete in terms of fully assessing the machine.

"In retrospect it was no different from going to a typical talk by a sponsored vendor," Yeo said.

Dr. Stephen R. Master, one of the event's moderators, said the presentation might have gone better had Theranos not previously made so many grandiose claims, most notably that it could perform dozens of tests from just a few drops of blood.

"When push came to shove, what they actually showed  was far, far less," Master said. "It wasn’t 'Give me a couple of drops of blood and I’ll do everything. It was 'I'll  do a panel of four tests here, or six or eight. Can they do more? They didn't show it."

Master said that while many of his colleagues thought including different pieces of lab equipment in a box was an interesting feat of engineering, "It's not as if there's something here we haven't fundamentally seen before."

He contrasted Theranos' presentation with one he saw the night before, at a plenary session. The presenter "also had a box with a bunch of different modules, but he had over 200 peer-reviewed papers."

In response to the criticism, Theranos Scientific and Medical Advisory Board member Susan Evans said Holmes demonstrated that the architecture of Theranos' technologies, as well as the associated science and technology, are sound.

"Innovations like this are very exciting," Evans said. "The Theranos miniLab has the potential to increase access and bring laboratory testing to the patient, whether close to home or in remote locations that are currently underserved."

(This response, by the way, was sent through public relations colossus Hill+Knowlton Strategies -- now, apparently, working with Theranos.)

Many if not most of the audience were expecting Holmes to present data concerning the technology, known as "Edison," behind the problematic consumer blood tests  the company ran over the past few years. As part of an attempt to correct problems flagged by the Centers for Medicare and Medicaid Services, Theranos was forced to invalidate tens of thousands of these tests, some of which are now the subject of lawsuits, including one that claims incorrect test results led to a heart attack.

But nowhere in her presentation did Holmes present validating data for Edison. Nor did she address problems at the company's California lab that have led to her two-year suspension from owning or operating a lab, and to severe penalties for Theranos.

"We know there’s a lot of questions about the past,” Holmes said, "and in the appropriate forum, we’ll address those."

She said the technology known as "Edison," the subject of so much controversy over the past year, is obsolete.

After the presentation, Holmes answered questions from the audience for about 35 minutes.

Also in the audience was Dr. David Koch, director of clinical chemistry at Grady Memorial Hospital in Atlanta and a past president of the AACC. About the miniLab, he said, "I was not that impressed with the technology. I think it’s apparently still coming along, but at least she shared the technology they have.”

Koch said colleagues noted the absence of any data concerning Theranos' famous claim to be able to perform multiple tests from just a few drops of blood from a fingerprick.

But, he said, "I'm glad she came. It's a start."

Dr. Greg Miller, professor of pathology at Virginia Commonwealth University, said he thought Holmes did a "credible job."

"She explained the technology they’ve developed. She made it clear it’s the newest version of the technology and that it’s gone through several generations.”

While Miller wouldn't call the data presented "a full validation, of the method or the technology," he said "it showed credibility for the example she showed, and it shows potential for it being a useful piece of technology."

Consensus in the Media: A Failure

No on has taken a poll of the couple of thousand lab experts in the audience, but the reaction that has made its way through to media covering Theranos, as well as the media's characterization of the presentation itself, mostly ranges from tepid to extremely negative.

Nick Stockton of Wired expressed a common theme:

The miniLab itself still has to get validated by an independent lab. And even that wouldn’t give Theranos the clean slate it desires. Not from this crowd. Remember, this company got famous claiming faster, cheaper, painless, and above all revolutionary tests. Now it wants that same level of attention to pitch a mini lab that features no radically new technology.

From Jen Christensen, at CNN:

(T)he broader feeling seemed to be that there was not enough there. Some of the crowd lingered after the session to discuss the ideas with their peers.

"There's just not enough data here," said Bobby Reddy, who runs a lab-related tech startup in Urbana, Illinois.

"The failing here is to show up without enough data to show that the technology works. It might be enough for a poster session, but not this," said Joshua Hayden at Weill Cornell Medicine, who works with one of the moderators at the conference session. "This industry is hungry for new ideas, we want disruptive technology in this field. But this doesn't seem to be it so far, and most of us don't like all the smoke and mirrors they've used over the years."

And Matthew Herper of Forbes:

"Everything we’re interested in, they’re not talking about,” says Daniel Holmes (no relation), head of clinical chemistry at St. Paul’s Hospital in British Columbia, who reviewed the presentation for FORBES. “This is essentially a prototype instrument. They’re doing initial studies of a prototype.”

Geoffrey Baird, an associate professor of medicine at the University of Washington who has been critical of Theranos, went further, calling the presentation “a bait and switch.” He says: “We’re not seeing Theranos technology. We’re seeing an early-stage technology, an idea.”

As far as it goes, the data do seem to show a promising prototype. The data do show some agreement between the miniLab and existing devices used by Siemens. The question will be how convincing outside experts find these results.

And Christina Farr at Fast Company:

"Clearly, the scientific and medical community was hoping for a data-driven discussion today, and instead got a new product announcement," says John Torous, a psychiatrist and clinical informatics fellow at Harvard Medical School. ...

(S)cientists are questioning whether the MiniLab technology is a breakthrough. The current market is already fairly saturated: Abbott's iStat system, for instance, is a handheld device for clinicians to test patients for a plethora of common tests. Roche just received FDA clearance for its Cobas device, which can test for ailments like the flu and some strep infections in under 20 minutes. And Theranos competitors Quest and Labcorp already operate versions of this type of equipment in their own labs.

"I can't imagine why they're wasting their time," says MIT-trained material scientist and biotech entrepreneur Kaveh Milaninia by phone. "It's a tough market."

And (Theranos' Public Enemy No. 1) John Carreyrou of The Wall Street Journal:

The data covered 11 blood tests, but the miniLab can’t perform all of those tests on a single 160-microliter blood sample, according to a person familiar with the matter. At most, the miniLab can perform only three or four tests on a sample of that size, so a patient ordering all 11 tests would need to have his finger pricked three to four times, the person said….

Some lab experts at the conference said the underlying lab techniques relied on by Theranos’s new device, such as spectrophotometry, have been in use for years. The length of time needed to process samples and produce results, ranging from 20 minutes to two hours, also doesn’t improve on what is commercially available, some lab experts said.

And Beth Mole of Ars Technica:

Yesterday’s presentation could have been the data-driven turning point Holmes has been promising. And the conference attendees—experts in clinical testing—gathered anxiously to hear the results.

But instead of validation studies of its existing technology, they got, essentially, a sales pitch for future technology.

Holmes spent her entire 45-minute time slot talking about a new blood testing device called the miniLab. Any questions about the company's old, disastrous blood testing device (the Edison) were flatly refused—even including questions about the differences between the Edison and miniLab.

Some in the audience, or those who were following online, expressed their dissatisfaction with the presentation on Twitter:

Previous post

Hey, try this archetypal nightmare out for size ...

Youʼre in a room. Speaking. There are people there -- hundreds and hundreds of people. They are not particularly friendly. Actually, they are hostile. You have displeased them. You had made certain unsubstantiated claims about something they know a great deal about. Yes, they are experts. They know exactly how to determine what is truth. And what is ... 

Then you wake up. Covered in sweat ...

This is pretty much the scenario Theranos CEO and founder Elizabeth Holmes is going to face on Monday in Philadelphia, at the American Association for Clinical Chemistry's 68th Annual Scientific Meeting & Clinical Lab Expo.

Holmes will present, for the first time and in front of hundreds of lab experts, “reproducibility and correlation data” for the blood-testing technology her company claims to have developed but has never proven to the satisfaction of the scientific community. Holmes will also “share data to demonstrate the precision and accuracy” of its tests.

Those tests can ostensibly be performed using only a small amount of capillary blood from a patient's finger, instead of the traditional tubes-full taken from a vein.

But a Wall Street Journal investigation characterized those claims as dubious. Later, a Theranos lab in California was found by its regulator to be so deficient as to represent a threat to patients' lives. And this month, regulators suspended the lab’s license and banned Holmes from owning, operating or directing a blood-testing lab for two years. (The company has until early September to appeal the government’s decision, and the suspensions don’t take place for the appeal period.)

Both the U.S. Attorneyʼs Office for the Northern District of California and the SEC are investigating, and lawsuits aplenty have been filed.

Now Theranos is typically described with adjectives ranging from “troubled” to “beleaguered” to “disgraced.” And the company, once valued at $9 billion, is in serious danger of becoming permanent shorthand for greed-driven Silicon Valley hubris.

If Holmes has any chance of rebuilding her shattered reputation, and if Theranos is to have any chance of clawing its way back to respectability, it will start in Philadelphia, on Monday. The company has appointed a scientific and medical advisory board, in part to help with "publication and presentation in scientific journals and at scientific meetings." The board includes several former AACC officials, so one would assume Holmes will, as they say, come correct.

90 Minutes

Still, whatever she presents may or may not be enough to convert hardcore skeptics. The presentation will last 45 minutes, followed by a 45-minute question-and-answer period, according to AACC members helping with the session. Anywhere from 800 to 1,500 people are expected to be in attendance, and the presentation is open to anyone who registers for the conference, including the media, though no questions from reporters will be taken.

Three moderators will ask their own questions as well as questions from the audience, which will be forwarded to the moderators after three other experts come to a consensus as what is most scientifically relevant to the discussion.

The AACC had extended an invitation to Holmes for several years before she agreed to come speak. The announcement of this year's talk came soon after the Centers for Medicare & Medicaid Services threatened unprecedented sanctions against Theranos, as well as news of the U.S. Attorneyʼs and SEC's investigations.I asked Dr. Patti Jones, who made the decision to invite Holmes this year, if she had any reservations after all that news shook out.

“Of course,” said Jones, who is the clinical director of the chemistry and metabolic disease labs at Children’s Medical Center in Dallas. “I reconsidered every step along the way. But the bottom line is, I really think there is an opportunity to really see data here, to see the science. AACCʼs members have the expertise to interpret what she tells us. To be able to look at that data and say, 'Okay,' or 'Not okay.' "

Jones said she had heard from members both opposed and in support of the invitation, and that the opposition was “much more
vocal.” She said no one from the AACC leadership will see Holmesʼ presentation in advance.

"Weʼre not endorsing her in any way," Jones said. "Weʼre just providing a room to show us what she has.”

I asked her if Holmes had any chance, given recent history, of convincing her colleagues she's legit.

“I think she can,” Jones said. “I think if she presents us with scientific data that shows not only that her system -- both the collection and the analysis work -- but how well they work in comparison to other methods, then she could feasibly convince people in the room that she has what she has been saying she has all along.”

I asked five other experts in laboratory medicine, most of whom will be in attendance Monday, what Holmes has to do to turn things around. Here are edited excerpts from those conversations:

Dr. David Koch, director of clinical chemistry, Grady Memorial Hospital, Atlanta; former president AACC

"She really needs to talk about how she and Theranos didn't understand clinical laboratories and how best to practice laboratory medicine. She needs to acknowledge that they've made mistakes and that she has, in fact, made a turn from past practices and behaviors. She needs to say one of those mistakes was not sharing data, not having it vetted by professionals in the field, and not coming to meetings such as the AACC Annual Meeting.

The reason why a lot of us are skeptical, and perhaps even disgusted with the whole thing, is that they had a very cavalier attitude about the data they were producing, much of which turns out to apparently not be correct.

The laboratory has advanced year by year for the whole history of our profession, and if this is one more advancement, fine. But we want to see the data and even evaluate the instrument or device in our laboratories. Or in physicians offices, drugstores, or even in patients' homes.

I expect some hard questions, especially if her presentation is a little lacking, weak on substance.

Dr. Eleftherios Diamandis, division head of clinical biochemistry, Mount Sinai Hospital, Toronto; "Theranos phenomenon: Promises and fallacies"

I think the AACC is in a very odd situation, to have a lecture by somebody who was caught manipulating quality assurance, delivering results that some people believe have endangered patients.

I think her reputation is in big question here. Irrespective of the opinion of people like me, that she shouldn't be invited in the first place, apparently she's going to show up and give the lecture, although I will wait until I see her before I believe that.

Their technology is still a black box, which nobody knows what the details are. They still haven't published a single word. Anything that she presents would not be peer-reviewed. It will be something that experts in the field have not seen or had a chance to really criticize.

That will be a major hurdle, because she can come and tell us anything she wants. She's got the magic black box. You put in one little drop of blood and you get 1,000 analyses with impossible accuracy,

But how am I going to be able to verify that the claims are actually correct? I will only believe it if experts evaluate this technology and find it to be robust.

Dr. Fred Apple, medical director of clinical laboratories, Hennepin County Medical Center, Minnesota

Iʼm hoping we're going to see, ʻHereʼs my technology, hereʼs my process, how we generate data, and here is the R&D weʼve done that proves our technology works.'

We want to look at clinical sensitivity, limits of detection, reproducibility, analytical correlation against another assay, turnaround time, stability of samples, as well as how it compares to whatʼs already out there in the literature.

Show me some data and a statistically relevant number of patients, so we can do some comparisons.

Dr. Jerry Yeo, director, clinical chemistry laboratories, Department of Pathology, University of Chicago

At this point, Holmes has a very high bar to clear to gain
credibility, given her history of secrecy, lack of peer-reviewed publications and most damaging, the abject failures with compliance with basic requirements of the CLIA (Clinical Labroatory Improvement Amendments) guidelines leading to CMS banning her for 2 years.

Even if she suddenly revealed “revolutionary performance data” of her mysterious Edison invention, she still has to explain how that system managed to put out two years of bad data that had to be voided, corrected. Or how the system failed to comply with CLIA
regulations.

I seriously doubt she can do all that at this AACC venue. If it turns out to be a highly staged presentation with sanitized, prescreened questions, it might produce an opposite effect of loss of credibility to
both Holmes and AACC leadership.

Dr. Norman Paradis, professor of medicine, Dartmouth; "Why Medical Lab Experts Were Always Skeptical of Theranos"

If she puts up real data, there's got to be some credibility associated with that. But this has never really happened before.

Big pharma gets in trouble every so often in big ways, but this just doesn't tend to happen in diagnostics: People know how to validate assays. That's not a new thing. You could take your tech and just go to a completely independent university-based group to validate it. You can do it internally, by bringing in what's called a clinical research organization.

The FDA has published a voluminous document on good clinical practices. Why it's taken so long is just not clear.

So I do think if they show real data, that will be something. If she's managed to get data in a stealth manner, get data that is independent, then that obviously would be a much more reassuring thing.

But they still need to get FDA approval. And this won't move them along at all on that front.

Their situation is so unique that they really just need to turn this machine over to a completely independent person.

This is absolutely an amazing story. I've never seen anything like it.