By ANISH KOKA, MD

Click here for part 1 “A Hole in the Heart, Part I” If you have NOT read part I, we highly recommend that you go there now and then come back and read the continuation. – The Editors 

Analysis of the Randomized Control Trials

The 3 randomized trials performed tested two closure devices – CLOSURE I tested the STARFLEX device (NMT Medical, Boston, MA). The PC and RESPECT trial tested the AMPLATZER device (St. Jude Medical, St. Paul, MN).  Let’s look at each study closely.

CLOSURE I (Starflex device)

This study looked at patients between 18 and 60 years of age who had a prior stroke or a transient ischemic attack within the prior 6 months. This was a resoundingly negative study.  At two years, the 12/447 patients in the closure group suffered a recurrent stroke, while 13/462 patients in the medical therapy suffered a recurrent stroke.  The trial, however, reveals one of the important chinks in the armor of the randomized control trial (RCT).  Randomized control trials are only as good as the patients they enroll.  Enroll the wrong patients, and the results don’t tell you much.  CLOSURE-1 didn’t provide closure because they included patients with transient ischemic attacks (TIAs).

Transient ischemic attacks are no joking matter – they refer to a transient occlusion of a brain blood vessel that causes stroke like symptoms that only last a few minutes and cause no permanent damage. The problem with TIA’s is that they are incredibly subjective.  For instance, I recently saw a patient who presented with tingling over the left side of her face.  She had no other neurologic findings, but the neurology team noted that this may ‘possibly’ be a TIA.  A subsequent MRI showed no objective evidence of a stroke, but in the end no one can clearly say whether she actually suffered a cerebral ischemic event or  not.

This represents a clinical challenge that I thankfully – as a cardiologist – want no part of.  I can say from my observations that neurologists do frequently err on the side of caution and when called to evaluate patients in the emergency department, will lean towards admitting, observing and imaging these patients.  Recall that 25% of the human population has a PFO.  How many patients in CLOSURE who had a transient symptoms that were not actually neurologic ended up being closed?  We will never know.  The other major problem with the trial was that there is no clear definition of cryptogenic stroke.  This was again a clinical decision left to the discretion of the investigators.  Alarmingly, alternative explanations for a TIA or stroke became apparent in 20 of 23 patients after they had a closure device placed! Investigators, for whatever reason, in this trial, did a bad job of enrolling patients with true cryptogenic strokes.  How, then could one extrapolate this data to patient with cryptogenic strokes?  Finally, the medical therapy arm could, at the investigators discretion, receive aspirin or coumadin.  Since the pathophysiology of strokes in this patient population involves thrombus formation, it is difficult to fathom how a closure device would be superior to therapy with anticoagulant such as coumadin that is used to treat and prevent blood clots.

PC (Amplatzer Device)

Similar to the CLOSURE trial, this study identified patients less than 60 years old with a PFO who had a stroke or TIA.  Peculiarly, the endpoints of the study included death, recurrent stroke or TIA.  I say peculiarly because the trial only had 200 patients in each arm – mortality was a very high bar to cross.  Only 6 patients had strokes in the entire trial – 1 in the PFO closure arm, and 5 in the medical therapy arm.  As you may be able to surmise, this trial was markedly underpowered to say much of anything, and still suffered from the cardinal sin of CLOSURE – enrolling patients with TIA’s, and even using TIA’s as an outcome measure.

RESPECT (Amplatzer Device)

The RESPECT trial was by far the best of the three trials.  ~500 patients in each arm were enrolled after 8 painstaking years (twice as long as expected).  Happily TIA was NOT one of the criteria to be enrolled and also was not an outcome measure.  All patients considered for closure had to have a neurologic deficit present for more than 24 hours or a neuroradiologically confirmed stroke on CT or MRI.  As usual, patients with other explanations for stroke were excluded, but also excluded were lacunar strokes – this is a type of stroke thought to be related to disease in small vessels.  Essentially, excluding TIAs as well as lacunar strokes meant that RESPECT had the most rigorous entry criteria of the three trials by far.

Under these conditions – 9 of the patients assigned to a closure device had a stroke, while 16 patients assigned to medical therapy had a stroke.  This difference was even greater when accounting for the fact that 3 of the patients assigned to a closure device had a stroke prior to getting their procedure.  The final analysis used what is known as intention to treat – strokes  were attributed based on the treatment group that was assigned, not based on the treatment that was ultimately received.  This is a commonly used technique to avoid the bias and erroneous conclusion that may arise if a large number of patients assigned to a certain treatment drop out prior to the end of the trial.  In this case, however, using intention to treat means ascribing strokes that happened before the patient received a device to the device.  This makes no sense.

The other important point buried within RESPECT was an analysis of stroke risk by quantification of the size of the PFO.  All PFO’s are not born equal.  Visualizing the size of a PFO directly can be challenging, but a good surrogate is to observe the number of bubbles that cross from the right side of the heart to the left.  The clip below (Video 3) shows a small number of bubbles that make their appearance in the left heart.(Grade 1 shunt)

Video 3.  Small PFO

Compare this to the clip below (Video 4) which shows bubbles fill the left side of the heart.

Video 4. Large PFO

Clearly, while the above 2 patients both have a PFO, one is much different from the other. In the RESPECT trial,  10/231 patients that had a substantial shunt and were randomized to medical therapy suffered a recurrent stroke compared to only 2/241 patients who had a closure device placed.  Despite the small numbers, this was a statistically significant difference.

This important subgroup analysis was ignored by the AAN panel.  I know why.   Drawing conclusions from subgroup analyses is a risky business.

Subgroup Analyses

The need to break large groups of patients into smaller subsets is an understandable one. Unlike machines, every human body is different in ways we have yet to completely unravel.  This technically speaking describes heterogeneity of response that just means that within a selected population, the individual response to therapy may vary widely.  For instance,  2% of a population tested may have  a genetic profile that results in complete remission with treatment of a cancer drug.  Comparing the ‘average’ response tells us little about these ‘super’ responders, and consequently makes subset analysis attractive. In many cases though, and this is what data purists focus on – subset analyses are unhelpful.  By nature, the groups within subsets are small, and are thus underpowered to find a difference.  So a negative finding when analyzing a subgroup may simply mean there weren’t enough patients to find a difference – a false negative.  On the other hand, the probability of finding at least one statistically significant interaction within subgroups is directly proportional to the number of subgroups tested  To put it simply, do enough subgroup analyses and you may end up finding that the best day to close PFO’s is on a Tuesday.

Figure 4. False positives and subset analyses

Ideally, authors pre-specify subgroups they plan to analyze, so it doesn’t appear that they indulged in mining the data posthoc to find silver linings within a negative trial.  While I would have pre-specified shunt size as a subgroup to analyze, it does not appear readily obvious from reading the manuscript that the trial investigators did.

Regardless, interpreting data from subsets requires expertise.  A neurologist may ignore the data on shunts, but to a cardiologist who has seen clots pass across a PFO, and seen bubbles pour across this same opening during a bubble study to fill the left side of the heart – the shunt size data (Figure 5) is difficult to ignore.

Figure 5. Subset analysis – RESPECT Trial

Summarizing the evidence 

My conclusions from the review of the data are somewhat different than the American Association of Neurology.

• Most PFO’s should not be closed
• An exhaustive search to rule out other causes of stroke is imperative prior to consideration of PFO closure
• Patients with a true cryptogenic strokes with a large PFO likely reduce their risk of recurrent stroke with PFO closure.
• Antiplatelet therapy alone is probably inferior to anticoagulant therapy in patients with large PFOs
• The AMPLATZER device is the only device with demonstrated efficacy in reducing strokes
• The AMPLATZER device was associated with complications I would deem worse than having a recurrent stroke in 6 patients (1.2%)

All of these trials were hampered by very slow enrollment. This was in large part related to the reluctance of cardiologists to enroll their patients in these trials.  Before you pass judgement on those anti science cardiologists, realize that physicians can only randomize those patients that they feel therapeutic benefit is uncertain.  While we will never know the characteristics of patients who were not enrolled, and critics have speculated nefarious conflicts of interest predominate, I can say that the accumulated data would make it difficult for me to enroll any of these patients into a trial.  I wouldn’t run the risk of giving the patient with a small PFO a closure device and I think it would be unethical to take a patient with a large PFO and not offer closure or at least more intensive anticoagulation.

This does not mean that the randomized control data was useless.  Quite the contrary, it provided important data to confirm that the large majority of patients with strokes who have PFO’s should NOT be offered closure.  Of the ~2800 patients with strokes I have been personally involved with in the last 6 years – I have recommended closure fewer than 10 times.

The American Association of Neurology panel unfortunately came to a non-conclusion (no evidence of benefit) based on a relatively superficial analysis of the data.  In doing so, they follow the recent trend in the analysis of medical trials that relies exclusively on the derivation of statistically significant data from randomized control trials.  It is an approach that I now recognize as a tool to make non-experts experts in fields they have no background or training in.  The randomized control trial and the almighty p value allows the health care economist, or health care journalist who has never treated a patient with a stroke to tweet with great certainty that there is no evidence to suggest PFO closures are indicated in any patients with strokes. In this brave new world, nuance has no place, and the expertise required to interpret data is irrelevant, and may actually be a handicap if you happen to have used this expertise to endorse a particular drug or device.

I do recognize the fallacy of eminence based medicine that has in the past lead us to use leeches to bleed patients of their ill humors.  We do need data, and frequently we need evidence that randomized control trials provide.  Yet I see great harm in completely removing clinical judgement, experience, and intuition when trying to interpret data. Subset analyses should be often ignored, but this does not mean they should always be ignored.  An expert with knowledge of the history and context of the particular topic is invaluable in sorting signal from noise to make a clinical decision about the 33 y/o mother of two with a large PFO who was lucky enough to survive her first stroke with few deficits, and is anxious to know if there is anything that will reduce the risk of another stroke.  I know what the p-value, intention-to-treat, disdainer of subsets would say.  For her sake, I hope she’ll hear my side of the story as well.

Anish Koka is a cardiologist who unfortunately sees a lot of patients with strokes.  He has no financial conflicts of interest related to PFO closure devices.  He can be reached on Twitter @anish_koka