Research That is Positive When It Is Negative

This weeks post is the first in a series touching on some of the challenges when you start researching technology for the prehospital setting (or anywhere really). Dr Andrew Weatherall (@AndyDW_) on why some monitors aren’t the monitors you’re sold. 

I am new to the research game. As is often the case, that brings with it plenty of zeal and some very rapid learning. When we first started talking about the project that’s now my PhD, we set out wondering if we could show something that was both a bit new and a positive thing to add to patient clinical care.

It didn’t take long to realise we’d still be doing something worthwhile if the project didn’t work one little bit.

Yep, if this thing doesn’t work, that would still be fine.

 

Simple Questions

I’m going to assume no one knows anything about this project (seems the most realistic possibility). It’s a project about brains and lights and monitors.

It came out of two separate areas of work. One of these was the prehospital bit of my practice. All too often I’d be at an accident scene, with an unconscious patient and irritated by the big fuzzy mess at the middle of the clinical puzzle.

“How’s the brain?”

Not “how are the peripheral readings of saturation and blood pressure against the normative bell curve?” Not “how are the gross clinical neurological signs that will change mostly when things get really grim?”

“How’s the brain?”

At the same time at the kids’ hospital where I do most of my anaesthesia we were introducing near-infrared spectroscopy tissue oximetry to monitor the brain, particularly in cardiac surgery cases.

The story sounded good. A noninvasive monitor, not relying on pulsatile flow, that provides a measure of oxygen levels in the tissue where you place the probe (referred to as regional oxygen saturation, or tissue saturation or some other variant and turned in to the ideal number on a scale between 0 and 100) and which reacts quickly to any changes. You can test it out by putting a tourniquet on your arm and watching the magic oxygen number dive while you inflate it.

Except of course it’s not really as simple as that. If you ask a rep trying to sell one of these non-invasive reflectance spectroscopy (NIRS) devices, they’ll dazzle you with all sorts of things that are a bit true. They’re more accurate now. They use more wavelengths now. Lower numbers in the brain are associated with things on scans.

But it’s still not that simple. Maybe if I expand on why that is, it will be clearer why I say I would be OK with showing it doesn’t work. And along the way, there’s a few things that are pertinent when considering the claims of any new monitoring systems.

 

A Bit About Tech

Back in 1977, a researcher by the name of Franz Jöbsis described a technique where you could shine light through brain tissue, look at the light that made it out the other side and figure out stuff about the levels of oxygen and metabolism happening deep in that brain tissue. This was the start of tissue spectroscopy.

Now, it’s 38 years later and this technology isn’t standard. We’re still trying to figure out what the hell to do with it. That might just be the first clue that it’s a bit complicated.

Of course the marketing will mention it’s taken a while to figure it out. Sometimes they’ll refer to the clinical monitors of the 1990’s and early 2000’s and mention it’s got better just recently. They don’t really give you the full breadth of all the challenges they’ve dealt with along the way. So why not look at just a few?

  1. Humans Aren’t Much Like Cats

Jöbsis originally tested his technique on cats. And while you might find it hard to convince cat lovers, the brain of a cat isn’t that close to a human’s, at least in size. (As an aside, I’m told by clever bionic eye researchers the cat visual cortex actually has lots of similarities with that of humans – not sure that explains why the aquarium is strangely mesmerising though).

He also described it as a technique where you shone the light all the way across the head and picked up the transmitted light on the other side. But even the most absent-minded of us has quite a bit more cortex to get through than our feline friends and you’d never pick up anything trying that in anything but a neonate.

So the solution in humans has been to send out near-infrared light and then detect the amount that returns to a detector at the skin, on the same side of the head as you initially shone those photons.

When you get handed a brochure by a rep for one of these devices, they’ll show a magical beam of light heading out into the tissues and tracing a graceful arc through the tissues and returning to be picked up. You are given to believe it’s an orderly process, and that every bit of lost light intensity has been absorbed by helpful chromophores. In that case that would be oxy- and deoxyhaemoglobin, cytochromes in the cell and pesky old melanin if you get too much hair in the way.

See? Here's the pretty version that comes with the monitor we're using in the study? [It's the Nonin EQUANOX and we bought it outright.]
See? Here’s the pretty version that comes with the monitor we’re using in the study? [It’s the Nonin EQUANOX and we bought it outright.]
Except that’s the version of the picture where they’ve put Vaseline on the lens. Each one of those photons bounces eratically all over the place. It’s more like a small flying insect with the bug equivalent of ADHD bouncing around the room and eventually finding its way back to the window it flew in.

So when you try to perform the underlying calculations for what that reduction in light intensity you detect means, you need to come up with a very particular means of trying to allow for all that extra distance the photons travel. Then you need to average the different paths of all those photons not just the one photon. Then you need to allow for all the scattering that means some of the light will never come back your way.

That’s some of those decades of development chewed up right there.

  1. Everyone Looks the Same But They Are Different

So that explains the delay then. Well there’s another thing that might make it hard to apply the technology in the prehospital environment. Every machine is different. Yep. If you go between systems, it’s might just be that you’re not comparing apples with apples.

That particular challenge of calculating the distance the light travels? Every manufacturer pretty much has a different method for doing it. And they won’t tell you how they do it (with the notable exception of the team that makes the NIRO device who have their algorithms as open access – and their device weighs 6 kg and is as elegant to carry as a grumpy baby walrus).

So when you read a paper describing the findings with any one device, you can’t be 100% sure it will match another device. This is some of the reason that each company calls their version of the magic oxygen number something slightly different from their competitor (regional saturation, tissue oxygenation index, absolute tissue oxygen saturation just to name a few). It might be similar, but it’s hard to be sure.

Maybe that's harsh. Could a walrus be anything but elegant? [via Allan Hopkins on flickr under CC 2.0]
Maybe that’s harsh. Could a walrus be anything but elegant? [via Allan Hopkins  on flickr without mods under CC 2.0]
  1. When “Absolute” Absolutely Isn’t Absolute

You get your magic number (I’m going to keep calling it regional saturation for simplicity) and it’s somewhere between 60 and 75% in the normal person. The thing is it hasn’t been directly correlated with a gold standard real world measurement that correlates with the same area sampled.

The NIRS oximeter makes assumptions about the proportions of arterial, venous and capillary blood in the tissue that’s there. The regional saturations are validated against an approximation via other measures, like jugular venous saturation or direct tissue oximetry.

On top of that all those “absolute NIRS monitors” that give you a definite number that means something? No. “Absolute” is not a thing.

It’s true the monitors have got much better in responding to changes quickly. And they’ve added more wavelengths and are based on more testing so they are more accurate than monitors from decades past. But they can still have significant variation in their readings (anywhere up to 10% is described).

And they spit out a number, regional saturation, that is actually an attempt to take lots of parameters and spit out a number a clinician can use. How many parameters? Check the photo.

This is from an excellent review by Elwell and Cooper. [Elwell CE, Cooper CE. Making light work: illuminating the future of biomedical optics. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2011;369:4358-4379.]
This is from an excellent review by Elwell and Cooper. [Elwell CE, Cooper CE. Making light work: illuminating the future of biomedical optics. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2011;369:4358-4379.]
  1. The Practical bits

And after all that, we reach the practical issues. Will sunlight contaminate the sample? Can it cope with movement? Do you need a baseline measurement first? Does it matter that we can only really sample the forehead in our setting?

All the joy of uncertainty before you even try to start to research.

 

So why bother?

Well the quick answer is that it might be better for patients for clinicians to actually know what is happening to the tissue in the brain. And acknowledging challenges doesn’t mean that it isn’t worth seeing if it’s still useful despite the compromises you have to make to take the basic spectroscopy technique to the clinical environment.

But even if we find it just doesn’t tell us useful stuff, we could at least provide some real world information to counter the glossy advertising brochure.

There are already people saying things like,

“You can pick up haematomas.” (In a device that just tells you if there’s a difference between the two hemispheres.)

“Low regional saturations are associated with worse outcomes.” (But that’s probably been demonstrated more in particular surgical settings and the monitoring hasn’t been shown to improve patient outcomes yet.)

“You can even pick up cytochromes.” (In the research setting in a specially set-up system that are way more accurate than any clinical devices.)

All of those statements are a bit true, but not quite the whole story. The other message I extract from all of this is that all this uncertainty in the detail behind the monitor can’t be unique to NIRS oximetry. I have little doubt it’s similar for most of the newer modalities being pushed by companies. Peripherally derived Hb measurements from your pulse oximeter sound familiar?

After all this it’s still true that if we can study NIRS oximetry in the environment that matters to us we might get an exciting new answer. Or we might not. And sometimes,

“Yeah … nah.”

Is still an answer that’s pretty useful.

 

 

This is the first in a series. The next time around, I’ll chat about the things we’re trying in the design of the study to overcome some of these challenges.

If you made it this far and want to read a bit more about the NIRS project, you can check out the blog I set up ages ago that’s more related to that (though it frequently diverts to other stuff). It’s here

 

Working with Standards that are Forgetful – Australian NSQHS Standards and Retrieval Medicine

In times where external standards are increasingly applied to health services, where does retrieval medicine fit in? Dr Alan Garner shares his insights after wrestling with the Australian National Safety and Quality Health Service Standards process. 

In Australia, national reform processes for health services began in the years following the 2007 election. Many of the proposed funding reforms did not survive negotiation with the States/Territories but other aspects went on to become part of the Health landscape in Australia.

Components which made it through were things like a national registration framework for health professionals. Although the intent of this was to stop dodgy practitioners moving between jurisdictions, the result for an organisation like CareFlight was that we did not have to organise registration for our doctors and nurses in 2, 3 or even more jurisdictions as they moved across bases all over the country. Other components that made it through were the national 4 hour emergency department targets although I think the jury is still out on whether this was a good thing or not.

NSQHS copy

Other Survivors

Another major component to survive was the National Safety and Quality Health Service Standards. The idea is that all public and private hospitals, day surgical centres and even some dental practices must gain accreditation with these new standards by 2016. The standards cover 10 areas:

  • Governance for Safety and Quality in Health Service Organisations
  • Partnering with Consumers
  • Preventing and Controlling Healthcare Associated Infections
  • Medication Safety
  • Patient Identification and Procedure Matching
  • Clinical Handover
  • Blood and Blood Products
  • Preventing and Managing Pressure Injuries
  • Recognising and Responding to Clinical Deterioration in Acute Health Care
  • Preventing Falls and Harm from Falls

Are these the right areas? Many of the themes were chosen because there is evidence that harm is widespread and interventions can make a real difference. A good example is hand washing. Lots of data says this is done badly and lots of data says that doing it badly results in real patient harm. This is a major theme of Standard 3: preventing and controlling healthcare associated infections.

Here is a visual metaphor for the next segue [via www.worldette.com]
Here is a visual metaphor for the next segue [via http://www.worldette.com]

What about those of us who bridge all sorts of health services?

So what about retrieval? We are often operating as the link between very different areas of the health system. And we pride ourselves on measuring up to the highest level of care within that broader system. So do these apply to us? Did they even think about all the places in between?

Well, whether these Standards will indeed be applied to retrieval and transport services remains unclear as retrieval services are not mentioned in any of the documentation. CareFlight took the proactive stance of gaining accreditation anyway so that we are participating in the same process and held to the same standards as the rest of the health system.

So when we approached the accrediting agency, this is what they said: “Well, I guess the closest set of standards is the day surgical centre standards.” We took it as a starting point.

Applying Other Standards More Sensibly

This resulted in 264 individual items with which we had to comply across the ten Standards. And we had to comply with all standards to gain accreditation – it is all or nothing. However as we worked through the standards with the accrediting body it became clear that some items were just not going to apply in the retrieval context.

A good example is the process for recognising deteriorating patients and escalating care that is contained in Standard 9. There are obvious difficulties for a retrieval organisation with this item as the reason we have been called is due to recognition of a patient being in the wrong place for the care they need. This is part of the process of escalating care. It would be like trying to apply this item to a hospital MET team – it doesn’t really make sense.

With some discussion we were able to gain exemptions from 40 items but that still left us with 224 with which to comply. Fortunately our quality manager is an absolute machine or I don’t think we would have made it through the process. There’s take away message number one: find an obsessive-compulsive quality manager.

It took months of work leading up to our inspection in December 2014 and granting of our accreditation in early 2015. Indeed I am pleased to say that we received a couple of “met with merits” in the governance section for our work developing a system of Carebundles derived from best available evidence for a number of diagnosis groups (and yes I’ve flagged a completely different post).

So yes or no?

Was the process worth it? I think independent verification is always worthwhile. As a non-government organisation I think that we have to be better than government provided services just to be perceived as equivalent. This is not particularly rational but nevertheless true. NGOs are sometimes assumed to be less rigorous but there are plenty of stories of issues with quality care (and associated cover-ups) within government services to say those groups shouldn’t be assumed to be better (think Staffordshire NHS Trust in the UK or Bundaberg closer to home)

As an NGO however we don’t even have a profit motive to usurp patient care as our primary focus. The problem with NGOs tends rather to be trying to do too much with too little because we are so focused on service delivery. External verification is a good reality check for us to ensure we are not spreading our resources too thinly, and the quality of the services we provide is high. The NSQHS allow us to do this in a general sense but they are not retrieval specific.

Is there another option for retrieval services?

Are there any external agencies specifically accrediting retrieval organisations in Australia? The Aeromedical Society of Australasia is currently developing standards but they are not yet complete.

Internationally there are two main players: The Commission for Accreditation of Medical Transport Systems (CAMTS) from North America and the European Aeromedical Institute (EURAMI). Late last year we were also re-accredited against the EURAMI standards. They are now up to version 4 which can be found here. We chose to go with the European organisation as we do a lot of work for European based assistance companies in this part of the world and EURAMI is an external standard that they recognised. For our recent accreditation EURAMI sent out an Emergency Physician who is originally from Germany and who has more than 20 years retrieval experience. He spent a couple of days going through our systems and documentation with the result that we were re-accredited for adult and paediatric critical care transport for another three years. We remain the only organisation in Australasia to have either CAMTS or EURAMI accreditation.

For me personally this is some comfort that I am not deluding myself. Group think is a well-documented phenomenon. Groups operating without external oversight can develop some bizarre practices over time. They talk up evidence that supports their point of view even if it is flimsy and low level (confirmation bias) whilst discounting anything that would disprove their pet theories. External accreditation at least compares us against a set of measures on which there is consensus of opinion that the measure matters.

What would be particularly encouraging is if national accreditation bodies didn’t need reminding that retrieval services are already providing a crucial link in high quality care within the health system. There are good organisations all over the place delivering first rate care.

Maybe that’s the problem. Retrievals across Australia, including all those remote spots, is done really well. Maybe the NSQHS needed more smoke to alert them.

For that reason alone, it was worth reminding them we’re here.