Daniel Joyce: Hi, my name's Dan Joyce. I'm a urologic oncologist at Vanderbilt University Medical Center. I have the distinct pleasure of being joined today by a good friend and really world leader in financial toxicity in health services, who is a urologic oncologist at Jacksonville at the Mayo Clinic, Tim Lyon. Tim, thanks so much for being here.

Timothy Lyon: Morning, Dan, really appreciate the opportunity to join you.

Daniel Joyce: You had a great publication recently talking about U.S. News and World Report rankings, which I think anybody in healthcare is well aware of and has probably pretty strong feelings about one way or the other, depending on how you're ranked in those rankings.

Specifically, you looked at kind of more of the outpatient procedure metrics that are used in those rankings. Can you tell us a little bit about just why you want to do the study, the rationale for the study just to kind of set things up for us?

Timothy Lyon: Absolutely. Happy to. Great. Well, thanks so much.

As most of us know in the urology community, the U.S. News and World Report rankings are often criticized, but really widely utilized both by patients and often by our hospital marketing teams, depending on where we end up in the rankings.

But we know that there are some considerable methodologic problems with these rankings. We published a few years ago that the primary assessment of the urology subspecialty is based on clinical outcomes, things like 30-day mortality and discharge to home, but only among Medicare inpatients. Meaning these are patients that spent at least two nights in the hospital and had Medicare as their insurance.

And in today's day and age where many of our cases go home the same day or only spend one night in the hospital, that's really a minority of what we do. And we recently published that less than 5% of the cases that were done in our department actually met these criteria. So really, U.S. News is looking at a tiny subset of what we all do every day as urology departments.

Now, they've recognized that as a potential limitation, and to their credit, have begun in 2023 using a new software designed to look at potentially preventable complications following outpatient urologic surgeries to improve the validity of their rankings to contemporary urologic practice.

And they do this by using a software made by Solventum, which is a healthcare arm of the 3M Corporation, that identifies urology procedures using procedural subgroups, or PSGs, which are billing codes. And then once it's identified an outpatient urology procedure, it then looks over the next 30 days to identify things like ER visits or hospital readmissions and assigns a measure of whether there are potentially preventable post-operative complications following these outpatient procedures.
So to our knowledge, no one had really externally validated this. So the objective of our study was to look specifically at this outpatient software and see how good a job it was doing based on the patients that were treated in our hospital.

So to do this, we took patients treated from 2019 to 2023 at one of our four Mayo Clinic practice sites, Minnesota, Arizona, Florida, and the health system in Wisconsin, and we applied the Solventum software to all of our urology cases during this period of time.

We then randomly selected 80 pairs of procedure and complications, 20 from each practice site, and individually reviewed their medical records to answer three key questions about the software's performance.

Number one, were the identified procedures actual urological procedures done by urologists? Number two, was the complication type correct? And number three, was the complication clinically related? So was it reasonably something that occurred after the procedure?

And surprisingly, for the first question, we found that only 53% or a little over half of the procedures that were identified by the software were actually performed by a urologist. And nearly half were done by interventional radiology. They were procedures done on the genitourinary tract, things like nephrostomy tube or suprapubic tube placements, but did not necessarily interact with the urologist during that episode of care.

For the second question, was the complication type correct, we found fairly significant agreement, 78%. And in fact, all cases of disagreement were in cases of urinary colonization that we felt were inaccurately classified as urinary tract infection.

We see these patients all the time, right? They come in after procedure with a catheter, they have nausea or fatigue. Someone checks a urinalysis, they see that it's contaminated and it's coded as a UTI even if they don't end up having any other clinical symptoms of a UTI or a positive culture.

And then lastly, we thought that 80% of these cases agreed on clinical relatedness, meaning they were reasonably a complication related to the procedure. And several of the notable exceptions were things like planned stage procedures. You get a stent and you come back two weeks later to get your stone taken out, we didn't think that was a complication, and things of that nature.

So overall, for these two questions we found 71% agreement on both of these measures, which we thought was pretty good for a first iteration. Now, we're very proud to say that Solventum has really had a good-faith effort in engaging with us, and on the basis of these published data has actually made a change to the definition of their urology procedures.

And they have excluded the three PSGs that we've highlighted here that capture a lot of IR procedures to try to help their classifier be more specific to urology, which we're excited about the impact of.

So in conclusion, we felt there was a reasonable level of agreement for this outpatient complication software, especially as a first iteration. It raises the question whether IR procedures belong in the urology rankings. Personally, I don't think so. Solventum seems to agree, and that's been changed in the outpatient urology measure, and we hope that U.S. News will follow suit for all of their outcome assessments.

Given this performance, we think that outpatient complications could probably play a larger role in these urology rankings going further. Currently, it's only 5% of the rankings, but clearly represents much more than 5% of the clinical care we provide.

And another opportunity for improvement would be to use a more stringent definition of catheter-associated UTI to reduce the false positive rate that's being counted as a complication. Thanks very much, and happy to take any questions.

Daniel Joyce: Thanks so much, Tim. I mean, really, really impressive work that obviously has already made a big impact on these rankings. So I mean, congratulations, a really smart idea and really important work to look at.

And I think say what you will about these rankings, it's an honest effort to do what we all want. We want an ability to compare value of where we're going to get healthcare, especially for patients. And I think these rankings do provide a starting place for patients to do that.

Can you talk a little bit, my understanding of this software is that it's identifying potentially preventable complications, is that correct?

Timothy Lyon: Correct.

Daniel Joyce: How does that work? How does the software do that? To me, it's complex, but can you break it down for us?

Timothy Lyon: Sure. I believe some of that information is business confidential, and they haven't shared with us exactly how they've determined that, which was part of the reason for us to do this study, was to get a sense of how good a job are they doing.

So I don't know a specific answer to your question, but I imagine that they've spent a lot of time looking at potential complications after certain operations, right? Bleeding, urinary tract infection, things of that nature, and excluding things that are very unlikely to be related.

Daniel Joyce: Got it. I think that's interesting too, as artificial intelligence becomes more incorporated in the EHR as well, it seems to be another kind of black box for us of not knowing how are we coming up with this and is it real?

And I think that's a lot of people's grievances with the rankings is they don't really feel like they have a good sense of how a lot of these metrics are happening. So it is super important, like what you've done, to really look at the EHR, see what actually happened, and make your own judgments. And it's promising to see how well it's doing.

Timothy Lyon: Yeah.

Daniel Joyce: I get the criticism of not wanting IR to affect urology value. That makes sense to me. However, on the flip side, if I'm a patient and I'm thinking about where to get my stone treated and I've got a two-centimeter renal pelvis stone, maybe the IR component, it's not just the urologist whose quality I'm looking at.

I am looking at the whole institution, and that's something, kudos to the Mayo Clinic, I think you guys excel more than anybody in creating an environment that's holistic there that's not just specialty-driven. The whole institution really supports itself and provides that confidence in the quality of care that's being received.

So do you think, how would you advise a patient looking at a urology specialty ranking, knowing what you know now about these outpatient procedures, to say, "Yes, your urology care is going to be outstanding based on these rankings, however, there may be some other aspects of your care that you might want to look at"? Well, how would you guide them through that process?

Timothy Lyon: Absolutely. And I think you bring up a really good point, which is I think we should have a discussion in our field and our community about what should be assessed when we measure a urology department. We look at service line, we just look at specific physicians.

There are pros and cons to including IR. Obviously you've noted some of the benefits, which are to get a more comprehensive assessment of a service line. I think some of the drawbacks are there could be some unintentional biasing negatively towards larger departments that have larger IR groups or do more complex procedures.

I think that if a urologist never actually sees the patient and can't salvage a complication, that's a potential drawback, but a meaningful conversation we should have.
To answer your question specifically, what do I tell patients? Probably the best way to get an assessment of quality is to look at multivariable inputs. There are several quality rankings. Each of them has their own drawbacks and their own strengths, and I think looking at places that are consistently in the higher echelon across things like Leapfrog, CMS stars, U.S. News, et cetera, would probably give a more comprehensive assessment of the quality of care in a place.

Daniel Joyce: Yeah, you've just given me a brilliant idea, which I'm sure other people have thought of, but that you could create a tool that sort of brings that together for patients and makes it easy to understand and navigate through, so.

Timothy Lyon: Yeah. I mean, multivariable inputs I think are the cornerstone of much of our clinical decision-making. We should do the same in quality assessment.

Daniel Joyce: Couldn't agree more.

Well, Tim, thank you again. Really, really great work. I'm excited to see kind of where this is headed next.

Timothy Lyon: Appreciate it. Thanks very much.