Successful Clinical Trial Feasibility and Design
in Near Real Time:
Using AI-enabled Precision Matching Technologies to Ensure Accurate Patient Populations at Cedars-Sinai
In our recent webinar, BJ Rimel, MD, assistant professor at Cedars-Sinai, and Deep 6 AI experts Shabnam Shademan, Ph.D., and Allison Tyler, BSN, RN, OCN, demonstrated how feasibility and study success are facilitated using AI-enabled analysis of structured and unstructured patient data in real time. What follows are the questions we received from attendees, and the answers our experts provided.
What’s your favorite feature of cohort builder? What feature are you using the most?
[Allison] My personal favorite is the filter by document type, as it allows for a specific search that excludes items that don’t fit what I’m looking for. For example, if a user searches for pulmonary embolism (PE), filtering by diagnosis excludes results that mention physical exams.
Of course, favorite features vary from user to user and the reason for the search. For example, if you spend a lot of time doing grant applications or site feasibility questionnaires, you might really enjoy the calendar filter as it allows you to identify numbers of patients that were available within certain time frames.
How long did it take you to learn to use the tool comfortably?
[Dr. Rimel] I think I was pretty comfortable using it within the first couple of months. Though, that was probably only a few hours’ worth of work in total.. if I had a block of time to concentrate on it, I probably could have done it in a day.
We’ve been very lucky my institution has had this tool for so long, so in some ways I’ve “grown up” with it. I still find that I’m constantly learning new tricks. Additionally, I find it continues to increase in functionality. Almost every time I open the tool, I find something new that I’m able to use.
How often do you use the tool?
[Dr. Rimel] I use the tool more around grant building time than on a day-to-day basis. My staff helps me with the ongoing use of the tool for accrual.
But when I’m doing feasibility for sponsored studies and I want to determine upfront which study I want to run, I will design a query using the inclusion and exclusion criteria in the protocol and determine to which of the studies I’m more likely to accrue.. When I’m designing trials or trying to get data together for grants, I rely on the Deep 6 tool enormously for both gathering retrospective data quickly for proposals and doing retrospective chart review studies with an IRB. I also use the tool when I’m assessing feasibility for study design, especially one that’s going to be enrolling in my institution. This is important because our local numbers vary so much from the national numbers, and it’s really exciting to be able to know what’s going to work in my population.
Has Deep 6 considered this to be a tool to provide “aggregate feedback” back to the medical coding teams for enriching their process and/or error counting and quality assurance (QA)?
[Allison] Yes. We always love to hear alternative use cases. While designed to identify patients for clinical trials, there is no reason not to use the platform in any other way that improves workflow, accuracy, or billing by the user or institution. We’ve had feedback that the tool could be useful in these scenarios.
Can you use clinicaltrials.gov for inclusion/exclusion criteria, or do you need to use the protocol?
[Allison] There are multiple methods to utilize eligibility criteria, and you can use whatever works best for you. If you have the NCT code, you can import the eligibility criteria directly as they are from clinicaltrials.gov. You can also copy and paste from another document. Lastly, you could write free text in your desired search criteria.
How portable are queries across different instances? Can you share queries?
[Allison] Sharing queries from one site to another is an upcoming feature — stay tuned!
Can you build a search for any kind of disease?
[Allison] Yes. So, the Deep 6 platform is disease-agnostic. Today’s webinar featured an oncology study, but we have users across the spectrum from population health to cardiology and neurology. You can generate a query around any illness.
Will Trial Recommender also return pending studies when matching medical record numbers (MRNs) to studies? Studies without Institutional Review Board (IRB) numbers?
[Dr. Shademan] I believe this is referring to a study without IRB approval and therefore is pending. If so, we will match a patient to that trial.
When searching for a list of patients who are eligible for a trial, what we are working on now is to make sure that, if a patient is already on a trial, they don’t show up in another search counts.
How many users and/or institutions are currently using the platform?
[Allison] There isn’t a straightforward answer to this question as the number of users is pretty fluid. Some active sites allow anyone that wants an account to have one, which means that site might have several hundred users. Other sites may have a centralized Clinical Trials Office, where there are a small number of “super users” that build out all the institution’s trials, then push those builds to the study teams for use. In another scenario, there might be a controlled rollout, where one department trains a handful of users at a time, but more users and departments are brought onto the platform incrementally.
Deep 6 AI works closely with the institution and will assist in training, workflow development, and user adoption based on the site’s preferred rollout plan.
The number of active sites is also changing as more sites onboard with the system.
What is the population of users of the platform at Cedars? Are they mostly Principal Investigators (PIs) or highly technically competent study coordinators?
[Dr. Rimel] I would say it’s about 50/50. Most people using it for feasibility are PIs. Most of the people using it for workflow and finding patients are probably very tech-savvy. We also have staff members that are helping to maintain the queries as they apply to our workflows.
[Dr. Shademan] One thing to keep in mind is that, in this webinar, we’ve really focused on a particular workflow. But from a high-level perspective, the tool allows for a multitude of people to access it for different types of use cases and information. For example, somebody who is heading research in an institution may not be the person who actually actively builds a query, but they will be able to look at and have access to the overall analytics. Or you could have somebody who is a site monitor who, again, doesn’t build queries, but needs to monitor how many patients are being enrolled and how they’re progressing through the trial. Finally, you have people who actually build the query, which could be the PI or a staff member. The variation of users across the tool is expansive; I can think of about eight or nine different personas who can use the tool for different use cases.
[Allison] I want to mention that you don’t have to be technical or have a technical background to use the tool. My background is in nursing, and I am probably the least technical person that you will ever meet. It’s just a matter of knowing the terms you’re looking for and how they are going to be reflected in the medical records.
Is there a list of medical institutions that have the Deep 6 tool available to them? As a researcher with an FDA approved protocol, it would be nice to be able to approach particular institutions.
[Allison] Yes! We share this information with interested Deep 6 AI platform users. Contact us today to get started.
Did you receive resistance from your security, compliance, or IT office? If so, what was your strategy to overcome the resistance?
[Allison] Our implementation team will work with each organization’s security, compliance, and IT office to get the project approved. This may include security assessments and questionnaires. A common compliance or security-related concern is around the institution’s data leaving the network. To address this concern, we discuss with the institution’s team how previous implementations allowing personal health information (PHI) to leave the hospital networks came about. Then, we can then follow that model with the security team.
Deep 6 AI conforms to the highest standards in healthcare security. Our cloud security framework is built on standards from National Institutes of Standards and Technology (NIST), International Organization for Standardization (ISO), and HIPAA. We are in the process of obtaining a SOC2 Type 2 certification, which is expected to be complete in Q4 2021.
What happens to the data ingested by your clients?
[Dr. Shademan] PHI data is stored behind a secure firewall and devoted to that institution. The data doesn’t leave that institution, and data from multiple institutions aren’t mixed. Our software only accesses the data to respond to queries by sponsors, CROs, and physicians within a protected environment, sharing only site-level information. For example, when a sponsor conducting a feasibility assessment queries the system to see how many patients are potentially available for their trial, the system only shows the patient count meeting the criteria at a specific site, but the PHI is not shared until a registered institutional review board (IRB) protocol number is entered in the system. This allows organizations the ability to maintain strict rules around data accessibility.
When a study is terminated, the IRB protocol number is no longer valid. Since a valid IRB protocol number is required to access patient data from a query, without it, PHI isn’t accessible.
Does it matter what EMR is being used?
[Dr. Shademan] We’re EMR-agnostic, so we have no trouble integrating with any platform.
How often does Deep 6 update from EMRs? How often can a user run a search?
[Dr. Shademan] Currently, EMR data are updated daily. In the future, we plan to provide the ability to do more frequent updates. Queries can be run as often and as many times as the user wants — there is no limit.
How can I improve accuracy and minimize false positives?
[Allison] You can do that a few ways. The most basic, easiest way is to apply the filters. For example, with something like rheumatoid arthritis (RA), you are destined to get results that you’re not looking for, simply because the abbreviation is the same as other terms. We don’t want to take the abbreviation (RA) out of our code trail because that’s how a majority of people are going to document rheumatoid arthritis — they will write “RA” in their notes rather than typing out rheumatoid arthritis every time. So, without affecting the coding and searching, you would apply a filter to “Only show me mentions of this concept in certain data types” to improve the accuracy. So, if I search for a diagnosis of rheumatoid arthritis and omit mentions in social histories or imaging, that will be a lot more accurate. You’re less likely to get mentions of notes like room air or right axilla. So, the filters will be your friends when it comes to creating a more specific search with fewer false positives, just given the nature of human language and how it is translated with natural language processing (NLP) and artificial intelligence (AI). Of course, we will use our experience to help you as you learn.
Are you doing vocabulary normalization as part of your process — like mapping labs to Logical Observation Identifiers Names and Codes (LOINC) and normalizing units?
[Allison] We do map labs and other procedures to LOINC. However, we’re unit agnostic, meaning that users will see results as they appear in the patient’s electronic medical record (EMR). If the hospital you work for uses non-LOINC codes for their labs, then we understandably can’t map to LOINC as it doesn’t exist. In that scenario, the site would provide the mappings for us, and our engineering team will work their magic to make the items searchable. We have experience with facilities that use and do not use LOINC codes, so can work with either situation.
Do you have more confidence in feasibility findings from Deep 6 AI rather than Epic SlicerDicer?
[Dr. Rimel] For my particular studies, where I’m dealing with a lot of different drug combinations, novel therapeutics, and genomic data, I get better results from Deep 6 than SlicerDicer. SlicerDicer can work well for gathering some of the population data like, race, ethnicity, and language, which is fantastic. So, it definitely has a role. For me, though, to determine feasibility, especially for some of the novel therapeutics that I’m looking at with very narrow criteria, I’m getting better feasibility results with Deep 6.
Can you talk a little bit more about the artificial intelligence (AI)/machine learning (ML) algorithms that are actually doing the natural language processing (NLP) search? Are there any chances of bias within these query models that somehow systematically include or exclude certain populations?
[Allison] This requires a lengthier and more technical explanation than we have room for here, so look for a blog post dedicated specifically to our platform’s use of AI/ML!
What is the purchasing model?
[Allison] Deep 6 AI’s platform is purchased as a SaaS subscription as either yearly licensing or on a study by study basis. Please reach out if you would like to find out more.
As discussed in the presentation, through AI-enabled precision matching technology, Deep 6 AI’s patient recruitment tool finds patients in minutes, not months. By analyzing both structured and unstructured patient data, Deep 6 AI can provide sites with live, actionable data to make accurate decisions that facilitate the success of clinical trial feasibility and design.
Many thanks to our speakers, as well as our wonderfully engaged audience!
To learn more, view the on-demand webinar.
About the Speakers
BJ Rimel, MD, Assistant Professor, Cedars-Sinai
Dr. Rimel is an Assistant Professor of Obstetrics and Gynecology at Cedars-Sinai. Her research focuses on clinical trial accrual, recruitment and informed consent. She is also very active in healthcare social media and using digital transactions to simplify cancer care for patients.
Shabnam Shademan, Ph.D., Director of Product Management, Deep 6 AI
Dr. Shademan has extensive experience creating and implementing successful product roadmaps. Shabnam has held roles at tech companies like Snap, Factual, Google, and UCLA’s eye-tracking and psycholinguistics lab. She holds a B.A. and Master’s in Linguistics, as well as a PhD in Psycholinguistics from UCLA. She also holds degrees from Pierce College and Tehran Open University and was a visiting scholar in Linguistics at MIT.
Allison Tyler, Clinical Application Specialist, Deep 6 AI
Allison Tyler, BSN, RN, OCN has been a nurse for over 20 years, working in a variety of therapeutic areas, including the departments of Physical Medicine & Rehabilitation and Neonatal Intensive Care at the Mayo Clinic in Rochester, MN and over a decade as an oncology research coordinator at the Cleveland Clinic in Cleveland, OH. She has spoken at the national level on the topics of prostate and kidney cancer and has published journal articles and co-authored a book chapter on the care of patients with advanced bladder cancer. She is also a crazy dog & horse mom and can’t keep a houseplant alive to save her life.