Business Challenge

A pharmaceutical company about to begin an FDA Phase 3 trial approached Urvin AI with a business challenge they faced. They had Phase 2 trial results from a handful of patients for a new medication. Ordinarily, the FDA accepts statistical analysis of such results to demonstrate efficacy and safety in order to move to a Phase 3 trial. However, this time the FDA wanted more. The medication was for the eye, and the FDA wanted to know exactly which parts of the eye would be improved. While pharmaceutical companies tend to be very good at data science and descriptive statistics, they can often struggle with prediction and forecasting. They needed to find a partner who could make predictions on a small, very noisy dataset with confidence. Standard statistical techniques were not up to the challenge, but Urvin understood how to leverage our experience with artificial intelligence to help them.

Urvin’s Approach

Urvin’s AI team took a look at the dataset, and quickly realized that the low amount of data and extreme level of noise was not well suited for a naive approach with standard artificial intelligence techniques. When presented with such datasets, blindly applying a technique such as deep learning would likely show fantastic in-sample results due to curve fitting, and would not provide robust or generalizable predictions. We were also acutely aware of the problem of information leakage or data snooping bias.

Urvin produced a research pipeline to ensure procedural rigor and a repeatable approach. The major stages in such a pipeline are:

• Universe: the set of subjects and data on which the algorithm operates.
• Targets: the types of predictions or decisions that the algorithm generates.
• Features: the set of information inputs that the algorithm uses to generate the targets.
• Models: the techniques, architectures, and parameters that comprise the algorithm.
• Metrics: the criteria on which the performance of the algorithm is judged.
• Testing & Validation: the procedures used to evaluate the performance and generalizability of the algorithm.

The first problem Urvin had to solve was to ensure that model calibrations would not take into account information that the model couldn’t know before the outcome was reached (the Target for prediction, which the model is being trained to forecast). In practice, this is easier said than done. Very often, information about the target values in the test set can find its way into the training set. We say that “information has leaked” when a predictor or decision maker has access to information that would not have been available at the time a simulated prediction or decision is supposed to be made. This happens in many subtle ways as data is selected, excluded, and transformed throughout the research pipeline.

One of the ways to prevent this is by separating training data from test data. Often a k-fold approach is used, however in certain circumstances (such as when you have multiple datapoints for a single subject) k-fold can leak information if individual datapoints are not independent. Instead, Urvin created a custom cross-validation procedure that ensured data was partitioned by subject. We then explored multiple linear and non-linear models.

The small sample size, high noise, and lack of independent features created challenges for the nonlinear models in the training, testing, and leave-one-subject-out cross-validation workflow. Performance could vary wildly depending on which subset of subjects the models were trained on, and there was little convergence between model training using the same subset of subjects.

While non-linear models were not useful in general, Urvin was able to leverage deep learning to engineer a unique feature for the linear model, which substantially improved model performance.

The Result

Urvin not only came up with an approach that worked, we did so quickly as the company gave us less than a month to develop this approach before the first patients were scheduled to come in to start the trial. We developed a novel model that was extremely effective, wrote a 21 page paper explaining it in detail, and created an executable for the company to use throughout the drug trial. The client’s reaction speaks volumes:

We went to Urvin with a request for a predictive solution that required a deep understanding of some torturously complicated data.  We needed it within a month.  Over a major holiday period.  I have to admit that there wasn’t any hope of a simple method to solve our problem, within the timeframe or not.

What we got was an astonishingly elegant solution.  On time, no less.

I can go even further, and say that this was a very pleasant collaboration as well.

Thank you Urvin.  We are true believers now.