Pitching R in Pharma

[rogojel]

hi,
I will pitch R to a pharma company on Monday. Their biggest fear is that they can not validate R, so they can not use it in any situation related to regulatory, like proving something statistically to the FDA , because they can not prove, that the packages they used in R are correct.

Did anyone have this type of argument about R ? It is not my first time BTW that I see this argument and commercial stat sw developers seem to use it as a strong selling point. Any answers to this ?

thanks and regatds

 

[JesperHP]

I have similarly been told about this argument but have never heard of any reply. And personally if I think about the point does seem valid, without knowing anything about the FDA requirements.

 

[Dason]

It's a little old but seems like a decent starting point for investigating this https://www.r-project.org/conferences/useR-2007/program/presentations/soukup.pdf

 

[hlsmith]

I would wonder about packages getting updated and interdependencies between packages. But I guess you could cite exactly what you used and versions. I read some (sorry no link on the gradual use of r with fda).

 

[Jake]

In what sense exactly does the pharma company think that they can "validate" the results of, say, SAS?

If the company believes that there is some kind of legal guarantee that the results of SAS are accurate, then they clearly have not read the user license -- no such guarantee exists. In fact, there are NUMEROUS cases where SAS results have been shown to be in error. And these errors typically take way longer to get corrected than errors found in R.

If the company feels that the results of SAS are functionally "guaranteed" because so many statisticians use SAS, they should consider the fact that statisticians actually overwhelmingly use R and not SAS, so in this sense the results of R are far more "guaranteed" than those of SAS.

 

[hlsmith]

Comment:

Many fda studies should be well designed with a solid study design. Given this, complex analyses are hardly used. Most analyses are bread and butter in complexity.

 

[rogojel]

In what sense exactly does the pharma company think that they can "validate" the results of, say, SAS?

If the company believes that there is some kind of legal guarantee that the results of SAS are accurate, then they clearly have not read the user license -- no such guarantee exists. In fact, there are NUMEROUS cases where SAS results have been shown to be in error. And these errors typically take way longer to get corrected than errors found in R.

If the company feels that the results of SAS are functionally "guaranteed" because so many statisticians use SAS, they should consider the fact that statisticians actually overwhelmingly use R and not SAS, so in this sense the results of R are far more "guaranteed" than those of SAS.

Thanks Jake, this was my feeling too, but I completely missed the idea to refer to the user license.

 

[hlsmith]

Jake, i will have to pull the ol' scholar card. Do you have any citations for an example where there had to be an erratum bases on SAS software or any other documentation or is this hearsay?

 

[GretaGarbo]

hi,
I will pitch R to a pharma company on Monday. Their biggest fear is that they can not validate R,

I thought that discussion had ended (a few years ago).

Maybe this blog post can help ("R, drug development and the FDA").

Note the links in there about "validation" where the FDA person says:

"The Food and Drug Administration does not endorse or
require use of any specific software including those software
mentioned in the presentation."

 

[rogojel]

Nope,
sales-guys and reps of stat sw companies are notoriously hard to convince. Also local statisticians who have worked for 20 years with STATA (or whatever)

 

[Jake]

Jake, i will have to pull the ol' scholar card. Do you have any citations for an example where there had to be an erratum bases on SAS software or any other documentation or is this hearsay?

Poke around the internet and find the release notes accompanying the various versions of SAS, that is, the documents that list all the bug fixes and etc. featured in each new release. Some of these fixes were to bugs that affect the accuracy of statistical results. I couldn't immediately find many of the release notes, and didn't want to spend too much time looking, but I did find this one for example: http://www.cosmos.esa.int/web/xmm-newton/sas-release-notes-1500

 

[spunky]

use the now classic post on R passes SPSS in scholarly use!

tell them that, at the end of the day, any new experimental design of method that ends up being used was initially discovered and validated through R!

 

[GretaGarbo]

Nope,
sales-guys and reps of stat sw companies are notoriously hard to convince. Also local statisticians who have worked for 20 years with STATA (or whatever)

Yes, they are hard to convince. (But don't insult their favorite software. It is worse than insulting their children )

But the point is that it is not forbidden to use R. They can use R also.

And it is free. They can download the software without asking the boss for permission. (Even if the cost is just US$ 10, then they must ask the manager for permission. How about the feeling of freedom? How about the feeling of deciding themselves if they should test the software?)

Have a look at Journal of Statistical software. Most papers are about R packages. Maybe one or two could be of interest for them? By looking at that journal, they can ask: what software will be dominating in the future?)

 

[GretaGarbo]

hi,
I will pitch R to a pharma company on Monday.

Today it would be interesting to hear how the reaktion was at the meeting. Did they accept what you said?
(I believe that we can learn from discussions like this.)

 

[rogojel]

Hi,
actually no - the reaction was that they need to get to know R first and anyway they have adequate tools for validation so this is not even the topic. I showed them the material you suggested - they are not yet ready. Interestingly the attraction of R is that one can do Bayes analysis with it

 

[rogojel]

hi,
there is a nice twist to the story - it turned out thar some of the team have an interest in learning Bayes - and they aee R as a way to get support in form of already made libraries. A good example of the advantages pf open source

 

[RWinters]

Your best chance to do this is to 'promise' that you will only use stable packages (Hmisc, lm, anova etc.) , and NOT update them. Other than some of the reasons you quoted, there are also requirements laid out in FDA regulations having to do with replicatation, ie you have to guarantee that the results will be exactly the same across all machines with slightly different configurations. This even extends to how plots appear on the screen.
This aspect is difficult to achieve even in SAS which has about a 2 year update cycle. If one of your packages break, for even the slightest reason, it will be sent back to the drawing board and delay time to market.
So the bottom line is that it CAN be done, but you need tight version control, and would loose a lot of the flexibility that you enjoy by using R.
The monetary potential for a pharma drug is extremely high. I'm guessing that a few SAS linux boxes would cost you < $10,000 a year to license for a pharma company.
This is worth it in my opininion. Saves a lot of headache.
Of course nothing would prevent you from replicating the SAS results in R. That is always good practice.