HYPGEOMDIST applies to two choices without replacement. This is three choices without replacement.

If you are wanting to use Excel, I suggest =COMBIN(n,r). List the possibilities of at least one of each of R and B. There are 10 of them RRRRB, etc RRBOO etc through to RBBBB. Imagine the cards in three piles and you need to find how many RRBOO hands there are. There will be COMBIN(5,2)*COMBIN(10,1)*COMBIN(35,2) = 133875 RRBOO hands. Find the total for all 10 types. This can be done quickly if you have set things out well in a table. The total number of possible hands is COMBIN(50,5). The fraction will give you the probability (between 25% and 30%).