I will try to propose something, but I am not sure if this would make any sense for your case..

Let A and B be the number of protiens in the two sets. Let assume A>B.

Let C be the number of common proteins you get.

I am trying to define a quantity which will vary from 0 to 1 where 0 indicates no overlap and 1 indicates maximum overlap

Basically, the first fraction gives you the overlap percenage and the second fraction just scales it so that the index is within [0,1]