Evaluating Ranked lists


I implemented ranking algorithm to rank relevant documents to a given topic. So the algorithm produces ranked list of documents for each topic.
For example: Lets say topic A has 25 relevant documents, our experiment select another 25 random documents and rank these 50 document for the topic. The output of the algorithm looks like,
doc1 0
doc2 1
doc3 1
doc50 0

here the documents are listed in ranked order, 0 and 1 indicate whether that document is in the relevant document list or not. We expect all highly ranked documents to have 1 in second column.

I want to evaluate this algorithm, to say that this ranking is not obtained by chance. I have 100 topics (hence 100 ranked lists). How can I apply chi square test for this problem. I am not sure what is the E (expected outcome) should be for chi square.

Please advice to solve this problem. Thanks in advance.