Comparing and ranking many groups

kanan

New Member
#1
I could use some advice on what techniques would be best for analyzing my data.

I work in computer vision. Typically in this field people make statements like, "The mean performance of my algorithm performs .5% better on dataset x compared to the mean performance of so and so's algorithm, so it is better." I'd like to be more rigorous...

I'm trying to compare 45 different algorithms with each algorithm being applied in two different frameworks. I run each algorithm+framework on a particular image dataset n times (typically n = 10, but I could run them more times if needed) using different training and test subsets each time. I do this for m datasets.

I want to determine which method performs best across the frameworks and datasets. I've been asked to rank them from best to worst. I could use advice on what kind of analysis would be appropriate in this sort of situation.

I don't know much about ranking things in statistics, so suggestions on what to read up on would be helpful.

I assume I could do something like an anova for the methods applied in a particular framework to a single dataset, but I'm not sure how to combine the data across frameworks and datasets. Simply concatenating the data across the frameworks and datasets doesn't seem correct since the mean and variance across these things is variable.

Any advice is much appreciated.