Study design
Suppose I want to compare algorithm runtime performance (measured in seconds) of different algorithms for the same problem type. There are several problem instances for this problem type on which the algorithms are tested and performance is measured.
Possible tests
As far as I know, the following is true (please correct me if not!):
Variant
Now each instance has a property "size", which is a non-negative integer. Let us assume that for each size in {10,20,30}, there are 10 instances of this size (30 instances in total). In this case, I have no idea which tests could be used for:
Suppose I want to compare algorithm runtime performance (measured in seconds) of different algorithms for the same problem type. There are several problem instances for this problem type on which the algorithms are tested and performance is measured.
Possible tests
As far as I know, the following is true (please correct me if not!):
- We want to compare 2 algorithms on 30 independent test instances: In this case, one could use a unpaired Student's t-test or Mann–Whitney U test.
- We want to compare 3 or more algorithms on 30 independent test instances: I know that pairwise comparison with Student's t-test should not be done, so one could use ANOVA or Friedman / Quade tests to test for H0: µ1 = µ2 = µ3. But how can one test for "the fastest" algorithm in case H0 has been rejected?
Variant
Now each instance has a property "size", which is a non-negative integer. Let us assume that for each size in {10,20,30}, there are 10 instances of this size (30 instances in total). In this case, I have no idea which tests could be used for:
- The comparison of 2 algorithms on 30 test instances, 10 instances for each size in {10,20,30}.
- The comparison of 3 or more algorithms on 30 test instances, 10 instances for each size in {10,20,30}.