Statistical analysis of Profits - use of U-test

#1
Hi

We have been working on a project where we have 2 or more trading agents on a market stock and for the outcome we have the profits that each trader generated after running 1000 times or more each market session. We need to statistically compare the profits of this traders to know which one is better, we have the idea of using confidence intervals on the mean, and we also where thinking about using the U-Test but we are not sure if its suitable to use it for such a large samples and comparing more than 2 traders.

Any help will be appreciated.

Thanks
 

gianmarco

TS Contributor
#2
Hi,
may be you should elaborate a little more about your goals.
For example: what do you mean by "comparing" or seeing "which is better"

Do you intend to test if there is a significant difference between your samples?
If you do, I think that given your samples sizes (1000) t-test should be fine.

Please, feel free to post back if you need further hints.

Regards
Gm
 
#3
Hi,
may be you should elaborate a little more about your goals.
For example: what do you mean by "comparing" or seeing "which is better"

Do you intend to test if there is a significant difference between your samples?
If you do, I think that given your samples sizes (1000) t-test should be fine.

Please, feel free to post back if you need further hints.

Regards
Gm
The meanning to see which one is better is to know gains more profits, what we are planing to do is first to compare with the confidence interval of the means, if they don't overlap we assume that one trader is generating more profits per market session than the other but if they do overlap then what test can we do to compare all the profits and know if they are different and truly one contains higher values, meaning that it will generate more profits.

So is the U test not suitable for comparing this?

Thanks!! :)
 

gianmarco

TS Contributor
#4
Hi,
I believe that you could leave aside the comparison of CI for the mean, and go straight to the use of t-test (or 1-way Anova for more than 2 samples).
Better, may be that a relaxed interpretation of the Mann-Whitney test (if you have 2 samples) or Kruskal-Wallis test (for more than two samples) would work for you: in fact, you could conceive these tests as testing the tendency of values from a given sample being greater than the ones of the others.

Please, be advised that if you have more than 2 samples, and if 1-way Anova or KW spot a significant difference between samples, you will need a post-hoc test to spot which sample(s) differ from the others.

Hope this helps
Regards
Gm
 
#5
Hello and thanks for the answer.
I am a little confused though. Are we sure that we can use the anova test? It is true that we have many traders and want to compare their profits and see which one is better. But from what I know, and correct me if I am wrong, isnt the anova test a parameterized test ran among some indepedent and depedent variables? for example, I want to know whether a certain parameter changes the behaviour of the algorithm. So, I would do an anova test if I wanted to check whether the learning rate for example, of the algorithm changes the profits or not. In that case I would state the NULL hypothesis that:
H0: the leanring rate does not affect the profits of the algorithm
H1: the leanring rate does affect the profits of the algorithm
and then running the anova test I would check whether H0 is valid or not.

BUT in our case, we want to compare 2 different algorithms and say that we are 95% sure that the algotithm A does better than B, not a parameter within an algorithm.
How can I test that?
Sorry If I am making you dizzy, and thanks for the reply
 

gianmarco

TS Contributor
#6
Hi,
may be I did not get well your problem.
I woule stick with the issue you described in your 1st post.
I understand that you have two or more traders, and you want to see what trader is "better" as far as the profits they generated are concerned.
So, it is my understanding that each trader makes up a sample, and each sample is made up of the values of the profits generated.
So, I guessed that you may want to test if there is a significant difference in profit between samples (ie, traders).
If all this holds true, you can use:
T-test (for 2 samples) or 1-way anova (more than 2 samples)
Or
Mann-Whitney (2 samples) or Kruskal-Wallis (more than 2)

Hope this makes sanse to you
Gm
 
#7
Thanks,
Yes, you understood the problem correctly.
But I am still a little confused.
Wiki says : In statistics, the Kruskal–Wallis one-way analysis of variance by ranks (named after William Kruskal and W. Allen Wallis) is a non-parametric method for testing whether samples originate from the same distribution. It is used for comparing more than two samples that are independent, or not related. The parametric equivalent of the Kruskal-Wallis test is the one-way analysis of variance (ANOVA).

From this I get that they are two different tests corresponding to different experiments.
I am confused of which test to use from the 2 as I do not recognize in which category my case belongs.
Or does it belongs to both categories and I can do whatever test I want?
 

gianmarco

TS Contributor
#8
Hi,
when one wish to test if there is a significant difference between two samples, he/she can use t-test (parametric) or M-W test (non-parametric). In case of more than 2 samples, 1-way Anova (parametric) or KW (non-parametric).
Now, the choice between t-test and 1-way Anova, on the one hand, and M-W and K-W, on the other hand, lies on whether or not your data meet the assumptions of parametric tests. I assume that you should know the difference between parametric vs non-parametric tests.

So, provided the fact that I do not know your data, the only guess I can do is that given the sample size (1000 for each sample), you could use t-test (or 1-way ANOVA in case of more than 2 samples).
t-test (or 1-way Anova) will test if there is a significant difference in mean value between samples.

Should you data not meet the assumptions for parametric tests, you should switch to the non-parametric "version", like MW or (in case of more than 2 samples) KW. As for these tests, I refer you to this earlier post.

So, in essence, the choice between parametric vs non-parametric depends on the features of your data:
1) inspect your data
2) see if they meet the assumptions for t-test (or 1-way Anova)
3) if they do, go on with parametric tests
4) if they do not, switch to non-parametric tests (MW or KW)

hope this helps
regards
Gm