Can someone please advise the appropriate test?

#1
Hi all,

I have 10 subjects, these are products we sell.
I have 80 days of data (amount sold on that day) for each subject.
40 days of the data are before and 40 days are after a change we made in our store.

The difference between the amount we sell from each product is very big, and there are also many outliers in the daily volumes, my data is not normally distributed at all.
I would like to investigate if there is a significant difference between the amount sold before and after the change.
Should I calculate the average amount for each product for the first 40 and the second 40 days and compare them or just compare the full list of daily volumes before and after?

With which test could I examine if the change had en effect on the amount sold?
 

Karabiner

TS Contributor
#2
Personally, I woul sum up the amounts across 40 days (pre and post, respectively), and compare them using Wilcoxon signed rank test.

With kind regards

Karabiner
 

obh

Active Member
#3
Hi Hifiloge,

Even if the data doesn't distribute normally when the sample size is 30 or more usually you can assume the average distribute normally. (CLT)
I didn't see your data but as the nature of sales I can assume you shouldn't remove the outliers.

If you suspect the change influence all product in the same way then you can combine all the data and run paired t-test to compare the averages before and after the changes.

Otherwise, you may check the option of multiple regression while using the product as IV.

You should also try to think if there are other DVs that influence sales?
Is there a dependency between the sales in day N and day n-1?
 
#4
Personally, I woul sum up the amounts across 40 days (pre and post, respectively), and compare them using Wilcoxon signed rank test.

With kind regards

Karabiner
Thank you for your answer. Do you mean summing up the pre and post amount for each product separately then comparing the 10 pre value with the 10 post value? Also wouldn't average or median be better when i have large outliers in my data?
If I use the wilcoxon test wouldn't the fact that one product has for example 20 times the amount of sales than an other one effect my results?
So lets say that product x1 has 10.000 sales while product x2 only has 500 in the pre period. Then in the post period x1 has 15.000 and x2 has 1.500. So the rank test would rank x1 higher because the amount of change is larger but actually x1 only got 1.5 times higher while x2 got 3 times higher. Or is this not an issue?
 
Last edited:
#5
Hi Hifiloge,

Even if the data doesn't distribute normally when the sample size is 30 or more usually you can assume the average distribute normally. (CLT)
I didn't see your data but as the nature of sales I can assume you shouldn't remove the outliers.

If you suspect the change influence all product in the same way then you can combine all the data and run paired t-test to compare the averages before and after the changes.

Otherwise, you may check the option of multiple regression while using the product as IV.

You should also try to think if there are other DVs that influence sales?
Is there a dependency between the sales in day N and day n-1?
If I build a regression model sales should be my dependent variable in my opinion. Then choose the product and other independent variables while using a dummy to signal pre-post period.
I would rather stick to a pair test though.
Do you mean if I take the pre and post 40 days average for each product separately I can assume that the outcome (10 pre and 10 post values) distribute normally? So I would technically compare 10 pairs where each of the pairs represent the average amount sold for the pre and post period for the specific product?
 

Karabiner

TS Contributor
#6
Thank you for your answer. Do you mean summing up the pre and post amount for each product separately then comparing the 10 pre value with the 10 post value?
Yes. If you want a very straight test, unaffected by outliers, very different amounts of sales etc., then you could perform the sign test.

With kind regards

Karabiner
 

obh

Active Member
#7
Hi Karabiner,

Doing Wilcoxon sign rank sounds nice to get rid of all the potential "problems":
Summing all days together reduces the risk of dependencies in the data, as there is the risk that sales in one-day depend on the sales in the day before, but also reduces the sample size.
In this case, you also give equal weight to each product.

If for example 3 products with low sales went done and all the rest up the test won't be significant.
for example before: (2,11,15,30,60,70,80,90,100,200) => after: (1,010,13,31,62,77,81,91,211,600)
 
Last edited:

obh

Active Member
#8
I promise you that I prepared the example before Hili showed the histogram :)

Hili, Can you please do a Histogram with daily data instead? each value is one day sale.
If daily data contained manny zeroes you can treat each value as two days data or more if necessary like each value is 4 days sale. (with no overlap)
 

obh

Active Member
#9
Hi Hifi, Karabiner

Despite the fact that the data looks far from normal if you check the distribution of the average it should be close to the normal

Just for the fan, I run a simulation with Pois(lamda=1) which looks a bit (but not too much) similar to the distribution.
The result looks close to the normal distribution

n <- 400
lambda <- 1
reps <- 1000 # number of simulations
set.seed(1)

x <- numeric(n)
y <- numeric(reps)

for (i in 1:reps)
{
for (j in 1:n)
{
x <- rpois(n, lambda)
}
y <-mean(x)
}
hist(y)
shapiro.test(y)
--------------------------------------
Shapiro-Wilk normality test
data: y
W = 0.99764, p-value = 0.1619

1569988040493.png
 

obh

Active Member
#11
Can you please try the two-sample t-test with a sample size of 400 per each group (before/after)?
one value per day per product?

The potential problem is that there are dependencies between the days.
I would probably try to draw a chart and see if there are such dependencies.
You can also try to run a simple regression of total sales by day (like: Y = b + ax, Y: total sales, X: day 1 to 40, checking that "a" is not significance. do it separately for each group (before/after))

@Karabiner what do you think?
 

Karabiner

TS Contributor
#12
I don't kow how to adjust degrees of freedom, which seems necessary because of the dependency of day-wise observations.

Maybe one could employ some repeatd-measures ANOVA model, with n=10 products, and 2 repeated-measures factors
(one factor observation period, pre vs. post, the other factor being "day of observation", or maybe one uses aggregated
weekly or monthly sales).

With kind regards

Karabiner
 
#13
Can you please try the two-sample t-test with a sample size of 400 per each group (before/after)?
one value per day per product?

The potential problem is that there are dependencies between the days.
I would probably try to draw a chart and see if there are such dependencies.
You can also try to run a simple regression of total sales by day (like: Y = b + ax, Y: total sales, X: day 1 to 40, checking that "a" is not significance. do it separately for each group (before/after))

@Karabiner what do you think?
Using all 800 data entries, grouping them by pre, post and running two sample t test gives p value 0.09.
Using Mann-Whitney on this data gives p value of 0.015.