**Summary of study:**In a single results section, different ANOVAs are run on the different metrics – raw scores (such as RT, d prime, accuracy) and also collapsed/composite/index scores (for example combining RT and d prime, or normalised scores). Then these behavioural measures are correlated with survey data measures, and finally multiple regression analysis is run with behavioural and survey data.

**Definitions:**

**Genuine outliers**= participants with below chance-level accuracy, d-prime scores >3SDs below condition mean, participants not following instructions/technical issues

**Outlying values**= data points simply >3SDs away from condition mean

**Option 1: Remove genuine outliers at the start, but note outlying values in analyses on a per analysis basis**

Remove genuine outliers from whole dataset

Run each analysis for RT, d prime, normalised scores, correlations etc.

In each case (separately for each analysis), note if there are outliers (e.g. > 3SDs from condition means).

If data normally distributed:

Run the analysis with the outliers in.

Run the analysis with the outliers removed.

If results and assumptions not affected

Write up with outliers in – to keep the same N across analyses (and note outliers and the above analyses)

If data not normally distributed:

Remove outliers

if data now normally distributed:

Run analysis with outliers removed

Run non-parametric analysis with outliers included

If results are the same:

Write up non-parametric test with outliers in (keeps same N for all analyses) (note all above analyses)

If data not normally distributed:

Remove outliers

if data still not normally distributed

Run non-parametric analysis with outliers included

If results are the same:

Write up non-parametric test with outliers in (keeps same N for all analyses) (note all above analyses)

If a participant did not complete a survey, however, run the correlational analyses and regression with this smaller N

**Option 2: Remove all outliers and outlying values at the start**

Remove genuine outliers from whole dataset

Then check all raw scores, collapsed score/indices, and survey measures for outliers (> 3 SDs from condition means)

Remove all these outliers, including participants that did not also complete the survey measures (i.e. losing their behavioural data)

Run each of the analyses with this N

If there are new outliers and/or data not normally distributed

Run parametric tests only

My concern with Option 2 is that there are no checks to determine whether these outlying values (kept in or removed) affect the findings. Also participants with no outlying values in raw scores are removed for having outlying values in collapsed/composite scores from all analyses. Plus behavioural data is removed due to missing survey data from purely behavioural analyses.

However, to remove all these outlying values (participants) at the start, and then to test whether the removal of each one affects results for each of the separate analyses and also in combination requires a huge number of analyses!

Would be grateful for any suggestions or improvements to Options 1 or 2.

Thank you!