+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 15 of 16

Thread: Minimum control size for determining effectiveness of treatment

  1. #1
    Points: 946, Level: 16
    Level completed: 46%, Points required for next Level: 54

    Posts
    11
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Minimum control size for determining effectiveness of treatment




    So the following scenario
    • I have a treatment that I know to be effective, but I want to determine precisely how effective it in a (large) group of individuals
    • I'm dividing them in to a test group and a control group so I can compare the effect of the treatment
    • However, I want to keep the test group as large as possible (so as many people as possible get the treatment) and the control as small as possible, while still being able to determine how effective the treatment is with some confidence
    • With other (unrelated) tests in the past, we have arbitrarily used an 80/20 test control split, but we now want to try to make our tests more effective by keeping the control group as small as possible

    What approach can I use to determine this?

    Thanks

  2. #2
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Minimum control size for determining effectiveness of treatment

    Well as you know, the smaller the control group the greater the risk for sampling variability in that group (reflective in its calculated standard error). I would perhaps play around with simulating these data and running tests on the simulated data.
    Stop cowardice, ban guns!

  3. #3
    Points: 946, Level: 16
    Level completed: 46%, Points required for next Level: 54

    Posts
    11
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Minimum control size for determining effectiveness of treatment

    There's no direct way to calculate this? I'm looking for something like this http://www.evanmiller.org/ab-testing/sample-size.html (though this is for an evenly split A/B test)

    I would imagine that this issue comes up all the time (need to have a control group for verification, but want to keep it as small as possible). Are there any resources or papers published on how to approach this problem?

    Thanks

    EDIT: I did find a potential calculator and am trying to determine if they do what I want them to do. Does this look right (or wrong) to anyone?

    http://stochasticsolutions.com/cgi-b...iss/fleiss.cgi

    It uses Fleiss formula - more info here http://stochasticsolutions.com/fleiss-help.html

    Reading through it it does sound like it's what I'm looking for
    Last edited by xandalorian; 01-08-2016 at 10:59 AM.

  4. #4
    Devorador de queso
    Points: 95,705, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,931
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Minimum control size for determining effectiveness of treatment

    What is the most important question for you to answer. Keep in mind that by moving the split away from 50/50 you can make a smaller confidence interval around the mean of the treatment but you're actually increasing the width of the confidence interval for the difference between the treatment and control means.
    I don't have emotions and sometimes that makes me very sad.

  5. #5
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Minimum control size for determining effectiveness of treatment

    Is there randomization of treatment?


    Are there sample size limitations.


    How is the outcome defined?


    Do you have hypothesized effect estimates and dispersion for outcome (if applicable).


    You can move forward anyway you like, but this seems ideal for a data simulation.
    Stop cowardice, ban guns!

  6. #6
    Points: 946, Level: 16
    Level completed: 46%, Points required for next Level: 54

    Posts
    11
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Minimum control size for determining effectiveness of treatment

    > Is there randomization of treatment?

    Yes

    > Are there sample size limitations.

    Total population is 500K, control group should be as small as possible



    > How is the outcome defined?

    It's a binary outcome.



    > Do you have hypothesized effect estimates and dispersion for outcome (if applicable).

    Not sure, but we would have some baseline data.

    This is actually for an email campaign, but it sounded like something related to epidemiology so I thought I tried to translate it here. Basically, I've been asked to either find a simple tool, or create a simple tool for determining minimum sample size based on some parameters. I found this tool http://stochasticsolutions.com/cgi-b...iss/fleiss.cgi but I've found a few bugs with it, and not a lot of technical detail so I'm hesitant (some explanation for it here though http://stochasticsolutions.com/fleiss-help.html)

    It uses Fleiss formula, and the method seems to come from this paper http://www.jstor.org/stable/2529990?...n_tab_contents

    It does sound like this is what I need, but I can't find any R packages for it (which would allow me to do it myself and verify the tool)...

  7. #7
    Devorador de queso
    Points: 95,705, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,931
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Minimum control size for determining effectiveness of treatment

    Quoted since it got ignored:

    Quote Originally Posted by Dason View Post
    What is the most important question for you to answer. Keep in mind that by moving the split away from 50/50 you can make a smaller confidence interval around the mean of the treatment but you're actually increasing the width of the confidence interval for the difference between the treatment and control means.
    You keep saying you want the control as small as possible and I can understand why intuitively you would think that would be the way to go. But once again - what is your actual goal? What is the most important question you want to answer?
    I don't have emotions and sometimes that makes me very sad.

  8. #8
    Points: 946, Level: 16
    Level completed: 46%, Points required for next Level: 54

    Posts
    11
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Minimum control size for determining effectiveness of treatment

    Sorry missed that reply. The question to answer is "how much better is the test group performing", but it comes with the constraint that we want as many people as possible to receive the treatment (email).

  9. #9
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Minimum control size for determining effectiveness of treatment

    So you have an exposure (control or experiment group) and binary outcome. You are looking at a simple logistic model (it seems).


    Now you need to hypothesize the proportion of successful outcome for each exposure group.


    And you are assuming there will be 500k observations? So you can find a difference here probably pretty easily if it truly exists, given the sample size. You will also need to make a decision on the meaningfulness of the difference, since the test will be over powered, until you start shifting the number of observations into the experimental group.
    Stop cowardice, ban guns!

  10. #10
    Devorador de queso
    Points: 95,705, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,931
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Minimum control size for determining effectiveness of treatment

    Quote Originally Posted by hlsmith View Post
    since the test will be over powered, until you start shifting the number of observations into the experimental group.
    This is bad logic. No test is over powered. If you ever think that a test is "over powered" then you're not actually performing a test of what you're interested in.
    I don't have emotions and sometimes that makes me very sad.

  11. #11
    Points: 946, Level: 16
    Level completed: 46%, Points required for next Level: 54

    Posts
    11
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Minimum control size for determining effectiveness of treatment

    Sorry let me reduce that to a better example I have 20K people to send the email to. I'm tasked with answering "what is the minimum number of people needed in the control group for us to be able to determine the improvement of the test group". Previously they have just done an 80/20 split just as a standard, now they are looking for a specific answer to the question, along with some kind of motivation. Should we have 1%, 5%, 10% etc in the test group?

  12. #12
    Devorador de queso
    Points: 95,705, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,931
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Minimum control size for determining effectiveness of treatment

    It entirely depends on what the effect size is. What is your response variable anyways? Is it a number or just a yes/no binary kind of thing?
    I don't have emotions and sometimes that makes me very sad.

  13. #13
    Points: 946, Level: 16
    Level completed: 46%, Points required for next Level: 54

    Posts
    11
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Minimum control size for determining effectiveness of treatment

    Well there are a few different scenarios. In the simplest case, it's just binary. But there are situations where it will be a number (didn't want to bring that up at first due to it being more complicated).

    > It entirely depends on what the effect size is.

    Yes, but let's assume it's x...how would I proceed (in the binary case)?

  14. #14
    Points: 466, Level: 9
    Level completed: 32%, Points required for next Level: 34

    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Minimum control size for determining effectiveness of treatment

    I see why the size difference is so much. I forgot about the other dimensions. But is there really always risk with a community set up for turtles? What if they are brothers? If they get along when they are hatchlings will they get along when they are adults? What if there are no females?

  15. #15
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Minimum control size for determining effectiveness of treatment


    I like turtles of the teenage mutant variety.


    Dason, you are being too logical. Overpower is a broad idea, of say you want to show two parameter estimates are different, which are truly different and you have a sample approaching the population size. Well, you have plenty of people to show a difference. Now say I start making one subgroup a smaller proportion, well its n-value used in the standard error will get smaller, etc. So you are taking a very well powered comparison and shrinking it down to the bare minimal needed to power the test. So overpowered to marginally powered. You get this concept, but love to be a stickler. And no you can never be statistically overpowered but from a study design standpoint you can be overpowered given caveats (risk, cost, burden, etc.).


    P.S., The overpower idea usually comes into play when you are able to find non-relevant or clinically insignificant differences.
    Stop cowardice, ban guns!

+ Reply to Thread
Page 1 of 2 1 2 LastLast

           




Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats