+ Reply to Thread
Results 1 to 12 of 12

Thread: Analysing a designed experiment - is the theory wrong?

  1. #1
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Analysing a designed experiment - is the theory wrong?




    hi,
    just had a scary thought . in all industrial DoEs we definitely have measurement error on the IVs. However I have always seen the OLS as the proposed method to analyse them. Is this an error? Should we actually use a Deming regression or some more sophisticated method?

    If the variance of the X measurements is less then the variance of the Y measurement this might be ok, but I have never seen this comparison be actually performed or even proposed.

    regards

  2. #2
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    How do you know the measurements are off? Is there a validation variable collected. If you know the bias, there may be a sensitivity analysis to incorporate.

    What is this variance ratio you mentioned?
    Stop cowardice, ban guns!

  3. #3
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    Hi,
    they are generally not off pe se but have their own measurement variance so I habe no way to know whether the level thatbwas set is the level actually achieved. This should be incorporated in the analysis - actually I can not see an application of multiple fegression where this would not be the case.

  4. #4
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    Well issues are typically selection bias, measurement error, confounding, and model misclassification. Your concerns are that there may be bias? I can remember what Deming reg entails. Would it help.

    I think you need to think about if there is systematic error such as a pattern in the mismeasure or if it was random.
    Stop cowardice, ban guns!

  5. #5
    Cookie Scientist
    Points: 13,431, Level: 75
    Level completed: 46%, Points required for next Level: 219
    Jake's Avatar
    Location
    Austin, TX
    Posts
    1,293
    Thanks
    66
    Thanked 584 Times in 438 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    If all you want to know about is the relationship between the X measurements and Y, then there is no need to correct for measurement error. We need only to assume that future X measurements will contain about as much error as the current X measurements. But if your interest is in the relationship between Y and the underlying X "true scores"--that is, the underlying thing supposedly being measured--then correcting for measurement error would be appropriate. In science we are more often interested in the latter, but in industrial experiments you may well only care about the former.
    “In God we trust. All others must bring data.”
    ~W. Edwards Deming

  6. #6
    TS Contributor
    Points: 22,383, Level: 93
    Level completed: 4%, Points required for next Level: 967
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    Quote Originally Posted by Jake View Post
    If all you want to know about is the relationship between the X measurements and Y, then there is no need to correct for measurement error. We need only to assume that future X measurements will contain about as much error as the current X measurements. But if your interest is in the relationship between Y and the underlying X "true scores"--that is, the underlying thing supposedly being measured--then correcting for measurement error would be appropriate. In science we are more often interested in the latter, but in industrial experiments you may well only care about the former.
    Taking on the psychometric perspective of measurement error, I see...

    ...

    ...

    well, then I guess I can't really add anything else, LOL.

    But I am curious, though... is there any reason as far as why you would make a distinction between, say the process of measurement and (statistical) analysis in science VS in industry? I mean, as a business owner I could see myself interested in making sure I can measure things (sales, amount of product distributed, etc.) as accurately as I can.
    for all your psychometric needs! https://psychometroscar.wordpress.com/about/

  7. #7
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    So you are not able to measure with the exactitude fully desired. I guess it comes down to your purpose and if there is any concern toward systematic error. What are your thoughts?
    Stop cowardice, ban guns!

  8. #8
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    Quote Originally Posted by Jake View Post
    If all you want to know about is the relationship between the X measurements and Y, then there is no need to correct for measurement error. We need only to assume that future X measurements will contain about as much error as the current X measurements.
    Hi,
    I could imagine several scenarios here : e.g. I could measure X precisely at a given moment but I can not keep it constant during the experiment - imagine it is a reaction temperature that is regulated by a thermostat for instance. I set the value and the "real "temperature will go up and down according to the algorithm the thermostat follows. So my original X is just a rough approximation of what I actually do - but I would consider it a fixed given value in the OLS regression.

    The other alternative would be a measurement that really has a lot of variation - like measuring viscosity for instance. My "real" input could be anywhere in a wide range of possible values due to measurement variation (it is not a bias BTW) I just do not know the real value. Again, the traditional DoE analysis with OLS would just assume the X has the value we measured.

    Of course, in both cases we would be interested in the relationship between the real X and the real Y, the measured /set values being more or less random.

  9. #9
    TS Contributor
    Points: 12,227, Level: 72
    Level completed: 45%, Points required for next Level: 223
    rogojel's Avatar
    Location
    I work in Europe, live in Hungary
    Posts
    1,470
    Thanks
    160
    Thanked 332 Times in 312 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    Quote Originally Posted by spunky View Post
    But I am curious, though... is there any reason as far as why you would make a distinction between, say the process of measurement and (statistical) analysis in science VS in industry? I mean, as a business owner I could see myself interested in making sure I can measure things (sales, amount of product distributed, etc.) as accurately as I can.
    It is a bit more complex then that but that would be a loong story. If you report the measurement to a customer for instance, then maybe not. If the accurate measurement costs a lot more and you can not control the process as accurately then again maybe not ...

  10. #10
    Human
    Points: 12,676, Level: 73
    Level completed: 57%, Points required for next Level: 174
    Awards:
    Master Tagger
    GretaGarbo's Avatar
    Posts
    1,362
    Thanks
    455
    Thanked 462 Times in 402 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    When I don't know the answer to a good question, like rogojels, I try to google and search for it.

    This is what I found, a thesis by Angela Mcmahan "Measurement Error in Designed Experiments for Second Order Models".

    The first 4 - 5 pages seems to summarize about 'classical measurement errors' and 'Berksom errors' and properties for OLS estimates.

  11. The Following 2 Users Say Thank You to GretaGarbo For This Useful Post:

    Miner (07-18-2016), rogojel (07-18-2016)

  12. #11
    Omega Contributor
    Points: 38,303, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,993
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: Analysing a designed experiment - is the theory wrong?

    I could see real world temperatures always being variable given where it was taken and ambient characteristics. It isn't measurement error given the variable being recorded, but I still think it falls under bias, because it is not a recording of the true measure of the underlying phenomena. We just don't like hearing the word bias and our minds cringe and associate it with systematic errors. But if I want to know the temp on something given an experiment (equation) and I can't not control for everything my errors result in a bias. Think about this under the guise of the bias/variance tradeoff.


    I digress. The big thing is that I believe this always happens in experiments, but most move forward with standard approaches and just document these limitation in their write-up. I am off to check out GG's link.
    Stop cowardice, ban guns!

  13. #12
    TS Contributor
    Points: 14,811, Level: 78
    Level completed: 91%, Points required for next Level: 39
    Miner's Avatar
    Location
    Greater Milwaukee area
    Posts
    1,171
    Thanks
    34
    Thanked 405 Times in 363 Posts

    Re: Analysing a designed experiment - is the theory wrong?


    rogojel,

    You work in manufacturing, correct? What follows, is correct for industrial statistics. I'm not certain whether it can generalize to other branches of statistics.

    In industrial statistics, DOEs are used for 4 purposes:
    1. Screening: to identify the few significant factors that affect a process
    2. Modeling: to create a mathematical model describing the relationship between process factors and outputs
    3. Optimizing: to use the model to minimize/maximize/target the process output
    4. Robust optimizing (desensitizing): to make the process output less sensitive to variation in the factors
    In both Screening and Robust Optimizing, you are typically looking for moderate to large effects and the errors in measurements (EIM) issue is too small to worry about. If they were large enough to impact the results and decisions made, you have other problems to worry about and the effects are too small to be of practical value.

    For Modeling and Optimizing, the EIM issue might cause a practical problem if your model needs to be very precise. In my personal experience, this issue was never a problem in the manufacturing environment. However, I have seen several product design applications where this could have caused some practical issues. These were situations where the model needed to be very precise and was used by the product's control software.

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats