software speed

#1
Not homework BTW ... I want to measure the processing time (performance) of some software under a set of fixed conditions. I have a means to log the start and stop of the processing. The only possible factors are some uncontrollable (noise) factors, e.g., the CPU and memory usage. Then there is random variation.

I know a little about typical design of experiments to determine the effect of factors. However, in this case, should I just see if there is any correlation of the uncontrollable (noise) factors on the results. I`m not sure yet how to go about that --- determining if the noise factors affect the results over just random variation.

It`s not clear to me how to determine how many runs I need to make for statistically valid results.

If someone could just point me in the right direction, I would appreciate it.

Thanks, Stanley
 

hlsmith

Less is more. Stay pure. Stay poor.
#2
Is there a certain software you are interested in? Is there a certain function / procedure you are interested in? Does a certain sample size requirement need to be met?

Provide a few more details please.
 
#3
Is there a certain software you are interested in? Is there a certain function / procedure you are interested in? Does a certain sample size requirement need to be met?

Provide a few more details please.
The software receives an input message at a specified rate, does some processing, and publishes an output message. The input and output messages are time stamped and logged. I need to measure the time from message input to message output, and to understand the variability in this measured time.
 

hlsmith

Less is more. Stay pure. Stay poor.
#4
Some software has a temporary memory that gets built up if not purges or restarted, does your software have this?

So it will be the exact same input and function over and over again?
 
#5
It is very possible that there might be some sort of "warm up" effect like this, and it could be complicated. The longer term behavior is of more interest. One thing I should do is determine if there is such an effect.

In general, I suppose I should be looking at time series analysis for each run for that, and then use some method to look at data across runs.

I am still not sure how to determine how many runs I need, even after I have some data to analyze.
 

hlsmith

Less is more. Stay pure. Stay poor.
#7
Well as long as nothing is changing and the process is stable, perhaps some kind of convergence can be established based on some many iterations without substantive change.