Data visualization and overview

[mgill]

Hi,

I 30,000 paired data points (x,y) that are generated by my program. I need to create an overview display like here: http://finance.google.com/finance?q=NASDAQ:CSCO where the bottom graph shows the all the data.

What method of analysis/averaging is best to us to show the user an overview of the data? I thought about avergaing every 20 points or so, but I'm afraid to lose big spikes followed by small spikes. Any ideas?

 

[Mean Joe]

Hi,

I 30,000 paired data points (x,y) that are generated by my program. I need to create an overview display like here: http://finance.google.com/finance?q=NASDAQ:CSCO where the bottom graph shows the all the data.

What method of analysis/averaging is best to us to show the user an overview of the data? I thought about avergaing every 20 points or so, but I'm afraid to lose big spikes followed by small spikes. Any ideas?

I may be mis-understanding what you are going to do with your data. But the display that you link to (stock price over 3 days) does not do any averaging. You can find (if you have REALLY good eyes or a super microscope or something) the stock price at 12:24 pm and 55 seconds on April 15, then find the price at 12:24pm and 56 seconds on April 15, etc.

If you want to do that, just put all 30,000 of your data points. Or do you have multiple y values for a given x value?

 

[Keo]

I couldn't understand what the graph on the bottom means, but I think that what you are looking for is a function called "banking", which forces the aspect ratio of the graph to have the slopes centered on 45 degrees, giving a nice overview of the bigger cycles. (http://projetos.inpa.gov.br/i3geo/pacotes/r/win/library/lattice/html/banking.html)
You can achieve this also using the loess function (http://sekhon.berkeley.edu/stats/html/loess.html), and setting the span attribute to whatever smoothing you like.
The amount of smoothing you need will depend on how much of the local variance of a given segment you attribute to noise.