Goodness of fit for 2D histograms

Gaba

New Member
#1
Hello all,

I've dragging this problem for quite a while now and I'd really appreciate it if someone could help me whit it.

I have two sets of data representing stars parameters: an observed one and a modeled one. With these sets I create what is called a two-color-diagram (TCD). A sample can be seen here:



A being the observed data and B the data extracted from the model (never mind the black lines, the dots represent the data) I have only one A diagram, but can produce as much different B diagrams as I want and I need to keep the one hat best fits A.

So what I need is a reliable way to check the goodness of fit of diagram B (model) to diagram A (observed).

Important:

1- Points in B are not related one-to-one with points in A. That's an important thing to keep in mind when searching for the best fit: the number of points in A and B is not necessarily the same and the goodness of fit test should also account for this discrepancy and try to minimize it.

2- The number of points in every B data set (model output) I try to fit to A is not fixed.


Right now what I do is I create a 2D histogram or grid (that's what I call it, maybe it has a more proper name) for each diagram by binning both axis (100 bins for each) Then I go through each cell of the grid and I find the absolute difference in counts between A and B for that particular cell. After having gone through all the cells, I sum the values for each cell and so I end up with a single positive parameter representing the goodness of fit (gf) between A and B. The closest to zero, the better the fit. Basically, this is what that parameter looks like:

[TEX] gf=\sum_{ij}|a_{ij}-b_{ij}|[/TEX] ; where aij is the number of stars in diagram A for that particular cell (determined by ij) and bij is the number for B.

This is what those (aij−bij) differences in counts in each cell look like in the grid I create (note that I'm not using absolute values of (aij−bij) in this image but I do use them when calculating the gf parameter):



The problem is that I've been advised that this might not be a good estimator, mainly because other than saying this fit is better than this other one because the parameter is lower, I really can't say anything more.



I've seen the Chi-Squared test used in some cases:

∑i(Oi−Ei)2/Ei; where Oi is observed frequency (model) and Ei is expected frequency (observation).

but the problem is: what do I do if Ei is zero? As you can see in the image above, if I create a grid of those diagrams in that range there will be lots of cells where Ei is zero.

Also, I've read some people recommend a log likelihood Poisson test to be applied in cases like this where histograms are involved. If this is correct I'd really appreciate it if someone could instruct me on how to use that test to this particular case (my knowledge of statistics is pretty poor, so please keep it as simple as you can :)