Thread: How meaningful is to pool data from different surveys on the same target population?

1. How meaningful is to pool data from different surveys on the same target population?

HI everybody, this is a very theoretical question for me but I would like to find an answer as I might want to use this in the (near) future.
In many cases same questions are used in different surveys which are targeted for the same (or almost the same) population (and all designed to give national estimates). However, as most surveys have long time periods between waves, sometimes I would like to compare results from different data sources which, perhaps, have data from two consecutive years.
I have done some basic reading (Korn & Graubard 1999; Thomas 2007) but in many cases they provide examples on how to pool data from different waves of the same survey and, sub-sequentially, how to normalize the weights.
However, many papers open to the possibility to pool data from different data sources, bearing in mind that populations must be comparable and there should be no evident differences in terms of sampling strategy/survey methodology.

But what if surveys have been conducted on the same population and the questions asked are pretty much the same but with the two surveys being conducted with a (slightly) different sampling strategy (e.g. three sampling units vs two sampling units, but both surveys are country representative and have similar sample size)? Is it still meaningful to pool data together? and if so, how should the weights be adjusted? In same cases I read that authors advised to rescale the weights according to the new sample size. Not sure if it is the case in this situation as well though. What should you recommend?

2. Re: How meaningful is to pool data from different surveys on the same target populati

I don't know the answer to this although I suspect meta-analysis is something you should look at. Simply asking the same population does not mean you can compare across eras because populations change. Asking questions of white males in 1960 would likely have led to very different answers (and ways of understanding the questions even) than in 1968 even with exactly the same question. People interpret reality through a culture and against the political reality of a given time which can have dramatic impact on results.

3. Re: How meaningful is to pool data from different surveys on the same target populati

So you don't have a unique ID to match individual responses? What is your hypothesis?

4. Re: How meaningful is to pool data from different surveys on the same target populati

I am aware of possible bias but in many cases the time span between two surveys would be 1-2 years.
I excluded the meta-analysis option as I would be interested in time trends on the same population.
As we are talking about different surveys I wouldn't be able to link data longitudinally but I would like to look at changes over time (given that in theory each sample would be representative of the population at that specific time). I do not have a specific question yet but for instance I might want to look at physical activity in people over 65 yo in the UK using different data sources which would give me the possibility to have at least two data points within 5 years. This to see whether some policies put in place have an impact on this.

I would certainly consider each result carefully given the different types of bias, but I am not sure whether this can be done if the sampling method is slightly different even if the population target is the same. Also, I have doubts about how to re-scale weights when pooling different data sources together. Normally I would either divide the weights by N or I would divide the weights by the mean of the weights in each sub-sample before pooling data together.

 Tweet