[AMOS] differing sample sizes in a covariance matrix

I have a correlation matrix which I want to use to create a path diagram in AMOS.

But for some variables I have N=70 and for others I have N=120

So would i have to go back to the original data and impute it? OR cut out the remaining so, all N=70? or will it cope ok with differeng sample sizes given that I do have a correlation coefficient for each variable?
I have never actually tried an analysis with different sample sizes. Something about it seems wrong but I can't place it.

However, two ideas:

1. Use the raw data.
2. If no raw data, when creating your corr/cov matrix the second line is for the "N" (under the varname_). Why not just enter the appropriate N value differently for different variables?

Seems to me though that you would run into a "missing cases" issue though.