It is difficult to reason about this without actual data, but maybe I could just look at the correlation structure of the residuals first, as this seems to be the problem?

regards