At the time I reflexively answered that the remaining correlation must be the product of the two known correlations. Which of course is totally wrong. I think I was getting some mental interference from some of the equations for simple mediation floating around in my head. Anyway, after thinking about it for a while I have come up with a convincing geometric argument for what the constraints actually are. I have also verified that my answer agrees with a more complicated-looking answer to this question that I found elsewhere online.

Because I ending up spending a lot of time on this and I thought some of you would find the results interesting, I thought I would share my work here. Comments are welcome!

Okay. So imagine our variables x, y, z as vectors in n-dimensional space. The Pearson correlation coefficient between any two of these variables can be interpreted as the cosine of the angle between the corresponding vectors. This is an interesting and well-known geometric fact about correlation coefficients.

So now imagine that the x and y vectors are fixed (and hence so is their correlation), but that the vector z is free to vary so long as \(r_{yz}\) is constant. This constraint on \(r_{yz}\) means that the set of possible z vectors will form a sort of "cone" around the y vector, as in the following image:

Now it is intuitively obvious (I know this is a sneaky phrase, but that's why I call this just an "argument" and not a "proof") that the two possible z vectors that will lead to the minimum/maximum values of \(r_{xz}\) are the z vectors that lie on the same plane as the x and y vectors. This leads to the following expression for the minimum/maximum values of \(r_{xz}\) given \(r_{xy}\) and \(r_{yz}\):

\(cos[arccos(r_{xy}) \pm arccos(r_{yz})]\)

One notable result following from this is that if x is orthogonal to y and y is orthogonal to z, then there is no constraint on \(r_{xz}\), it can be anywhere from -1 to +1. But under any other circumstances, fixing \(r_{xy}\) and \(r_{yz}\) will place some constraint on the range of \(r_{xz}\).

Okay, now for the verification part, which requires a bit of math.

So in this stats.stackexchange.com thread (LINK), it is stated that the three correlations must satisfy

\(1+2r_{xy}r_{xz}r_{yz}-(r_{xy}^2+r_{xz}^2+r_{xy}^2) \ge 0\)

The reasoning here being "because this is the determinant of the correlation matrix and it cannot be negative." Anyway, this can be viewed as a quadratic inequality in \(r_{xz}\), already in standard form:

\((-1)r_{xz}^2 + (2r_{xy}r_{yz})r_{xz} + (1 - r_{xy} - r_{yz}) \ge 0\)

So if we apply the quadratic formula and simplify the result, we get the following for the minimum/maximum values of \(r_{xz}\):

\(r_{xy}r_{yz} \pm \sqrt{(1-r_{xy}^2)(1-r_{yz}^2)}\)

Now taking my answer and applying the trig identity \(cos(a \pm b) = cos(a)cos(b) \mp sin(a)sin(b)\) we get

\(r_{xy}r_{yz} \pm sin(arccos(r_{xy}))sin(arccos(r_{yz}))\)

Now applying the identity \(sin(x) = \sqrt{1 - [cos(x)]^2}\) we get

\(r_{xy}r_{yz} \pm \sqrt{(1-r_{xy}^2)(1-r_{yz}^2)}\)

Which is the answer we got from the stackexchange post. So our simpler, geometrically based answer agrees with the more conventional answer that is harder to understand.