Looking at the vector field is not the way to look at the correlation, you should try to understand the result of each subprocess. You have to check visually that the two ortho-images are well aligned. If this is not the case, it's useless to go any further in the processing.
Then you look at the correlation looking separately on each band, and you try to estimate the noise level of each band. You'll need to threshold the values with low SNR and large unphysical displacements. You may see the ASTER attitude undulations.
When you said that the results are not valid, you should try to quantify them. Talk about the range of the correlation values, their spread, distribution function, etc...
If you didn't use any GCP, you may have a general offset between the 2 images, corrupting the measurements. You may need to be removing some first order polynomial in the correlation.