Right, you can format one column based on the values from another column (that is not even in your pivot).
However if you are trying to compare the actual vs target columns, you probably need to create a calculated field first that finds the difference, and then base your conditional formatting off of this field.