Bicycle Tutor

What is difference between compression and non compression brake cable housing? I found out that non compression will not work on Shimano 7800 brake and shift levers after installing a new FSA egg shaped bar. The current Avid Flak Jacket I have installed are non compression and cable will not retract via spring in brake.

Difference is as it says. Non-compression usually have the steel strands running length wise through the casing.
Compression has the metal strand spiraling around along the length. Compression would normally be easier to bend.
Be aware there are 2 different cable sizes. Brake cables (1.6mm) are normally thicker than gear cables (1.2mm) so the outers should have different bores to correspond.
If you are using the same cables as before then look for kinks or broken strands on the inners. If nothing obvious then undo the cable from brake or gear and see if it's free then.

Non-compression (NC) brake housing does have the strands running parallel like on gear cable housing, but it is specially reinforced. (Note to all that you should never use gear cable housing for brakes as the sides will blow out.)
NC housing does not compress like standard housing. This will give a stiffer, more responsive feel to the brakes. Downside seems to be that the housing is very stiff and doesn't bend well, especially on suspension bikes where it needs to constantly flex. You also have to use ferrules specifically made for NC housing or the ends will blow out.
I'm not familiar with your bar. But would guess that something about the shape forces a sharp bend that the NC housing can't negotiate well. It may have kinked or is just pinching the cable a little so the friction is too much for the brakes springs to pull the cable back.
I've never tried the NC stuff so can't compare, but have found that high end standard housing has very little compression anyway. I especially like the stuff where the inner wires are coated as well. Nice and slippery even with sharp bends in cables.