Small optimizations to VisibilityCache

Short-circuit isVisibleFrom when no refs passed. Exclude heads and
tags from other refs, since those were already checked.

Change-Id: I5d5d0952a1db67c33ffc20df402f976946459137
1 file changed