The visibility map is a tiny side file with two bits per heap page: all-visible (every row on the page is visible to every snapshot) and all-frozen (every row is frozen). A normal vacuum skips all-visible pages entirely (on a large, mostly-static table that is almost all of them), and index-only scans use the same bit to skip heap fetches. An aggressive vacuum trusts only the all-frozen bit.