Modulation instability in filamentary self-compression