If we're fed `to-address', then always use that.