There's no reason for the whole CrossAttention object to be repeated when only the operation in the middle changes.