This causes some small changes to the uni pc inpainting behavior but it seems to improve results slightly.