My geometry has errors in it after reopening the file.
After further experimenting, it seems to be a bug with the boolean operation.