I think the attack goes something like: You figure out how you should name a temp file by checking to see if files with the same name already exist. Between when you decide on the name and when you open the file, a nefarious user creates her own temp file with mode = 0777. You open your file and write to it, not realizing that another user can now read all your temporary data. Because the file was already created when you opened it, whether your umask is set properly doesn't matter.
It sounds like the OpenBSD implementation would throw an error if the file was owned by someone else when you tried to shm_open() it, which mitigates this race attack. mkstemp mitigates this attack by atomically determining the name and opening the file without the opportunity for a nefarious process to touch the file system in between.
I suppose it depends on how good you are at coming up with random file names. You could also use O_EXCL | O_CREAT to fail if the file already existed. The more I think about it, though, the worse this whole interface is starting to smell.
It sounds like the OpenBSD implementation would throw an error if the file was owned by someone else when you tried to shm_open() it, which mitigates this race attack. mkstemp mitigates this attack by atomically determining the name and opening the file without the opportunity for a nefarious process to touch the file system in between.