Based on this answer this starts vim and exits if there's nothing in the buffer. The disadvantage is that it starts vim so the screen "flashes". It also doesn't set an exit code when a man page isn't found.
viman () { vim -R +':set ft=man|exe !search(".")?"quit!":""' <(man "$@" 2>/dev/null); }
This is an improvement on Jeff Schaller's answer in that it doesn't load the the man page twice when it exists. It also doesn't load vim unnecessarily like my previous example. And it does set an exit code when there's no man page.
viman () { man -f "$@" >/dev/null 2>&1 && vim -R +":set ft=man" <(man "$@"); }
Both examples use Bash process substitution in order to avoid the "Vim: Reading from stdin..." message.
Neither loads the page into a variable.