I've seen armor carriers that mimic the appearance of the uniform, so that when you put it on, it blends in with your uniform and therefore is somewhat concealed right out in the open. As others have indicated in this (old) thread, if you're responding to an event where a vest is essentially mandatory, you should consider waiting for Law Enforcement to secure the area before you enter.
That being said, a Level II vest with protection against ice pick stabs is a good idea because sometimes the situation you're in can turn on you really quickly. The vest can also provide some protection to your torso in the event of a crash and you're unsecured...
And a Level IV plate won't stop a .50 BMG, they're supposed to stop ONE shot of .30-06 AP (M2 Armor Piercing, specifically). A SAPI plate might stop that round, but it'll pass to a slightly lower standard... A .50 BMG round has a LOT more energy than that...