I actually have a different take on this.
I recently transported a patient who I discovered was a fellow alumni member of my Army unit, American Legionnaire, etc. His health is declining, kids are out of the house, etc. and he asked me for my number in order to stay in contact with me and some mutual friends we have on his behalf. I gave it to him because, well, he's a brother-in-arms. I heard back from him today and we chatted a bit.
I don't want anything from him (except a copy of the unit DVD) and I'm certainly not going to take advantage of him, but if it helps him to share how he's doing with a fellow veteran, or if he needs help moving something or whatnot, I'm a neighbor and a friend. Plus, look at their perception- if you refuse on the grounds of privacy for this situation, you're basically That Guy who no one likes, whereas acceptance is part of the community and comraderie.
I don't think you should go trying to get dates in the back of an ambulance, or randomly handing out your number to people. But there are some times it is acceptable and probably a good thing to do.