I think this problem is somewhat (possibly even substantially, if not completely) alleviated by the computer system (CAD) in Howard County's units. I haven't played around with it myself, but looking at the screen as others are using it, it looks like the mapping system can be zoomed in quite a bit, to the point where it shows individual boxes representing structures, each box with its street number.
So in theory, you could navigate to a specific street address solely with that, without even having to look at the numbers on the buildings, curbs, mailboxes, etc. I'm sure it's not flawless, but it seems pretty good.
The only time in my limited experience that I remember there being any problem was when we were going to an apartment building. We got to the building alright, but half of the units had their doors on the left side and the other half on the right side of the building. The letters were on the doors and they were not facing the street. There was no guide on the side of the building that we saw that indicated "A-G" on the left side, etc. (And yes, of course we went to the wrong side of the building first.) So even with a system such as Howard County's, there's still a need for proper marking.