There are 2 problems here: (a) OSM's doesn't have as much house numbering data (that'll improve with time) and (b) addressing → lat/long is pretty freaking hard. If you limit yourself to one country and one address system, it's much easier. Otherwise things get messy fast.