Is it understandable to refuse someone whom i haven't met pick me up at my home?

Hi.I am planning to join a fraternity and got to contact a member of the group. I got to talk to him over the phone who seemed very nice and we set up appointment to meet with other members as well. Since I don't have my own car, I asked for a ride and he asked what my address was. Even though I definitely don't think that he is a bad person, I wouldn't feel completely comfortable for someone whom I haven't even met know my address and stuff. So I just asked a place where he would p by on way and hop in. He didn't say anything about my idea, but is it rude to do so? It's not that I don't trust him but I don't feel as comfortable as with those whom I have known for a while. I just hope I didn't hurt anyone's feeling. I would appreciate your comment. Thanks!