This question is not an interview question, but a question usually asked by an interviewer right when you see them. This is especially common if the building location is hard to find. Here is another question you should tell exactly what happened.
With the first answer, they will say something like, that's good. Then they will move on to other questions. If you answer with the second question, they will probably talk a little more about how he or she agrees that this building is hard to find. Either answers are acceptable.