It's probably not a scam per se. He might just be trying to rustle up business, but it's also possible someone did point out your house and say they really liked it. My parents got a few calls like this, over the years, and whenever one of their neighbors had a house up for sale apparently there would be people who would come for a showing and then say, "what I'd REALLY like is to buy that house across the street...."

There is a house in St. Paul that is on a cul de sac overlooking Hillcrest Park and the library; if I lived there, my kids would literally be able to get to a library, a rec center, a sledding hill, an ice skating rink, and a playground, all without so much as crossing a street. It's also a big, beautiful house with big beautiful trees in the yard. I have been tempted a few times to slip a note under their door saying, "YOU ARE LIVING IN MY DREAM HOUSE. IF YOU EVER DECIDE TO MOVE, PLEASE CALL ME SO I KNOW TO CALL A REALTOR. HERE IS MY NUMBER." But it just seems a little creepy. (Also, honestly, I doubt we could afford it.)