No, I don't really have this problem. I can easily visualize the house I lived in 8 years ago.
However, this does remind me of something funny I do. I often imagine people I know in their home environment, even if I haven't ever been to their house. My mind automatically generates a house for each person... it's not something I consciously build or think about. It's usually constructed of various elements from houses I've lived in, or the houses of my extended family. I have a feeling most people don't do this. Actually, maybe people would think it's creepy to think of them at home. There's nothing creepy about it to me... I'll just think, "I wonder what so-and-so does/is doing at home?" Whatever house my mind has built for them remains their house in my mind until I've actually seen their real house.