My earliest memory

I have often wondered why I can remember this particular memory, but I can. 
I have a very clear memory of being out on a walk with my mother.  From my viewpoint I am being pushed in a stroller and it begins to rain.  I can clearly remember my Mom pushing me in the stroller into a building that had different colored panels on the outside of it.  We stood and waited in a stairway/foyer area of the building until the rain stopped.  I can recall my Mom having conversation with someone else that also came into the building to get out of the rain.  For some reason many, many, many years later, probably about 30 years later, I related that story to my Father.  He was shocked at the detail I could remember about the building, I remebered what my Mothers coat looked like etc. He then explained to me that the building that I described was an apartment building that was around the corner from our house, along the direction my Mother would take me for a walk every day.  He then told me that the apartment burned down when I was about a year old.  Being I was born in the summer, it would be safe to say that I was only around 9 months old when that happened.  Interesting, I havent thought about that for many years and the memory is still very clear.  I wonder why?