Stars in our galaxy aren't static. Rather moving around quickly like fishes in a tank. Our sun has already passed though the center of our galaxy 50 times.
There in comes another twist. If there were other intelligent life forms in the universe they wouldn't need to have invented long range interstellar travel to colonize the universe. They could have just hopped to the nearest star and colonized it all.
The only reason we don't see them is that, humans are the only life form in the visible universe. There's no one else out there. That's Hart's theory.