Is it really outrageous to believe that the Earth may have been visited by other-world beings at some point in its history?

Until we find evidence of other-world beings capable of crossing interstellar distances? Yes.