A haunted house is one that is scary, usually because of the presence of spirits or ghosts (and typically also in need of a paint job). Though the verb haunt does mean “to visit or inhabit as a ghost,” that’s not the original sense of the word. In fact, for centuries the word had a perfectly unfrightening set of meanings: “to visit often” and “to continually seek the company of.” In the 1500s, haunt began to mean “to have a disquieting or harmful effect on,” as in “that problem may come back to haunt you.” The meaning here is simply the lingering presence of the problem, not the possibly scary nature of the problem itself; it is applied to thoughts, memories, and emotions. The noun haunt retains this fright-neutral definition, “a place that you go to often,” as in “one of my favorite old haunts.”
A lingering idea, memory, or feeling may have led to the ghostly meaning of haunt, or one by a disembodied or imaginary spirit, proving that what we truly fear is in our head.