A proper adventure suggests a clear course of action to resolving a problem. If the house is haunted, there is a reason why and a way to unhaunt it. Even if it's just kicking the Boss Ghost's pale transparent ass till he leaves which leads to all his Casperlings leave with him, it's more of an interesting scenario than "Uhh there's a key out there, go find it."
Plus you can do way more interesting things with the ghosts as you now can have friendly ghosts, unfriendly ghosts, hostile ghosts, and invite the PCs to wonder why the place is haunted. Slap on a time limit by saying the spirits can only be contacted on the Winter Solstice or some shit and now you have a sense of urgency in the adventure too. And if they're successful, there's an additional satisfaction in how they can see the progression of the castle from a dingy, spooky, haunted pile of shit to a cleaned up half decent adventurer's base.
Also, "Y-you can't, it's enchan-" doesn't sound much better.