I think we should delete them.
- What makes an answer a joke answer?
- How many joke answers are we ready to have?
Since it's not really possible to have a definite answer to these two questions, it will make moderation harder, and may cause undefined behaviour from passing-by users; I'm afraid every question would get jokes as answers, that these answers get up-voted 'cause they're funny' and obfuscate the more serious content, ultimately leading to a joke Q&A site.
I don't mind jokes in the comments, though.