Answers with jokes VS. joke-answers
I think we should start by differentiating the following. An answer that tries to address a question but has an incidental joke within it is very different from an answer that is only a joke - for the only sake of being funny.
The first case should be allowed because of the obvious reason that the primary function of an answer is trying to answer, no matter if it uses an academic, an informal or a funny tone. However, the second case should not be allowed simply because, well, if it's not even trying to answer, then it's not an answer (an example of a fairly hilarious answer like that is the following: How can I create borders in an open-world game that don't feel artificial?).
It's subjective. But just as always
Of course, in practice the boundary that separates answers with jokes from joke-answers can become less clear. However, it is just as subjective as the similar decisions that are enacted when questions are flagged as "too broad" or when someone thinks a given answer looks more like a comment and then flags it as "it's not an answer". When it happens, a decision has to be made on whether in fact the answer was just a comment (i.e. related to the original question, but not trying to answer it). I don't see why that is so different from the case where jokes were what motivated the flagging.
Sure, one person deciding that is always a hard call. There is where the voting system for deleting answers would theoretically work the best, since multiple people summing up their interpretations about a given answer being or not an answer (due to whatever the reason) is always the ideal due to the highly subjective nature of the decision. However, the problem is two-folded.
In practice, moderators are the only barrier against joke-answers
As @JoshPetrie has brought to my attention in the comments (I admitedly forgot that), only answers scored -1 or less can be voted for deletion. Considering that funny answers tend to be often upvoted, that alone puts them more often than not, outside the helm of possibly being voted for deletion.
If that barrier was not enough, we have to also recall that only users with at least 20K can vote for deletion. It means that in the rare event of a joke answer to have negative scores for long enough to be voted for deletion, even then, considering that gamedev.stackexchange currently has only 10 members with such rep (two of them are the moderators) and not all are necessarily active in reviewing tasks, that additional barrier to delete joke answers makes it virtually impossible that any joke answers ever get deleted.
This would open up a risky possibility. Therefore, it seems that in the vast majority of the cases, the moderators themselves are the only option for not letting a joke answer endure. In such a case, I sustain that yes, the moderators should have to make such decision when joke answers get flagged.
TL;DR
And to sum up, as I argued above, the minimum criteria for it should be the same used in all other cases of "not an answer" flagging. I.e. deciding whether the answer really tries to usefully answer the original question, despite of the joke it carries.