"But there are the first two, formula and probability that seem too specific and should be synonyms for mathematics or perhaps a similar tag."
I make no argument with regard to the formula tag, but I recommend keeping probability as its own standalone tag. Probability is a very distinct area of expertise, with lots of counter-intuitive quirks and potential gotchas that are easy for non-specialists to fall into, so it requires particular styles of approach.
Game probability in particular encompasses specialized topics like loot drop systems, pseudorandom number generation methods, shuffling mechanics like Tetris piece selection algorithms, and mixed strategy equilibria in game theory - so even on a game-, not math-focused exchange, I think this category is meaty enough to stand on its own.
I can see an argument for considering math to be meta, but I think even with its breadth it's still specific enough to say something useful about the question's content.
Speaking for myself, I have it set as one of my favourite tags. I'm a math geek so I like checking out questions that demand some calculating, graphing, derivation, or a little light proving. I find the mathematics tag is currently effective at highlighting these questions, and correctly signals when my expertise might be relevant.
And, as you say, folks coming here with math questions are often doing so because they lack the terminology to define what they need in more specific terms like linear-algebra, so having a mathematics tag helps the folks who do know this terminology find the questions they might be able to help with or improve. Without the mathematics tag, I don't think the users asking these questions would find more specific terms to use - we'd just get more sparsely tagged questions that are harder to filter by expertise.