I hail from RPG.SE (we do tabletop, pen and paper RPGs), and RPG General Chat. I've not been here actively for quite a while (just in and out as an observer sometimes), but your chat room seems to be pretty similar to ours: similar levels of activity and chat regulars, and a generally sociable atmosphere which ebbs and flows between being on- and off-topic with priority given to on-topic stuff.
I don't know your exact situation, but given the similarities I might be able to offer some insight and suggestions. I've read this meta, and the attached chat. I note Alec's complaint that the room is very casual, and that there are large amounts of time devoted to topics he's (she's?) unhappy with.
If we're discussing the matter of chat's casualness, I need to bring up that chat was created to be a "third place" (see the SE blog and wikipedia). Third places are those places away from home or work we congregate and socialise, and among their hallmarks is that the mood is playful:
The tone of conversation in third places are never marked with tension or hostility. Instead, they have a playful nature, where witty conversation and frivolous banter are not only common, but highly valued. — Wikipedia
Chats are always going to be casual and social because it's what we're in chat for.
We're also here because we like game development, so that's always going to be one of the fun topics. That means an "off topic" room is not going to be very good at staying that way. It's always going to drift back on topic, and probably just look exactly like what your main chat looks like now.
Which means other rooms are just going to be... other rooms. And actually that seems to be our answer here.
On creating extra rooms to facilitate whatever people need
We've gotten pretty good at this on RPG Chat. I'll share what we do about this.
Primarily we talk in RPG General Chat. This place looks pretty much like your main chat room: we socialise, go on- or off-topic, discuss advanced or simple stuff, etc.
We try to keep General Chat a safe place where anyone can join in. To those ends we created Not A Bar (the name's a joke). It's our pressure valve for unpleasant conversations and arguments. Those get moved there wholesale from General Chat when it's called for. It also makes for a pretty good side-conversation room when General Chat's busy.
We also have some purpose-built rooms that just naturally emerged and may fade out of use later:
The Campfire is where we share stories about our tabletop sessions. It started a few weeks ago to do just that while General Chat was busy, and has stuck around.
Genesis is where we do random character generation, usually for fun. We created it recently once we started doing a whole lot of that in Not A Bar.
- Before that we were using a room just for deadEarth, which is one of the worst designed RPGs ever, and on top of that offensive in most ways: sexist, homophobic, racist, you name it. (So, trigger warning, the material we quote from it might offend.) Characters are extremely likely to come out of character creation crippled, dying, or already dead, but that also makes character creation amusing and strangely relaxing.
Some users also have their very own rooms we've taken to calling spoil-lairs, which they'll use to plan their upcoming RPG sessions so that their fellow players (who are also in chat) don't see it.
And so on. The list has changed over time.
This is what it sounds like you need to be doing.
You might need a professional version of Not A Bar: a pressure-valve side-room where people can go to discuss advanced topics while Main Chat's busy. Main Chat can't do all things, and it will always be a social place. You might want an AI Guild room, or an AI Chat, or a Super Advanced Stuff chat, or a Super Basics chat.
You might want a "move it when someone asks" rule, you might not. (Our General Chat upholds a general rule that anyone can ping the room's owner and request a conversation be moved to Not A Bar, and it'll be moved. This has only ever been invoked for conversations that were genuinely unpleasant for that person: it's genuinely overstaying its welcome. We've only had to do this a few times, so your mileage may vary.)
You may want to just create chats about topics as they're needed. Is someone busy talking about pointers in the main room, but you want to talk about advanced physics for in-motion collision detection of soft bodies? Create a room, link it in main chat (if you paste just the URL, it oneboxes all fancy-like). If you find you're needing to talk about advanced mathematics often, maybe it's worth having an advanced mathematics room to be available while main chat's busy, or to just be the go-to place for that stuff.
If you need a place where "people know their stuff" (quoting that rationale conversation), it'll come about one of these ways.