Mixture of Experts (MoE) for Self-Hosters, Demystified
MoE LLMs like Mixtral and DeepSeek-V3 run 70B-class quality on 7B-ish active params. Here's how sparse activation works and how to run it at home.
Tag: architecture
All the articles with the tag "architecture".