lol yea nice catch. now that I think of it I should've described the conversion better in the paper, but tbh most of what is written in the code was guesswork since I didn't find any official GQA conversion scripts. also that specific part wasn't used bc turning linear QKV into MLPs did nothing lol
02.12.2024 22:13
👍 1
🔁 0
💬 1
📌 0