Repository navigation
Update Weights Operation #13
Description
Hi @eric-mitchell,
Thank you for the repo and the great paper. I was looking at your code and I have a doubt on this line of code: why is there a plus? Shouldn't be a minus?
As you also wrote in the paper, the final edited weights should be:
Thanks for your help.
Activity
eric-mitchell commentedon May 19, 2023
You're right about this- good catch! Turns out this is indeed a bug in the implementation. Have you been able to train a model with this corrected, and seen if there is a difference in performance?
EscVM commentedon May 21, 2023
Actually, I've never run your code, but I adapted it for another use case. Anyway, the code works also with the plus. The optimization process makes the network flip all gradient signs. However, I expect slightly better results with the right sign from the beginning, simply because the task should be a little simpler from the point of view of hypernetworks.