not good when I use BERT for seq2seq model in keyphrase generation

[whqwill]

Hi,

recently, I am researching about Keyphrase generation. Usually, people use seq2seq with attention model to deal with such problem. Specifically I use the framework: https://github.com/memray/seq2seq-keyphrase-pytorch, which is implementation of http://memray.me/uploads/acl17-keyphrase-generation.pdf .

Now I just change its encoder part to BERT, but the result is not good. The experiment comparison of two models is in the attachment.

Can you give me some advice if what I did is reasonable and if BERT is suitable for doing such a thing?

Thanks.
RNN vs BERT in Keyphrase generation.pdf

[waynedane]

have u tried transformer decoder ?instead of rnn decoder.

[whqwill]

not yet, I will try. But I think rnn decoder should not be such bad.

[waynedane]

not yet, I will try. But I think rnn decoder should not be such bad.

emmm，maybe u should used mean of last layer to initialize decoder, not the last token representation of last layer.
I am also very concerned about the results of using transformer decoder. If you are done, can you tell me? Thank you.

[waynedane]

I think the batch size of RNN with BERT is too small. pleas see

https://github.com/memray/seq2seq-keyphrase-pytorch/blob/master/pykp/dataloader.py
line 377-378

[whqwill]

I don't know what you mean by giving me this link. I set to 10 really because of the memory problem. Actually, when sentence length is 512, the max batch size is only 5, if it is 6 or bigger there will be memory error for my GPU.

[whqwill]

not yet, I will try. But I think rnn decoder should not be such bad.

emmm，maybe u should used mean of last layer to initialize decoder, not the last token representation of last layer.
I am also very concerned about the results of using transformer decoder. If you are done, can you tell me? Thank you.

You are right. Maybe the mean is better, I will try as well. Thanks.

[waynedane]

May i ask a question? R u chinese?23333

[waynedane]

Cause for one example, it has N targets. We wanna put all targets in the same batch. 10 is too small that the targets of one example would be in different batches probably.

[whqwill]

I know, but ... the same problem ... my memory is limited .. so ...

PS. I am Chinese

[waynedane]

I know, but ... the same problem ... my memory is limited .. so ...

PS. I am Chinese

i am as well hahaha

[waynedane]

是不是语料的问题，bert是在wiki上训练的。我用kp20k训练了一个mini bert，在测试集上的accuracy目前是80%，你要不要试试用我这个作为encoder？

[whqwill]

这个80%具体是什么数值这么高？f1 score吗？ 你的encoder能不能发来看一下呢 谢 waynedane <notifications@github.com>于2018年11月28日 周三下午11:14写道：

…

是不是语料的问题，bert是在wiki上训练的。我用kp20k训练了一个mini bert，在测试集上的accuracy目前是80%，你要不要试试用我这个作为encoder？ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#59 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHCjdIT1G6Icse3LK2SZXO194JJTiM1Qks5uzqhMgaJpZM4Y3HWV> .

[waynedane]

accuracy 是masklm和nextsentence两个任务的，不是key phrase generation，我没说清楚，抱歉。我的算力有限，两块p100, 快一个月了，目前还没训练完。80%是当前的表现。

[whqwill]

你提到的mini bert 是什么意思？

[whqwill]

我大概理解你的意思了，你相当于是用kp20重新预训练一个bert，不过这样做... 感觉确实蛮麻烦。