NER: some issues in PyTorch Lightning example

[stefan-it]

Hi,

I wanted to try out the new NER example script (./ner/run_pl_ner.py) that uses PyTorch Lightning.

Here are some bugs that I've found:

Dataset preparation method is not called. Usually, InputBatch batches or input features are written and stored in a file. However, the prepare_data() [1] method is not called and no input features are written. I fixed that adding this method to the train_dataloader() [2] function, but I'm not sure if it's the right place.

Model training will work then.

Evaluation is currently not working correctly. The checkpoint output file name is:

# ls
'checkpointepoch=0.ckpt'  'checkpointepoch=1.ckpt'  'checkpointepoch=2.ckpt'

so the pattern checkpointepoch=<number_epoch>.ckpt is used, whereas the main script expects an output checkpoint pattern of checkpoint_<number_epoch>.ckpt [3]

[1] https://github.com/huggingface/transformers/blob/master/examples/ner/run_pl_ner.py#L56-L80
[2] https://github.com/huggingface/transformers/blob/master/examples/ner/transformer_base.py#L126-L139
[3] https://github.com/huggingface/transformers/blob/master/examples/ner/run_pl_ner.py#L220

[stefan-it]

After fixing the these issues the output predictions seems very weird, maybe this is in an label 2 id bug?

Nachdem I-OTH
er I-PERpart
1907 B-ORGderiv
nicht I-PER
zum I-PER
Direktor I-PER
der I-PERderiv
Bibliothek B-ORGpart
berufen I-PER
wurde I-PER
, I-PER
verließ B-PERderiv
Gardthausen I-PERderiv
den I-ORG
Bibliotheksdienst B-ORGpart
. I-PERderiv

😂

[minimaxir]

cc @srush (via #3053)

[srush]

Thanks I will take a look. Can you verify you are on the latest pytorch-lightning? prepare_data was just added.

Also can you post your log. What was the Val accuracy? Are you on single GPU?

[shubhamagarwal92]

@srush Please see the PR #3180 I have updated the bash script and README to run effortlessly.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.