Add clause tokenize model

[wannaphong]

Clause tokenize model trained from LST20 Corpus with CRF model.
LST20 Corpus from National Electronics and Computer Technology Center, Thailand. You can download dataset from https://aiforthai.in.th/corpus.php.

Model train by Mr.Wannaphong Phatthiyaphaibun

Model License : CC-0

Code

• Model File : https://github.com/PyThaiNLP/pythainlp-corpus/releases/tag/lst20-cls-v0.1
• Train script : https://github.com/PyThaiNLP/pythainlp_notebook/blob/master/cls/lst20.ipynb
• Google Colab : https://colab.research.google.com/drive/13sKMQN6FbI9YhyN3rVW9Ydg9bUjKk0JP?usp=sharing

TODO

• document
• test

[pep8speaks]

Hello @wannaphong! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-10-03 09:47:32 UTC

[coveralls]

Coverage increased (+0.04%) to 95.47% when pulling b8ab03e on add-clause into 261f032 on dev.

[wannaphong]

Model Card - LST20 CLS

Model Details

• Developer : Wannaphong Phatthiyaphaibun
• Model date : 2020-10-03
• Model version : 0.2
• CRF Model
• License : CC0

Intended Use

• Segmenting Thai text into clauses (smaller than sentence, bigger than word)
• Not suitable for other language or non-news domain.

Factors

• Based on known problems with thai natural Language processing.

Metrics

• Evaluation metrics include precision, recall and f1-score.

Training Data
LST20 Corpus Train set (news domain)

Evaluation Data
LST20 Corpus Test set (news domain)

Quantitative Analyses

              precision    recall  f1-score   support

       B_CLS       0.90      0.94      0.92     16111
       E_CLS       0.90      0.94      0.92     15947
       I_CLS       0.99      0.97      0.98    169565

   micro avg       0.97      0.97      0.97    201623
   macro avg       0.93      0.95      0.94    201623
weighted avg       0.97      0.97      0.97    201623
 samples avg       0.94      0.94      0.94    201623

Ethical Considerations
no ideas

Caveats and Recommendations

• You should segmente words before use.
• Thai text only

[bact]

trying to make the feature names more consistent.