Should we merge syllable_tokenize() and subword_tokenize() together?

[bact]

They are both subword level.

Currently we have 2 level of analysis and 4 kinds of implementations ("engines"):

• TCC-based
  • Original TCC
  • Extended TCC
• Syllable
  • Using syllable dictionary
  • Using CRF (ssg)

Should we call them all "subword"?

Note: currently "ssg" is available as an option in both syllable_tokenize() and subword_tokenize().

[wannaphong]

We should probably combine these two things into subword.