Data Engineering

Get JSON for each translation

curl https://bolls.life/static/translations/NASB.json > nasb.json curl https://bolls.life/static/translations/NIV.json > niv.json curl https://bolls.life/static/translations/ESV.json > esv.json

Generate each csv dataset from JSON

Example.Dataset.gen_bible("nasb") Example.Dataset.gen_bible("niv") Example.Dataset.gen_bible("esv")

clean up the nasb csv and then niv & esv

%s/[([^]]*)]/\1/g %s/’/'/g %s/‘/'/g

generate the context len 100 chunks for all 3

Example.Prep.generate("nasb", encoder: true) Example.Prep.generate("niv", encoder: true) Example.Prep.generate("esv", encoder: true)

combine them all into a single csv

mkdir combined cd combined cp ../nasbtraining.csv . cp ../nivtraining.csv . cp ../esvtraining.csv . cat *.csv > pretraining.csv

shuffle and uniq the pretraining data

Example.Prep.shuffle()

Training

Example.Encoder.scheduled(280, 0.00017)

Evaluation

Example.Scoring.evaluate() Example.Scoring.comprehensive_eval()

Prepare the NLT dataset

curl https://bolls.life/static/translations/NLT.json > nlt.json Example.Dataset.gen_bible("nlt")

vim find/replace

%s/
/ /g %s/(.{-})</i>/\1/g %s/’/'/g %s/‘/'/g

Seed the database from a given text

Example.Utils.seed("nlt") Example.Utils.add_verse_embeddings()

Index verses for BM25 search

Example.Verse.index_verses()

Run the app and search

iex -S mix phx.server