Skip to content

Conversation

@Piezoid
Copy link
Contributor

@Piezoid Piezoid commented Mar 17, 2023

I do not expect this to be merged, but I figured it might help others. Although, I don't know if this is the right place.

This logs information to a ./out.log (hard-coded) file. I wrote this throwaway code before the switch to stderr, which is why it uses a global file handle.
The refactoring of the sampler code should produce the same results than the master branch.

For each predicted token, it logs:

in:' because' n_past=14, remaining_tokens=62, embd.size()=1, embd_inp.size()=13
soft_max: top_sact=25.503617 mean_sact=19.826111 top_p=0.357196 entropy=1.664120
top_p: n: 15 sum: 0.990421
->0: ' they' p=0.357196 act=17.853 temp=0.70
  1: ' I' p=0.231013 act=20.643 temp=0.82
  2: ' of' p=0.228527 act=17.540 temp=0.70
[...]
  15: ' the' p=0.000876 act=13.645 temp=0.70
  • The soft_max: lines reports statistics of the top k tokens' logits (divided by temp) and entropy (in nats, not bits),
  • top_p: line gives the number of retained tokens after the top p filtering, and the sum of their probabilities,
  • Last, a list of the top 16 tokens, along with their respective probabilities, original logits, and the product of their temperature and eventual repetition penalty.. The drawn token is indicated by an ->.

I will close this either when it becomes obsolete or when it can no longer be rebased.

@gjmulder gjmulder added the enhancement New feature or request label Mar 20, 2023
@Piezoid Piezoid closed this Mar 24, 2023
AAbushady pushed a commit to AAbushady/llama.cpp that referenced this pull request Jan 27, 2024
* add tokens per second output

* Update gpttype_adapter.cpp

simplify

---------

Co-authored-by: LostRuins <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants