prevent markdownish from modifying text inside code block #3181

nathan-sixnines · 2023-09-14T20:59:46Z

was trying to pull out blocks of python code but the markdownish replacements were breaking stuff in the code blocks.

Added some logic to skip the replacements inside code blocks.

tobi · 2023-09-17T22:49:49Z

Nice

Co-authored-by: root <[email protected]>

ggml-ci

… build ...`) (#6964) * readme: cmake . -B build && cmake --build build * build: fix typo Co-authored-by: Jared Van Bortel <[email protected]> * build: drop implicit . from cmake config command * build: remove another superfluous . * build: update MinGW cmake commands * Update README-sycl.md Co-authored-by: Neo Zhang Jianyu <[email protected]> * build: reinstate --config Release as not the default w/ some generators + document how to build Debug * build: revert more --config Release * build: nit / remove -H from cmake example * build: reword debug instructions around single/multi config split --------- Co-authored-by: Jared Van Bortel <[email protected]> Co-authored-by: Neo Zhang Jianyu <[email protected]>

Signed-off-by: Daniel Bevenius <[email protected]>

@ochafik

* Cleaning up integration tests to share code between tests and make it simpler to add new tests. * Add tests around quantifiers to ensure both matching and non-matching compliance. * Add slightly more complex grammar with quantifiers to test references with quantifiers. * Fixing build when C++17 is not present. * Separating test calls to give more helpful stack traces on failure. Adding verbose messages to give visibility for what is being tested. * Adding quotes around strings to explicitly show whitespace * Removing trailing whitespace. * Implementing suggestions from @ochafik -- grammars and test strings now print and flush before tests to aid in debugging segfaults and whatnot. * Cleaning up forgotten symbols. Modifying simple test to use test harness. Added comments for more verbose descriptions of what each test is accomplishing. * Unicode symbol modifications to hopefully make log easier to parse visually.

* args: default --model to models/ + filename from --model-url or --hf-file (or else legacy models/7B/ggml-model-f16.gguf) * args: main & server now call gpt_params_handle_model_default * args: define DEFAULT_MODEL_PATH + update cli docs * curl: check url of previous download (.json metadata w/ url, etag & lastModified) * args: fix update to quantize-stats.cpp * curl: support legacy .etag / .lastModified companion files * curl: rm legacy .etag file support * curl: reuse regex across headers callback calls * curl: unique_ptr to manage lifecycle of curl & outfile * curl: nit: no need for multiline regex flag * curl: update failed test (model file collision) + gitignore *.gguf.json

* convert : use utf8 encoding * convert : update instructions and warning message

* ggml : add ggml_flash_attn_ext API * ggml : fix GQA support in ggml_flash_attn_ext * ggml : online attention (CPU) * metal : initial implementation * metal : f16 precision * metal : reduce branches * metal : specialize for head size * wip : 8 rows per simd group * wip : 4 rows per simd group * wip : template for rows per warp * metal : parallelize across KV size * metal : parallel reduce across heads * metal : efficient flash_attn_f16 implementation * metal : avoid redundant loads of the attention * metal : scale and mask in matrix form * metal : fix comment * llama : avoid ggml_cast, use F32 query * metal : add parallel reduce version (disabled) * metal : move output into local memory + optimize - the result from each simdgroup now stays in the registers - significantly reduced SRAM usage - more efficient skipping of -INF blocks - avoid simdgroup barrier in hot loop - add comments * metal : add tests, fix scaling, support C > 32 * metal : improve precision * ggml : fix f16 mad * metal : minor * metal : support Q > 8 * tests : add ATTN tests * metal : disable buffer allocation logs * tests : more * metal : faster inner loop for C == 32 * metal : fix array initialization * tests : ifdef * ggml : switch to padded F16 mask for ggml_soft_max, ggml_flash_attn_ext * ggml : fix ggml_soft_max mask requirement * cuda : fix soft_max to use correct mask size * cuda : add flash_attn kernel (wip) * metal : optimize softmax for C > 32 * metal : optimize softmax * tests : minor fix * cuda : avoid zeroing fragments * tests : update dims * cuda : fix __hisinf() result check * cuda : avoid warp_reduce for smax * cuda : use int instead of int64_t Noticeably improves performance (thanks to Johannes) * cuda : make loops use the same loop values Thanks Johannes again for the tip * cuda : unroll some of the loops * cuda : avoid __hisinf branches * cuda : use half2 in softmax * cuda : switch to 1 warp for bs > 16 * cuda : speed-up reduce part of the kernel * cuda : unroll Q*K^T loop * cuda : fix -INF block check * cuda : simplify softmax * cuda : fix matrix names * cuda : minor * llama : adapt to F16 KQ_pos * llama : adapt new models to F16 KQ_mask * ggml : fix F16 store (ARM NEON) * llama : fix type of KQ_mask and KQ_pos * ggml : fix CPU soft_max * tests : add hs=256 * cuda : fix build * metal : improve perf via smaller int registers * cuda : adapt soft_max to F16 mask and pos * CUDA: faster FlashAttention, kernel for bs == 1 * 16 cols for Phi-2 * no vec for hs, no hs==256 ncols==32 for Volta * adjust kernel selection logic * 4 warps, 256 stride for all D * no ncols == 64 * Multiple parallel blocks for batch size 1 * fix compile warnings * fix excessive KQ_b loads * fix cmake build * fix KV cache padding, NaN from INFINITY (#6438) * llama : flash_attn cparam + fix defrag * server: support flash_attn param * server: bench: enable flash_attn param * CUDA: refactor host code, dyn. par. blocks * fix flash_attn_vec_f16 race condition * flush softmax exp below threshold to 0 * store temp KQ in registers * Calculate KQ as FP32 if KQV has GGML_PREC_F32 * Add __hgt2_mask implementation for CUDA 11 * fix KQ FP32 precision fpr parallel_blocks > 1 * llama-bench : add -fa,--flash-attn arg * metal : add BS=1 kernel for flash attention (#6508) * metal : add BS=1 kernel for flash attention (wip) * metal : support more than 1 warps * metal : opts * metal : opt * metal : switch to parallel reduce * metal : reduce registers * metal : simplify * metal : initial FA vec kernel * metal : use F32 attention accumulators * batched-bench : add fattn arg * llama : simplify llama_build_kv_store ggml-ci * llama : adapt build_olmo to changes * ggml : fix arm fp16 store on windows * metal : clean-up * metal : clean-up kernel code * metal : minor * tests : remove benchmarks ggml-ci * ggml : fix avx512 const correctness ggml-ci * ggml : fix soft_max with bias on CPU ggml-ci * common : print --flash-attn in help * ggml : fix num dimensions in ggml_flash_attn_ext * llama : force disable flash attention for incompatible models * ggml : ggml_soft_max support F16/F32 mask/pos ggml-ci * cuda : uint -> uint32_t * cuda : "constexpr dim3" -> "const dim3" ggml-ci * cuda : try to fix __hgt2_mask ggml-ci * ggml : add TODO's for F16/F32 mask/pos support in other backends * llama : replace bool need_kq_pos with use_alibi * llama : prep ALiBi support for BERT models ggml-ci * llama : fix n_batch requirements ggml-ci * cont * server : add help for --flash-attn arg * llama : disable FA for AMD * tests : remove TMP_ATTN_BENCH ggml-ci * llama : support save/load state with FA enabled ggml-ci * ci : add CUDA save-load-state tests ggml-ci * llama : llama_kv_cache_clear zeroes data + fix save-load seq ggml-ci * llama : fix copy-paste errors, add TODO * llama : disallow incompatible states * llama : update llama_state_get_size after v_trans field * metal : remove tmp log * llama : add static reminder for llama_state_get_size * metal : fix max nsg ggml-ci * ci : fix arg order ggml-ci --------- Co-authored-by: Johannes Gäßler <[email protected]> Co-authored-by: Pierrick HYMBERT <[email protected]>

* perplexity: more statistics, added documentation * add LLaMA 3 8b scoreboard

ROCm clang defines _MSC_VER which results in the wrong implementation of LOG_IMPL and LOG_TEE_IMPL being compiled. This fixes #6972

Co-authored-by: Jared Van Bortel <[email protected]>

* llama : rename ctx to user_data in progress_callback This commit renames the `ctx` parameter to `user_data` in the `llama_progress_callback` typedef. The motivation for this is that other callbacks use `user_data` or `data`, and using `ctx` in this case might be confusing as it could be confused with `llama_context`. --------- Signed-off-by: Daniel Bevenius <[email protected]>

* convert.py: add python logging instead of print() * convert.py: verbose flag takes priority over dump flag log suppression * convert.py: named instance logging * convert.py: use explicit logger id string * convert.py: convert extra print() to named logger * convert.py: sys.stderr.write --> logger.error * *.py: Convert all python scripts to use logging module * requirements.txt: remove extra line * flake8: update flake8 ignore and exclude to match ci settings * gh-actions: add flake8-no-print to flake8 lint step * pre-commit: add flake8-no-print to flake8 and also update pre-commit version * convert-hf-to-gguf.py: print() to logger conversion * *.py: logging basiconfig refactor to use conditional expression * *.py: removed commented out logging * fixup! *.py: logging basiconfig refactor to use conditional expression * constant.py: logger.error then exit should be a raise exception instead * *.py: Convert logger error and sys.exit() into a raise exception (for atypical error) * gguf-convert-endian.py: refactor convert_byteorder() to use tqdm progressbar * verify-checksum-model.py: This is the result of the program, it should be printed to stdout. * compare-llama-bench.py: add blank line for readability during missing repo response * reader.py: read_gguf_file() use print() over logging * convert.py: warning goes to stderr and won't hurt the dump output * gguf-dump.py: dump_metadata() should print to stdout * convert-hf-to-gguf.py: print --> logger.debug or ValueError() * verify-checksum-models.py: use print() for printing table * *.py: refactor logging.basicConfig() * gguf-py/gguf/*.py: use __name__ as logger name Since they will be imported and not run directly. * python-lint.yml: use .flake8 file instead * constants.py: logger no longer required * convert-hf-to-gguf.py: add additional logging * convert-hf-to-gguf.py: print() --> logger * *.py: fix flake8 warnings * revert changes to convert-hf-to-gguf.py for get_name() * convert-hf-to-gguf-update.py: use triple quoted f-string instead * *.py: accidentally corrected the wrong line * *.py: add compilade warning suggestions and style fixes

* tests : add test-tokenizer-0.sh * unicode : add all unicode number ranges * starcoder : fix pre-tokenizer * tests : add test that fails with DeepSeek tokenizers * falcon : fix regex * unicode : regenerate unicode tables * refact : add tokenizer model * lint : fix * tests : disable failing tests ggml-ci * refact : add tests files ggml-ci * convert : print -> logging ggml-ci * lint : fix * unicode : digit -> number * phi-3 : update

…will crash (#7038) This will reproduce the issue in llama13b { 'prompt': 'Q: hello world \nA: ', 'stop': ['\n'], 'temperature': 0.0, 'n_predict': 10, 'cache_prompt': True, 'n_probs': 10 }

* Tidy Android Instructions README.md Remove CLBlast instructions(outdated), added OpenBlas. * don't assume git is installed Added apt install git, so that git clone works * removed OpenBlas Linked to Linux build instructions * fix typo Remove word "run" * correct style Co-authored-by: slaren <[email protected]> * correct grammar Co-authored-by: slaren <[email protected]> * delete reference to Android API * remove Fdroid reference, link directly to Termux Fdroid is not required Co-authored-by: slaren <[email protected]> * Update README.md Co-authored-by: slaren <[email protected]> --------- Co-authored-by: slaren <[email protected]>

Set one as executable and add basicConfig() to another. Also added noqa tag to test scripts.

* Add BPE pre-tokenization for Command-R/R+. * Bump transformers convert requirement. * command-r : add individual digits regex --------- Co-authored-by: Georgi Gerganov <[email protected]>

github-actions · 2024-05-27T18:56:06Z

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 550 iterations 🚀

Expand details for performance related PR only

Concurrent users: 8, duration: 10m
HTTP request : avg=8496.55ms p(95)=19965.28ms fails=, finish reason: stop=504 truncated=46
Prompt processing (pp): avg=96.27tk/s p(95)=351.22tk/s
Token generation (tg): avg=34.0tk/s p(95)=46.21tk/s
ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=Markdownish_Codeblock_fix commit=10729fa03c81c59afefaf021f900de3fd9aafec4

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1716835530 --> 1716836160
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 829.83, 829.83, 829.83, 829.83, 829.83, 939.92, 939.92, 939.92, 939.92, 939.92, 924.63, 924.63, 924.63, 924.63, 924.63, 949.68, 949.68, 949.68, 949.68, 949.68, 991.32, 991.32, 991.32, 991.32, 991.32, 976.23, 976.23, 976.23, 976.23, 976.23, 977.23, 977.23, 977.23, 977.23, 977.23, 978.26, 978.26, 978.26, 978.26, 978.26, 982.88, 982.88, 982.88, 982.88, 982.88, 975.2, 975.2, 975.2, 975.2, 975.2, 984.36, 984.36, 984.36, 984.36, 984.36, 964.51, 964.51, 964.51, 964.51, 964.51, 936.34, 936.34, 936.34, 936.34, 936.34, 946.02, 946.02, 946.02, 946.02, 946.02, 887.06, 887.06, 887.06, 887.06, 887.06, 892.25, 892.25, 892.25, 892.25, 892.25, 891.87, 891.87, 891.87, 891.87, 891.87, 898.1, 898.1, 898.1, 898.1, 898.1, 911.31, 911.31, 911.31, 911.31, 911.31, 911.22, 911.22, 911.22, 911.22, 911.22, 915.58, 915.58, 915.58, 915.58, 915.58, 916.11, 916.11, 916.11, 916.11, 916.11, 880.81, 880.81, 880.81, 880.81, 880.81, 881.31, 881.31, 881.31, 881.31, 881.31, 883.76, 883.76, 883.76, 883.76, 883.76, 892.7, 892.7, 892.7, 892.7, 892.7, 890.55, 890.55, 890.55, 890.55, 890.55, 890.07, 890.07, 890.07, 890.07, 890.07, 891.05, 891.05, 891.05, 891.05, 891.05, 894.2, 894.2, 894.2, 894.2, 894.2, 891.07, 891.07, 891.07, 891.07, 891.07, 892.8, 892.8, 892.8, 892.8, 892.8, 897.85, 897.85, 897.85, 897.85, 897.85, 903.72, 903.72, 903.72, 903.72, 903.72, 910.37, 910.37, 910.37, 910.37, 910.37, 917.57, 917.57, 917.57, 917.57, 917.57, 912.66, 912.66, 912.66, 912.66, 912.66, 911.16, 911.16, 911.16, 911.16, 911.16, 914.77, 914.77, 914.77, 914.77, 914.77, 915.85, 915.85, 915.85, 915.85, 915.85, 922.23, 922.23, 922.23, 922.23, 922.23, 925.53, 925.53, 925.53, 925.53, 925.53, 905.34, 905.34, 905.34, 905.34, 905.34, 903.12, 903.12, 903.12, 903.12, 903.12, 905.17, 905.17, 905.17, 905.17, 905.17, 906.84, 906.84, 906.84, 906.84, 906.84, 905.6, 905.6, 905.6, 905.6, 905.6, 907.27, 907.27, 907.27, 907.27, 907.27, 907.49, 907.49, 907.49, 907.49, 907.49, 907.37, 907.37, 907.37, 907.37, 907.37, 908.36, 908.36, 908.36, 908.36, 908.36, 907.32, 907.32, 907.32, 907.32, 907.32, 912.74, 912.74, 912.74, 912.74, 912.74, 910.98, 910.98, 910.98, 910.98, 910.98, 910.87, 910.87, 910.87, 910.87, 910.87, 910.09, 910.09, 910.09, 910.09, 910.09, 910.69, 910.69, 910.69, 910.69, 910.69, 911.55, 911.55, 911.55, 911.55, 911.55, 911.89, 911.89, 911.89, 911.89, 911.89, 913.7, 913.7, 913.7, 913.7, 913.7, 913.97]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1716835530 --> 1716836160
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 41.38, 41.38, 41.38, 41.38, 41.38, 30.12, 30.12, 30.12, 30.12, 30.12, 29.11, 29.11, 29.11, 29.11, 29.11, 30.53, 30.53, 30.53, 30.53, 30.53, 30.78, 30.78, 30.78, 30.78, 30.78, 31.96, 31.96, 31.96, 31.96, 31.96, 32.96, 32.96, 32.96, 32.96, 32.96, 33.58, 33.58, 33.58, 33.58, 33.58, 33.47, 33.47, 33.47, 33.47, 33.47, 32.92, 32.92, 32.92, 32.92, 32.92, 32.6, 32.6, 32.6, 32.6, 32.6, 32.19, 32.19, 32.19, 32.19, 32.19, 31.5, 31.5, 31.5, 31.5, 31.5, 31.05, 31.05, 31.05, 31.05, 31.05, 30.55, 30.55, 30.55, 30.55, 30.55, 29.21, 29.21, 29.21, 29.21, 29.21, 29.11, 29.11, 29.11, 29.11, 29.11, 29.4, 29.4, 29.4, 29.4, 29.4, 29.49, 29.49, 29.49, 29.49, 29.49, 29.52, 29.52, 29.52, 29.52, 29.52, 29.71, 29.71, 29.71, 29.71, 29.71, 29.89, 29.89, 29.89, 29.89, 29.89, 30.09, 30.09, 30.09, 30.09, 30.09, 30.19, 30.19, 30.19, 30.19, 30.19, 30.55, 30.55, 30.55, 30.55, 30.55, 30.61, 30.61, 30.61, 30.61, 30.61, 30.58, 30.58, 30.58, 30.58, 30.58, 30.77, 30.77, 30.77, 30.77, 30.77, 30.98, 30.98, 30.98, 30.98, 30.98, 31.06, 31.06, 31.06, 31.06, 31.06, 31.14, 31.14, 31.14, 31.14, 31.14, 31.22, 31.22, 31.22, 31.22, 31.22, 31.24, 31.24, 31.24, 31.24, 31.24, 31.06, 31.06, 31.06, 31.06, 31.06, 30.89, 30.89, 30.89, 30.89, 30.89, 30.71, 30.71, 30.71, 30.71, 30.71, 30.55, 30.55, 30.55, 30.55, 30.55, 30.71, 30.71, 30.71, 30.71, 30.71, 30.87, 30.87, 30.87, 30.87, 30.87, 30.92, 30.92, 30.92, 30.92, 30.92, 30.99, 30.99, 30.99, 30.99, 30.99, 30.76, 30.76, 30.76, 30.76, 30.76, 30.57, 30.57, 30.57, 30.57, 30.57, 30.02, 30.02, 30.02, 30.02, 30.02, 29.5, 29.5, 29.5, 29.5, 29.5, 29.48, 29.48, 29.48, 29.48, 29.48, 29.45, 29.45, 29.45, 29.45, 29.45, 29.47, 29.47, 29.47, 29.47, 29.47, 29.49, 29.49, 29.49, 29.49, 29.49, 29.54, 29.54, 29.54, 29.54, 29.54, 29.57, 29.57, 29.57, 29.57, 29.57, 29.46, 29.46, 29.46, 29.46, 29.46, 29.49, 29.49, 29.49, 29.49, 29.49, 29.45, 29.45, 29.45, 29.45, 29.45, 29.41, 29.41, 29.41, 29.41, 29.41, 29.54, 29.54, 29.54, 29.54, 29.54, 29.66, 29.66, 29.66, 29.66, 29.66, 29.77, 29.77, 29.77, 29.77, 29.77, 29.88, 29.88, 29.88, 29.88, 29.88, 29.86, 29.86, 29.86, 29.86, 29.86, 29.93]

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1716835530 --> 1716836160
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.29, 0.29, 0.29, 0.29, 0.29, 0.4, 0.4, 0.4, 0.4, 0.4, 0.19, 0.19, 0.19, 0.19, 0.19, 0.1, 0.1, 0.1, 0.1, 0.1, 0.22, 0.22, 0.22, 0.22, 0.22, 0.13, 0.13, 0.13, 0.13, 0.13, 0.16, 0.16, 0.16, 0.16, 0.16, 0.2, 0.2, 0.2, 0.2, 0.2, 0.27, 0.27, 0.27, 0.27, 0.27, 0.23, 0.23, 0.23, 0.23, 0.23, 0.17, 0.17, 0.17, 0.17, 0.17, 0.35, 0.35, 0.35, 0.35, 0.35, 0.29, 0.29, 0.29, 0.29, 0.29, 0.36, 0.36, 0.36, 0.36, 0.36, 0.28, 0.28, 0.28, 0.28, 0.28, 0.32, 0.32, 0.32, 0.32, 0.32, 0.14, 0.14, 0.14, 0.14, 0.14, 0.18, 0.18, 0.18, 0.18, 0.18, 0.12, 0.12, 0.12, 0.12, 0.12, 0.15, 0.15, 0.15, 0.15, 0.15, 0.2, 0.2, 0.2, 0.2, 0.2, 0.16, 0.16, 0.16, 0.16, 0.16, 0.28, 0.28, 0.28, 0.28, 0.28, 0.11, 0.11, 0.11, 0.11, 0.11, 0.13, 0.13, 0.13, 0.13, 0.13, 0.19, 0.19, 0.19, 0.19, 0.19, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.2, 0.2, 0.2, 0.2, 0.2, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.29, 0.29, 0.29, 0.29, 0.29, 0.15, 0.15, 0.15, 0.15, 0.15, 0.1, 0.1, 0.1, 0.1, 0.1, 0.09, 0.09, 0.09, 0.09, 0.09, 0.17, 0.17, 0.17, 0.17, 0.17, 0.27, 0.27, 0.27, 0.27, 0.27, 0.52, 0.52, 0.52, 0.52, 0.52, 0.42, 0.42, 0.42, 0.42, 0.42, 0.36, 0.36, 0.36, 0.36, 0.36, 0.25, 0.25, 0.25, 0.25, 0.25, 0.23, 0.23, 0.23, 0.23, 0.23, 0.26, 0.26, 0.26, 0.26, 0.26, 0.14, 0.14, 0.14, 0.14, 0.14, 0.17, 0.17, 0.17, 0.17, 0.17, 0.23, 0.23, 0.23, 0.23, 0.23, 0.24, 0.24, 0.24, 0.24, 0.24, 0.17, 0.17, 0.17, 0.17, 0.17, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.27, 0.11, 0.11, 0.11, 0.11, 0.11, 0.1, 0.1, 0.1, 0.1, 0.1, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.21]

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1716835530 --> 1716836160
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 1.0, 1.0, 1.0, 1.0, 1.0, 6.0, 6.0, 6.0, 6.0, 6.0, 1.0]

cpumaxx and others added 29 commits April 29, 2024 17:34

llava-cli : multiple images (#6969)

ffe6665

Co-authored-by: root <[email protected]>

ggml : fix __MSC_VER -> _MSC_VER (#6977)

544f1f1

ggml-ci

ci : tmp disable gguf-split (#6983)

d2c898f

ggml-ci

main : fix typo in comment in main.cpp (#6985)

5539e6f

Signed-off-by: Daniel Bevenius <[email protected]>

convert : use utf8 encoding (#7000)

952d03d

* convert : use utf8 encoding * convert : update instructions and warning message

metal : log more info on error (#6987)

a68a1e7

metal : remove deprecated error code (#7008)

77e15be

switch to using localizedDescription (#7010)

f364eb6

perplexity: more statistics, added documentation (#6936)

a8f9b07

* perplexity: more statistics, added documentation * add LLaMA 3 8b scoreboard

ci : exempt confirmed bugs from being tagged as stale (#7014)

c4ec9c0

CUDA: CUDART < 11.7 workaround for __hmax, __hmax2 (#7019)

1613ef8

Server: add tests for batch size, different seeds (#6950)

3ea0d36

main : fix off by one error for context shift (#6921)

8d608a8

Update LOG_IMPL and LOG_TEE_IMPL (#7029)

b0d943d

ROCm clang defines _MSC_VER which results in the wrong implementation of LOG_IMPL and LOG_TEE_IMPL being compiled. This fixes #6972

chore: fix typo in llama.cpp (#7032)

6ecf318

Co-authored-by: Jared Van Bortel <[email protected]>

Remove .attention from skipped tensors to match more accurately (#7051)

60325fa

If first token generated from the server is the stop word the server …

03fb8a0

…will crash (#7038) This will reproduce the issue in llama13b { 'prompt': 'Q: hello world \nA: ', 'stop': ['\n'], 'temperature': 0.0, 'n_predict': 10, 'cache_prompt': True, 'n_probs': 10 }

Fix Linux /sys cpu path to guess number of cores (#7064)

fcd84a0

gguf-split: add --no-tensor-first-split (#7072)

8425001

py : logging and flake8 suppression refactoring (#7081)

6fbd432

Set one as executable and add basicConfig() to another. Also added noqa tag to test scripts.

command-r : add BPE pre-tokenization (#7063)

889bdd7

* Add BPE pre-tokenization for Command-R/R+. * Bump transformers convert requirement. * command-r : add individual digits regex --------- Co-authored-by: Georgi Gerganov <[email protected]>

nathan-sixnines mentioned this pull request May 27, 2024

Markdownish code block fix #7571

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

prevent markdownish from modifying text inside code block #3181

prevent markdownish from modifying text inside code block #3181

Uh oh!

nathan-sixnines commented Sep 14, 2023

Uh oh!

tobi commented Sep 17, 2023

Uh oh!

github-actions bot commented May 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

91 participants

prevent markdownish from modifying text inside code block #3181

prevent markdownish from modifying text inside code block #3181

Uh oh!

Conversation

nathan-sixnines commented Sep 14, 2023

Uh oh!

tobi commented Sep 17, 2023

Uh oh!

github-actions bot commented May 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

91 participants