Tidy Android Instructions README.md

[Jeximo]

It's better to tidy readme regarding CLBlast instructions for Android.

Removed CLBlast instructions(outdated). Simplified Android CPU Build instructions.

[slaren]

Is OpenBLAS actually worth using in Android? For quantized models, it may be faster without it. Ultimately though, without the OpenCL instructions, this basically looks like "install termux and follow the normal build instructions for linux". So maybe it would be simpler that way.

[Jeximo]

Is OpenBLAS actually worth using in Android?

I like leaving the decision to the user if OpenBlas is worth it. I don't use it, but I don't prompt large(supposedly that's where it shines).

this basically looks like "install termux and follow the normal build instructions for linux". So maybe it would be simpler that way.

Agreed.

[teleprint-me]

I build with OpenBLAS on Android, not that it matters. My chiming is, unfortunately, anecdotal. Is it really negligible? It's more difficult to tell on the phone if I'm being honest.

[slaren]

The easiest way to tell if OpenBLAS helps would be to run llama-bench and look at the pp performance. BLAS is only used for prompts with at least 32 tokens.

[Jeximo]

CPU is definitely faster with quants on my device:
OpenBlas:

| model                          |       size |     params | backend    |    threads |    n_batch | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ---------: | ---------: | ---------- | ---------------: |
| llama 8B IQ4_XS - 4.25 bpw     |   3.64 GiB |     7.24 B | BLAS       |          4 |         32 | pp 512     |      1.00 ± 0.00 |
build: a8f9b076 (2775)

CPU:

| model                          |     size   |     params | backend    |    threads |    n_batch | test       |              t/s |                 
| ------------------------------ | ---------: | ---------: | ---------- | ---------: | ---------: | ---------- | ---------------: |                 
| llama 8B IQ4_XS - 4.25 bpw     |   3.64 GiB |     7.24 B | CPU        |          4 |         32 | pp 512     |      3.12 ± 0.06 |                
build: a8f9b076 (2775)

[teleprint-me]

I had to update, fix the convert script by adding the hash, and the upload the model I use, rebuild, and then download the quant. Plus, I have a bunch of other scripts running, so I'll post once it's all set.

[teleprint-me]

CPU is much faster! Why is that?

~ $ ./llama.cpp/llama-bench -m models/stablelm-2-zephyr-1_6b.gguf -t 8
| model                          |       size |     params | backend    |    threads | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ---------: | ---------- | ---------------: |
| stablelm 1B F16 (guessed)      |   3.06 GiB |     1.64 B | BLAS       |          8 | pp 512     |     10.19 ± 1.87 |
| stablelm 1B F16 (guessed)      |   3.06 GiB |     1.64 B | BLAS       |          8 | tg 128     |      2.17 ± 0.17 |

build: a8f9b076 (2775)
~ $ ./llama.cpp/llama-bench -m models/stablelm-2-zephyr-1_6b.gguf -t 8
| model                          |       size |     params | backend    |    threads | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ---------: | ---------- | ---------------: |
| stablelm 1B F16 (guessed)      |   3.06 GiB |     1.64 B | CPU        |          8 | pp 512     |     32.35 ± 2.42 |
| stablelm 1B F16 (guessed)      |   3.06 GiB |     1.64 B | CPU        |          8 | tg 128     |      3.72 ± 0.79 |

build: a8f9b076 (2775)

[Jeximo]

CPU is much faster! Why is that?

I think libopenblas is not a full backend. Vulkan is the way forward for mobile GPU: #6395 (comment)

[Jeximo]

This should only affect the load time of the model though, but performance during inference should be the same.

Thank you. I'll try various options and post results later.

[Jeximo]

Tested --no-mmap on a model loaded frrom ~/ vs shared storage(Downloads). Performance is improved. It appears reduction is due to the combination of Android SAF API & mmap.

Here's some quick numbers, loading from shared:

llama_print_timings:        load time =   26232.51 ms
llama_print_timings:      sample time =      19.78 ms /    33 runs   (    0.60 ms per token,  1668.01 tokens per second)
llama_print_timings: prompt eval time =  186348.29 ms /    51 tokens ( 3653.89 ms per token,     0.27 tokens per second)
llama_print_timings:        eval time =  248449.20 ms /    32 runs   ( 7764.04 ms per token,     0.13 tokens per second)
llama_print_timings:       total time =  443161.08 ms /    83 tokens

load from shared & --no-mmap

llama_print_timings:        load time =   15297.21 ms
llama_print_timings:      sample time =      26.22 ms /    44 runs   (    0.60 ms per token,  1677.85 tokens per second)
llama_print_timings: prompt eval time =   54639.93 ms /    51 tokens ( 1071.37 ms per token,     0.93 tokens per second)
llama_print_timings:        eval time =   39760.87 ms /    43 runs   (  924.67 ms per token,     1.08 tokens per second)
llama_print_timings:       total time =   96297.49 ms /    94 tokens

load from ~/:

llama_print_timings:        load time =    6302.93 ms
llama_print_timings:      sample time =      32.26 ms /    54 runs   (    0.60 ms per token,  1673.85 tokens per second)
llama_print_timings: prompt eval time =   58406.42 ms /    51 tokens ( 1145.22 ms per token,     0.87 tokens per second)
llama_print_timings:        eval time =   48915.58 ms /    53 runs   (  922.94 ms per token,     1.08 tokens per second)
llama_print_timings:       total time =  108573.70 ms /   104 tokens

load from ~/ & --no-mmap:

llama_print_timings:        load time =    5184.56 ms
llama_print_timings:      sample time =      28.71 ms /    49 runs   (    0.59 ms per token,  1706.54 tokens per second)
llama_print_timings: prompt eval time =   46939.36 ms /    51 tokens (  920.38 ms per token,     1.09 tokens per second)
llama_print_timings:        eval time =   44217.39 ms /    48 runs   (  921.20 ms per token,     1.09 tokens per second)
llama_print_timings:       total time =   92946.78 ms /    99 tokens

Based on these figures, --no-mmap & ~/ is the best to load from. I used Meta-Llama-3-8B-Instruct-IQ3_M.gguf. I'll get a small model, and llama-bench later.

[Jeximo]

Tested with TinyLlama-1.1B-Chat-v1.0-Q8_0.gguf using llama-bench. ./llama-bench -t 4 -p 512 -n 128 --mmap 0, --mmap 1

Load from shared, -m /data/data/com.termux/files/home/storage/downloads/TinyLlama-1.1B-Chat-v1.0-Q8_0.gguf

| model                          |       size |     params | backend    |    threads |       mmap | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ---------: | ---------: | ---------- | ---------------: |
| llama 1B Q8_0                  |   1.09 GiB |     1.10 B | CPU        |          4 |          0 | pp 512     |     22.82 ± 0.27 |
| llama 1B Q8_0                  |   1.09 GiB |     1.10 B | CPU        |          4 |          0 | tg 128     |     11.68 ± 0.23 |
| llama 1B Q8_0                  |   1.09 GiB |     1.10 B | CPU        |          4 |          1 | pp 512     |     22.30 ± 0.09 |
| llama 1B Q8_0                  |   1.09 GiB |     1.10 B | CPU        |          4 |          1 | tg 128     |     11.53 ± 0.21 |

build: 628b2991 (2794)

Load from ~/, ~/TinyLlama-1.1B-Chat-v1.0-Q8_0.gguf

| model                          |       size |     params | backend    |    threads |       mmap | test       |              t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ---------: | ---------: | ---------- | ---------------: |
| llama 1B Q8_0                  |   1.09 GiB |     1.10 B | CPU        |          4 |          0 | pp 512     |     22.59 ± 0.22 |
| llama 1B Q8_0                  |   1.09 GiB |     1.10 B | CPU        |          4 |          0 | tg 128     |     11.54 ± 0.08 |
| llama 1B Q8_0                  |   1.09 GiB |     1.10 B | CPU        |          4 |          1 | pp 512     |     22.08 ± 0.08 |
| llama 1B Q8_0                  |   1.09 GiB |     1.10 B | CPU        |          4 |          1 | tg 128     |     11.28 ± 0.25 |

build: 628b2991 (2794)

The results are near identical. Probably Tiny Llama (1.09 GiB) is too small to emphasize difference for this test, even mmap made no difference. I'll leave larger model benching for someone with a better device than mine.

[gustrd]

Hey everyone,

As the original author of these README instructions, I have to admit that I now see how they might cause more confusion than clarity.

Just to clarify for future users: I've personally found CLBlast to be quite effective when used with llama.cpp, especially for certain model families like StableLM and OpenLlama (provided you're not offloading layers). In my experience, it has boosted prompt processing speed by roughly 40%.

However, it's important to note that while CLBlast does offer significant speed improvements, it's plagued by bugs. For many model families, or even within the aforementioned subsets when offloading layers, it tends to produce nonsensical output. This is disappointing, considering the untapped potential of the GPUs nestled within our smartphones.

If there's any way I can assist, I'd like to offer a few insights based on my experimentation:

• In my tests, OpenBLAS consistently outperforms noblas, particularly for prompts exceeding 256 tokens.

• The tip regarding disabling mmap on Android devices is a game-changer. I hadn't been aware of it previously, and it substantially accelerates prompt processing. I strongly advocate for emphasizing this point in the README.

Here's hoping that Vulkan proves to be a more robust solution than OpenGL.

[gpokat]

Hey everyone,

As the original author of these README instructions, I have to admit that I now see how they might cause more confusion than clarity.

Just to clarify for future users: I've personally found CLBlast to be quite effective when used with llama.cpp, especially for certain model families like StableLM and OpenLlama (provided you're not offloading layers). In my experience, it has boosted prompt processing speed by roughly 40%.

However, it's important to note that while CLBlast does offer significant speed improvements, it's plagued by bugs. For many model families, or even within the aforementioned subsets when offloading layers, it tends to produce nonsensical output. This is disappointing, considering the untapped potential of the GPUs nestled within our smartphones.

If there's any way I can assist, I'd like to offer a few insights based on my experimentation:

• In my tests, OpenBLAS consistently outperforms noblas, particularly for prompts exceeding 256 tokens.
• The tip regarding disabling mmap on Android devices is a game-changer. I hadn't been aware of it previously, and it substantially accelerates prompt processing. I strongly advocate for emphasizing this point in the README.

Here's hoping that Vulkan proves to be a more robust solution than OpenGL.

Did your CLBLAST experience involve running corresponding tunners to achive speed for your device ?
Just for reference that in my experience with CLBLAST nonsensical infirence was fixed when I run and applied tunners. Hovewer on low end android device inference speed was the same as on cpu only without any loss in output.

[gustrd]

Hey everyone,
As the original author of these README instructions, I have to admit that I now see how they might cause more confusion than clarity.
Just to clarify for future users: I've personally found CLBlast to be quite effective when used with llama.cpp, especially for certain model families like StableLM and OpenLlama (provided you're not offloading layers). In my experience, it has boosted prompt processing speed by roughly 40%.
However, it's important to note that while CLBlast does offer significant speed improvements, it's plagued by bugs. For many model families, or even within the aforementioned subsets when offloading layers, it tends to produce nonsensical output. This is disappointing, considering the untapped potential of the GPUs nestled within our smartphones.
If there's any way I can assist, I'd like to offer a few insights based on my experimentation:

• In my tests, OpenBLAS consistently outperforms noblas, particularly for prompts exceeding 256 tokens.
• The tip regarding disabling mmap on Android devices is a game-changer. I hadn't been aware of it previously, and it substantially accelerates prompt processing. I strongly advocate for emphasizing this point in the README.

Here's hoping that Vulkan proves to be a more robust solution than OpenGL.

Did your CLBLAST experience involve running corresponding tunners to achive speed for your device ? Just for reference that in my experience with CLBLAST nonsensical infirence was fixed when I run and applied tunners. Hovewer on low end android device inference speed was the same as on cpu only without any loss in output.

No, I have not tried the tunners yet. Good idea, it's a nice experiment to do. Thanks for the idea!

[shibe2]

For many model families, or even within the aforementioned subsets when offloading layers, it tends to produce nonsensical output.

Is this specific to Android builds or can be reproduced on PC too?

[gustrd]

For many model families, or even within the aforementioned subsets when offloading layers, it tends to produce nonsensical output.

Is this specific to Android builds or can be reproduced on PC too?

As far as I know, it only happens during Android builds. All my tests were conducted with Adreno GPUs from Snapdragon.