llama-cli: Could not load model: InvalidMagic { path: ... }

[mguinhos]

Model sucessfully runs on llama.cpp but not in llama-rs

Command:

cargo run --release -- -m C:\Users\Usuário\Downloads\LLaMA\7B\ggml-model-q4_0.bin -p "Tell me how cool the Rust programming language is:"

PS C:\Users\Usuário\Desktop\llama-rs> cargo run --release -- -m C:\Users\Usuário\Downloads\LLaMA\7B\ggml-model-q4_0.bin -p "Tell me how cool the Rust programming language is:"
    Finished release [optimized] target(s) in 2.83s
     Running `target\release\llama-cli.exe -m C:\Users\Usuário\Downloads\LLaMA\7B\ggml-model-q4_0.bin -p "Tell me how cool the Rust programming language is:"`
thread 'main' panicked at 'Could not load model: InvalidMagic { path: "C:\\Users\\Usuário\\Downloads\\LLaMA\\7B\\ggml-model-q4_0.bin" }', llama-cli\src\main.rs:147:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\release\llama-cli.exe -m C:\Users\Usuário\Downloads\LLaMA\7B\ggml-model-q4_0.bin -p "Tell me how cool the Rust programming language is:"` (exit code: 101)

[philpax]

ggml-org/llama.cpp#252 changed the model format, and we're not compatible with it yet. Thanks for spotting this - we'll need to expedite the fix.

In the meantime, you can re-quantize the model with a version of llama.cpp that predates that, or find a quantized model floating around the internet from before then.

[mguinhos]

Humm... thanks i will try to re-quantize the model to a previous version!

[mguinhos]

ggerganov/llama.cpp#252 changed the model format, and we're not compatible with it yet. Thanks for spotting this - we'll need to expedite the fix.

In the meantime, you can re-quantize the model with a version of llama.cpp that predates that, or find a quantized model floating around the internet from before then.

Got it! Tried with the previous alpaca version that i had!

[philpax]

Great! We'll leave this issue open as a reminder that we'll need to update to handle the new format.

[mguinhos]

Changing the code a bit is sufficient for running the new versioned file formats.

  #[error("file is pre-versioned, generate another please! at {path:?}")]
  PreVersioned { path: PathBuf },
  #[error("invalid magic number for {path:?}")]
  InvalidMagic { path: PathBuf },
  #[error("invalid version number for {path:?}")]
  InvalidVersion { path: PathBuf },

...

// Verify magic
{
    let magic = read_i32(&mut reader)?;
    if magic == 0x67676d6c {
        return Err(LoadError::PreVersioned {
            path: main_path.to_owned(),
        });
    }
    
    if magic != 0x67676d66 {
        return Err(LoadError::InvalidMagic {
            path: main_path.to_owned(),
        });
    }
}

// Verify the version
{
    let format_version = read_i32(&mut reader)?;
    if format_version != 1 {
        return Err(LoadError::InvalidVersion {
            path: main_path.to_owned(),
        });
    }
}

...

// Load vocabulary
let mut vocab = Vocabulary::default();
for i in 0..hparams.n_vocab {
    let len = read_i32(&mut reader)?;
    if let Ok(word) = read_string(&mut reader, len as usize) {
        vocab.mapping.push(word);
        
    } else {
        load_progress_callback(LoadProgress::BadToken {
            index: i.try_into()?,
        });
        vocab.mapping.push("�".to_string());
    }

    let score: f32 = read_i32(&mut reader)? as f32;
    vocab.score.push(score);
}

It works without issues.
But i dont know if its sufficiente, nothing panicked and it did the inference.

[mguinhos]

I think that the change in the binary format was just the inclusion of the version number, and the score in the load vocabulary. but i am not sure.

After fixing this bug, and adding score: Vec<f32> to the Vocabulary struct, the 7B model works, but 65B does not. It crashes with an allocation error in the ggml library.

[mguinhos]

Related pull request: #61

[RoyVorster]

@mguinhos thanks for referencing. Hadn't even seen the issue. Feel free to modify the PR. Was just running llama-rs for the first time and running into this issue, figured it'd be best to share the small fixes.

[RoyVorster]

Can probably close this issue now?