Skip to content

Slightly Safer Quantize Params #698

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 26, 2024
Merged

Slightly Safer Quantize Params #698

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions LLama/LLamaQuantizer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,12 @@ public static bool Quantize(
quantizeParams.nthread = nthread;
quantizeParams.allow_requantize = allowRequantize;
quantizeParams.quantize_output_tensor = quantizeOutputTensor;
//todo: fill in other quantize params fields.

unsafe
{
return NativeApi.llama_model_quantize(srcFileName, dstFilename, &quantizeParams) == 0;
}
// todo: fill in other quantize params fields.
// This method could probably do with a redesign - passing in a config object (maybe directly
// expose `LLamaModelQuantizeParams`) instead of an ever growing list of method parameters!

return NativeApi.llama_model_quantize(srcFileName, dstFilename, ref quantizeParams) == 0;
}

/// <summary>
Expand Down
2 changes: 1 addition & 1 deletion LLama/Native/NativeApi.Quantize.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,6 @@ public static partial class NativeApi
/// <param name="param"></param>
/// <returns>Returns 0 on success</returns>
[DllImport(libraryName, CallingConvention = CallingConvention.Cdecl)]
public static extern unsafe uint llama_model_quantize(string fname_inp, string fname_out, LLamaModelQuantizeParams* param);
public static extern uint llama_model_quantize(string fname_inp, string fname_out, ref LLamaModelQuantizeParams param);
}
}