Skip to content

KernelMemory update with adding the use of already loaded model #630

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 29, 2024
Merged

KernelMemory update with adding the use of already loaded model #630

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
127c3ed
KernelMemory update with adding the use of already loaded model
zsogitbe Mar 28, 2024
91e5a3f
Code optimization
zsogitbe Mar 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 13 additions & 6 deletions LLama.KernelMemory/BuilderExtensions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,10 @@ public static IKernelMemoryBuilder WithLLamaSharpTextGeneration(this IKernelMemo
/// </summary>
/// <param name="builder">The KernelMemoryBuilder instance.</param>
/// <param name="config">The LLamaSharpConfig instance.</param>
/// <returns>The KernelMemoryBuilder instance with LLamaSharpTextEmbeddingGeneration and LLamaSharpTextGeneration added.</returns>
public static IKernelMemoryBuilder WithLLamaSharpDefaults(this IKernelMemoryBuilder builder, LLamaSharpConfig config)
/// <param name="weights"></param>
/// <param name="context"></param>
/// <returns>The KernelMemoryBuilder instance with LLamaSharpTextEmbeddingGeneration and LLamaSharpTextGeneration added.</returns>
public static IKernelMemoryBuilder WithLLamaSharpDefaults(this IKernelMemoryBuilder builder, LLamaSharpConfig config, LLamaWeights? weights=null, LLamaContext? context=null)
{
var parameters = new ModelParams(config.ModelPath)
{
Expand All @@ -84,15 +86,20 @@ public static IKernelMemoryBuilder WithLLamaSharpDefaults(this IKernelMemoryBuil
GpuLayerCount = config?.GpuLayerCount ?? 20,
EmbeddingMode = true,
MainGpu = config?.MainGpu ?? 0,
SplitMode = config?.SplitMode ?? GPUSplitMode.None
SplitMode = config?.SplitMode ?? GPUSplitMode.None,
};
var weights = LLamaWeights.LoadFromFile(parameters);
var context = weights.CreateContext(parameters);

if (weights == null)
{
weights = LLamaWeights.LoadFromFile(parameters);
context = weights.CreateContext(parameters);
}

var executor = new StatelessExecutor(weights, parameters);
var embedder = new LLamaEmbedder(weights, parameters);
builder.WithLLamaSharpTextEmbeddingGeneration(new LLamaSharpTextEmbeddingGenerator(embedder));
builder.WithLLamaSharpTextGeneration(new LlamaSharpTextGenerator(weights, context, executor, config?.DefaultInferenceParams));
return builder;
}
}
}
}