Skip to content

Conversation

@luismanez
Copy link
Contributor

Added a new Handler that allows the deletion of the generated "physical" (disk, blob...) files after a document has been imported.

Motivation and Context (Why the change? What's the scenario?)

This should address this question: https://github.com/microsoft/kernel-memory/issues/107

High level description (Approach, Design)

This new handler as not been added to any default pipeline, so nothing changes for current clients. However, the Handler is not present, so it can be passed as step when calling any of the current ```ImportDocumentAsync````
i.e:

await memory.ImportDocumentAsync(
    "sample-Wikipedia-Moon.txt",
    steps: new[] { Constants.DeleteDocumentGeneratedFilesPipelineStepName });

@dluc this solution would give us the functionality we need, as would avoid any compliance issue with having enterprise docs out of M365. It's not a perfect solution, and I'm happy to make any changes if providing guidance.
Note: There's still the pipeline_status.json file, but that's ok for us, and can be addressed with a container policy.

Thanks.

@dluc dluc self-requested a review December 1, 2023 00:06
@dluc dluc force-pushed the 107-delete-document-generated-files branch from 171d428 to 06cb8c7 Compare December 6, 2023 04:36
@dluc
Copy link
Collaborator

dluc commented Dec 6, 2023

looking into it, stay tuned :-)

@dluc dluc force-pushed the 107-delete-document-generated-files branch from ed6d2be to 1b1efc0 Compare December 14, 2023 02:25
@dluc dluc merged commit f6daebd into microsoft:main Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants