User Guide

Home page

Overview

The RAGtime home page serves as the main dashboard for managing experiments and validation sets. It provides an overview of your current validation sets and experiment results, along with quick access to key functions.

Layout

The page is divided into two main sections:

1. Validation sets
2. All experiment results

A sidebar on the left provides navigation to other parts of the application.

Features

Validation Sets Table

This table displays all available validation sets with the following information:

• Name: The identifier for the validation set
• Date: Creation date of the set
• Questions: Number of questions in the set
• Réponses (Responses): Number of responses
• Facts: Number of facts in the set
• Chunks: Number of chunks

A filter field above the table allows you to search for specific validation sets by name.

Note:

• To modify a validation set:
  1. Click on the validation set's name.
  2. This will display all the questions along with their associated information.
  3. You can then edit the questions and other details as needed
• To delete a validation set:
  1. Check the box next to the validation set you want to delete.
  2. A delete icon (trash bin) will appear.
  3. Click on the delete icon to remove the validation set.

Remember, deleting a validation set is permanent and cannot be undone, so please be certain before proceeding with deletion.

All Experiment Results Table

This table shows the results of all experiments you've run, including:

• Name: Identifier for the experiment
• Date: When the experiment was conducted
• Models: The AI model(s) used in the experiment
• Questions: Number of questions processed
• Facts: Number of facts involved
• Chunks: Number of chunks processed
• Chunk Eval: Indicates whether chunk evaluation was performed (Yes/No)
• Retriever: The retrieval method used

A filter field is also available to search for specific experiments by name.

Note :

• To see the experiment’s results, click on the experiment name.
• To delete an experiment:
  1. Check the box next to the experiment you want to delete.
  2. A delete icon (trash bin) will appear.
  3. Click on the delete icon to remove the validation set.

Key Actions

Start Experiment

Click the "START EXPERIMENT" button in the top-right corner or in the sidebar to initiate a new experiment.

Create Validation Set

Use the "CREATE VALIDATION SET" button to generate a new validation set for your experiments.

Navigation

The sidebar menu provides quick access to:

• Dashboard (current page)
• Start experiment
• Create validation set
• Profile settings
• Logout

Profile Settings

Access the Profile settings to set up your API key and configure environment variables necessary for your experiments.

Additional Information

• The application's GitHub page can be accessed via the link at the bottom of the sidebar.

Creating and modifying a validation set

Overview

The "Create Validation Set" page in Ragtime allows users to create and modify validation sets, which are essential for evaluating RAG systems and LLM performance. This page covers the process of creating, importing, and modifying validation sets.

Accessing the Create Validation Set Page

• Click on "CREATE VALIDATION SET" in the sidebar menu or on the homepage.

Creating a New Validation Set

There are two methods to create a validation set:

• Import a JSON file

• Enter questions manually

Method 1: Importing a JSON File

1. Click on the "Import JSON" button.
2. Select a JSON file containing Q&A pairs in the Ragtime format.
3. The questions and answers will be automatically populated.

Method 2: Entering Questions Manually

1. Click on the "Add New Question" button.
2. Type your question in the provided field.
3. Add answers by clicking "Show Answers" then "Add Answer", or generate answers using available models.

Managing Questions and Answers

• View Answers: Click "Show Answers" for each question.
• Edit: Use the "Edit" button to modify questions or answers.
• Delete: Remove questions or answers using the "Delete" button.

Generating Facts

Facts are key ideas extracted from answers, used for evaluation purposes.

Before setting the API keys

After setting the API keys

1. Select a fact generation model from the dropdown (GPT-4 recommended).
   • Ensure you've set up the necessary API keys in Profile Settings.
2. Generate facts for all questions: Click the "Generate Facts" button at the top.
3. Generate facts for a specific question: Click "Show Facts", then "Generate Facts" for that question.

Note: Each question must have exactly one answer before generating facts, otherwise, it will not generate the facts.`

Modifying the Validation Set

• You can always modify the validation set by clicking its name in the Home page and following the same steps as those of the creation process.
• You can edit questions, answers, and regenerate facts at any time.
• For model-generated answers, details such as the model used, cost, and generation date are displayed.

Saving the Validation Set

1. Give your validation set a name.
2. Click the "Save" button.

The new validation set will appear on the homepage in the Validation Sets table and will be available for use in experiments.

Additional Notes

• Ensure API keys are set up in Profile Settings for full functionality.
• You can switch between importing JSON and manual entry as needed.
• The interface allows for easy viewing and management of questions, answers, and facts.

Profile settings

Overview

The API Key Settings page in RAGtime allows users to manage API keys and variables necessary for experiments. This interface is for setting up access to various language models (LLMs) and custom retrievers.

Accessing API Key Settings

• Navigate to the API Key Settings page through the "Profile settings" option in the sidebar menu.

Features

Viewing Existing API Keys

• Existing API keys are displayed at the top of the page.
• Each key shows its type (e.g., OPENAI_API_KEY) and a partially hidden key value.
• Options to EDIT or DELETE each key are available.

Adding New API Keys

Standard API Keys

1. Click on the "Select API Key Type" dropdown.

   

2. Choose from predefined options:

   • OpenAI API Key

   • Mistral API Key

     

3. Enter the API key value in the "API Key Value" field.

4. Click "ADD API KEY".

   

Custom API Keys

1. Select "Custom API Key" from the dropdown.

   

2. Enter the custom key name in the "Custom Key Name" field.

   

   • Note: The name should exactly match how it will be called by the system (e.g., GOOGLE_API_KEY for Google models).

3. Enter the API key value in the "API Key Value" field.

4. Click "ADD API KEY".

   

Managing API Keys

• Edit: Click the EDIT button next to a key to modify its value.
• Delete: Use the DELETE button to remove a key.

Saving Changes

• After making any modifications, click the "SAVE CHANGES" button at the bottom of the page to persist your updates.

• You will receive a pop-up notification confirming that your new changes have been successfully saved.

  

Security Considerations

• API keys are partially hidden for security reasons.
• Exercise caution when entering or editing API keys to prevent errors.

Remember to keep your API keys confidential and never share them publicly. The RAGtime application uses these keys locally and nothing is saved elsewhere so you can stay assured