Support for Mistral 7B

[TejaGollapudi]

Hi, will vLLM need additional changes to make Mistral 7B work? They use sliding window attention which I think would require small modification on the vLLM side.

mistralai/Mistral-7B-Instruct-v0.1

Thank you

[WoosukKwon]

Hi @TejaGollapudi Thanks for the question. Yes, #1196 will add support for the model.

[Naveenpandey27]

how to use vLLM library with mistral model to generate summary. i am using this code to generate summary for multiple chunk

Function to generate summary

def generate_summary(chunk, user_question):
try:
designation = 'Executive'
prompt_template = """
Task: Craft a succinct and polished summary report by leveraging the user's prompt and the provided data, ensuring coherence, completeness of sentences, and logical flow of information.

    User Prompt:
    ```{user_prompt}```

    Query Data:
    ```{query_data}```

    Instructions:
    Craft a concise and insightful summary tailored for a {designation} in our organization.
    Clearly indicate the urgency or priority levels for addressing the identified issues in the data.
    The summary should illuminate key points or insights derived from the data and should not exceed 2000 words.

    Helpful answer:
    """

    custom_prompt = prompt_template.format(user_prompt=user_question, query_data=chunk, designation=designation)
    start_time = time.time()
    
    # Generate the summary using Mistral with the provided configuration
    sampling_params = SamplingParams(temperature=0.01, top_p=0.95)
    outputs = llm.generate([custom_prompt], sampling_params)
    
    generated_texts = [output.outputs[0].text for output in outputs]  # Extract text from each output
    
    end_time = time.time()
    elapsed_time = end_time - start_time
    print(f"Summary generated in {elapsed_time} seconds")
    
    # Combine generated texts into a single string
    generated_summary = "\n".join(generated_texts)
    
    # Count tokens in the generated summary
    token_count = count_token(generated_summary)
    print("Summary generated")
    print('token count : ', token_count)
    
    return generated_summary, token_count
except Exception as e:
    print(f"Error generating summary: {e}")
    return None, 0
    
    it is give me this output:
    
    Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.62s/it]

Summary generated in 1.684791088104248 seconds
Summary generated
token count : 1
Summary for Sublist 1: