Skip to content

Commit f8aa32e

Browse files
tisniktisnik
authored
Merge pull request #425 from radofuchs/LCORE_166_E2E_tests_endpoints
LCORE-166: add e2e tests for the endpoints
2 parents 45eb299 + f914569 commit f8aa32e

18 files changed

+536
-73
lines changed

‎behave.ini‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[behave]
2+
paths = tests/e2e/features

‎tests/e2e/features/authorized.feature‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Feature: Authorized endpoint API tests
2+
# TODO: fix test
3+
4+
# Background:
5+
# Given The service is started locally
6+
# And REST API service hostname is localhost
7+
# And REST API service port is 8080
8+
# And REST API service prefix is /v1
9+
10+
# Scenario: Check if the OpenAPI endpoint works as expected
11+
# Given The system is in default state
12+
# When I access endpoint "authorized" using HTTP POST method
13+
# Then The status code of the response is 200
14+
# And The body of the response has proper username
15+
16+
# Scenario: Check if LLM responds to sent question with error when not authenticated
17+
# Given The system is in default state
18+
# And I remove the auth header
19+
# When I access endpoint "authorized" using HTTP POST method
20+
# Then The status code of the response is 400
21+
# And The body of the response is the following
22+
# """
23+
# {"detail": "Unauthorized: No auth header found"}
24+
# """
25+
26+
# Scenario: Check if LLM responds to sent question with error when not authorized
27+
# Given The system is in default state
28+
# And I modify the auth header so that the user is it authorized
29+
# When I access endpoint "authorized" using HTTP POST method
30+
# Then The status code of the response is 403
31+
# And The body of the response is the following
32+
# """
33+
# {"detail": "Forbidden: User is not authorized to access this resource"}
34+
# """

‎tests/e2e/features/conversations.feature‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Feature: conversations endpoint API tests
2+
#TODO: fix test
3+
4+
# Background:
5+
# Given The service is started locally
6+
# And REST API service hostname is localhost
7+
# And REST API service port is 8080
8+
# And REST API service prefix is /v1
9+
10+
11+
# Scenario: Check if conversations endpoint finds the correct conversation when it exists
12+
# Given The system is in default state
13+
# When I access REST API endpoint "conversations" using HTTP GET method
14+
# Then The status code of the response is 200
15+
# And the proper conversation is returned
16+
17+
# Scenario: Check if conversations endpoint does not finds the conversation when it does not exists
18+
# Given The system is in default state
19+
# When I access REST API endpoint "conversations" using HTTP GET method
20+
# Then The status code of the response is 404
21+
22+
# Scenario: Check if conversations endpoint fails when conversation id is not provided
23+
# Given The system is in default state
24+
# When I access REST API endpoint "conversations" using HTTP GET method
25+
# Then The status code of the response is 422
26+
27+
# Scenario: Check if conversations endpoint fails when service is unavailable
28+
# Given The system is in default state
29+
# And the service is stopped
30+
# When I access REST API endpoint "conversations" using HTTP GET method
31+
# Then The status code of the response is 503
32+
33+
# Scenario: Check if conversations/delete endpoint finds the correct conversation when it exists
34+
# Given The system is in default state
35+
# When I access REST API endpoint "conversations/delete" using HTTP GET method
36+
# Then The status code of the response is 200
37+
# And the deleted conversation is not found
38+
39+
# Scenario: Check if conversations/delete endpoint does not finds the conversation when it does not exists
40+
# Given The system is in default state
41+
# When I access REST API endpoint "conversations/delete" using HTTP GET method
42+
# Then The status code of the response is 404
43+
44+
# Scenario: Check if conversations/delete endpoint fails when conversation id is not provided
45+
# Given The system is in default state
46+
# When I access REST API endpoint "conversations/delete" using HTTP GET method
47+
# Then The status code of the response is 422
48+
49+
# Scenario: Check if conversations/delete endpoint fails when service is unavailable
50+
# Given The system is in default state
51+
# And the service is stopped
52+
# When I access REST API endpoint "conversations/delete" using HTTP GET method
53+
# Then The status code of the response is 503
54+

‎tests/e2e/features/feedback.feature‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Feature: feedback endpoint API tests
2+
3+
4+
# Background:
5+
# Given The service is started locally
6+
# And REST API service hostname is localhost
7+
# And REST API service port is 8080
8+
# And REST API service prefix is /v1
9+
10+
11+
# Scenario: Check if feedback endpoint is working
12+
# Given The system is in default state
13+
# When I access endpoint "feedback" using HTTP POST with conversation ID conversationID
14+
# """
15+
# {
16+
# "llm_response": "bar",
17+
# "sentiment": -1,
18+
# "user_feedback": "Not satisfied with the response quality",
19+
# "user_question": "random question"
20+
# }
21+
# """
22+
# Then The status code of the response is 200
23+
# And The body of the response is the following
24+
# """
25+
# {"response": "feedback received"}
26+
# """
27+
28+
# Scenario: Check if feedback endpoint is not working when not authorized
29+
# Given The system is in default state
30+
# And I remove the auth header
31+
# When I access endpoint "feedback" using HTTP POST with conversation ID conversationID
32+
# """
33+
# {
34+
# "llm_response": "bar",
35+
# "sentiment": -1,
36+
# "user_feedback": "Not satisfied with the response quality",
37+
# "user_question": "random question"
38+
# }
39+
# """
40+
# Then The status code of the response is 400
41+
# And The body of the response is the following
42+
# """
43+
# {"response": "feedback received"}
44+
# """
45+
46+
# Scenario: Check if feedback endpoint is not working when feedback is disabled
47+
# Given The system is in default state
48+
# And I disable the feedback
49+
# When I access endpoint "feedback" using HTTP POST with conversation ID conversationID
50+
# """
51+
# {
52+
# "llm_response": "bar",
53+
# "sentiment": -1,
54+
# "user_feedback": "Not satisfied with the response quality",
55+
# "user_question": "random question"
56+
# }
57+
# """
58+
# Then The status code of the response is 403
59+
# And The body of the response is the following
60+
# """
61+
# {"response": "feedback received"}
62+
# """
63+
64+
# Scenario: Check if feedback endpoint fails with incorrect body format when conversationID is not present
65+
# Given The system is in default state
66+
# When I access endpoint "feedback" using HTTP POST method
67+
# """
68+
# {
69+
# "llm_response": "bar",
70+
# "sentiment": -1,
71+
# "user_feedback": "Not satisfied with the response quality",
72+
# "user_question": "random question"
73+
# }
74+
# """
75+
# Then The status code of the response is 422
76+
# And The body of the response is the following
77+
# """
78+
# { "type": "missing", "loc": [ "body", "conversation_id" ], "msg": "Field required", }
79+
# """
80+
81+
# Scenario: Check if feedback/status endpoint is working
82+
# Given The system is in default state
83+
# When I access REST API endpoint "feedback/status" using HTTP GET method
84+
# Then The status code of the response is 200
85+
# And The body of the response is the following
86+
# """
87+
# {"functionality": "feedback", "status": { "enabled": true}}
88+
# """
89+
90+

‎tests/e2e/features/health.feature‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Feature: Health endpoint API tests
2+
#TODO: fix test
3+
4+
# Background:
5+
# Given The service is started locally
6+
# And REST API service hostname is localhost
7+
# And REST API service port is 8080
8+
# And REST API service prefix is /v1
9+
10+
11+
# Scenario: Check if service report proper readiness state
12+
# Given The system is in default state
13+
# When I access endpoint "readiness" using HTTP GET method
14+
# Then The status code of the response is 200
15+
# And The body of the response has the following schema
16+
# """
17+
# {
18+
# "ready": "bool",
19+
# "reason": "str",
20+
# "providers": "list[str]"
21+
# }
22+
# """
23+
# And The body of the response is the following
24+
# """
25+
# {"ready": true, "reason": "All providers are healthy", "providers": []}
26+
# """
27+
28+
# Scenario: Check if service report proper readiness state when llama stack is not available
29+
# Given The system is in default state
30+
# And The llama-stack connection is disrupted
31+
# When I access endpoint "readiness" using HTTP GET method
32+
# Then The status code of the response is 503
33+
34+
# Scenario: Check if service report proper liveness state
35+
# Given The system is in default state
36+
# When I access endpoint "liveness" using HTTP GET method
37+
# Then The status code of the response is 200
38+
# And The body of the response has the following schema
39+
# """
40+
# {
41+
# "alive": "bool"
42+
# }
43+
# """
44+
# And The body of the response is the following
45+
# """
46+
# {"alive":true}
47+
# """
48+
49+
# Scenario: Check if service report proper liveness state when llama stack is not available
50+
# Given The system is in default state
51+
# And The llama-stack connection is disrupted
52+
# When I access endpoint "liveness" using HTTP GET method
53+
# Then The status code of the response is 503

‎tests/e2e/features/info.feature‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Feature: Info endpoint API tests
2+
#TODO: fix test
3+
4+
# Background:
5+
# Given The service is started locally
6+
# And REST API service hostname is localhost
7+
# And REST API service port is 8080
8+
# And REST API service prefix is /v1
9+
10+
# Scenario: Check if the OpenAPI endpoint works as expected
11+
# Given The system is in default state
12+
# When I access endpoint "openapi.json" using HTTP GET method
13+
# Then The status code of the response is 200
14+
# And The body of the response contains OpenAPI
15+
16+
# Scenario: Check if info endpoint is working
17+
# Given The system is in default state
18+
# When I access REST API endpoint "info" using HTTP GET method
19+
# Then The status code of the response is 200
20+
# And The body of the response has proper name "lightspeed_stack" and version "0.2.0"
21+
22+
# Scenario: Check if models endpoint is working
23+
# Given The system is in default state
24+
# When I access REST API endpoint "models" using HTTP GET method
25+
# Then The status code of the response is 200
26+
# And The body of the response contains gpt
27+
28+
29+
# Scenario: Check if models endpoint is working
30+
# Given The system is in default state
31+
# And The llama-stack connection is disrupted
32+
# When I access REST API endpoint "models" using HTTP GET method
33+
# Then The status code of the response is 503
34+
35+
# Scenario: Check if metrics endpoint is working
36+
# Given The system is in default state
37+
# When I access REST API endpoint "metrics" using HTTP GET method
38+
# Then The status code of the response is 200
39+
# And The body of the response has proper metrics
40+
41+
# Scenario: Check if metrics endpoint is working
42+
# Given The system is in default state
43+
# And The llama-stack connection is disrupted
44+
# When I access REST API endpoint "metrics" using HTTP GET method
45+
# Then The status code of the response is 500
46+

‎tests/e2e/features/llm_interface.feature‎

Lines changed: 0 additions & 18 deletions
This file was deleted.

‎tests/e2e/features/query.feature‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# Feature: Query endpoint API tests
2+
#TODO: fix test
3+
4+
# Background:
5+
# Given The service is started locally
6+
# And REST API service hostname is localhost
7+
# And REST API service port is 8080
8+
# And REST API service prefix is /v1
9+
10+
11+
# Scenario: Check if LLM responds to sent question
12+
# Given The system is in default state
13+
# When I use "query" to ask question "Say hello"
14+
# Then The status code of the response is 200
15+
# And The response should have proper LLM response format
16+
# And The response should contain following fragments
17+
# | Fragments in LLM response |
18+
# | Hello |
19+
20+
# Scenario: Check if LLM responds to sent question with different system prompt
21+
# Given The system is in default state
22+
# And I change the system prompt to "new system prompt"
23+
# When I use "query" to ask question "Say hello"
24+
# Then The status code of the response is 200
25+
# And The response should have proper LLM response format
26+
# And The response should contain following fragments
27+
# | Fragments in LLM response |
28+
# | Hello |
29+
30+
# Scenario: Check if LLM responds with error for malformed request
31+
# Given The system is in default state
32+
# And I modify the request body by removing the "query"
33+
# When I use "query" to ask question "Say hello"
34+
# Then The status code of the response is 422
35+
# And The body of the response is the following
36+
# """
37+
# { "type": "missing", "loc": [ "body", "system_query" ], "msg": "Field required", }
38+
# """
39+
40+
# Scenario: Check if LLM responds to sent question with error when not authenticated
41+
# Given The system is in default state
42+
# And I remove the auth header
43+
# When I use "query" to ask question "Say hello"
44+
# Then The status code of the response is 200
45+
# Then The status code of the response is 400
46+
# And The body of the response is the following
47+
# """
48+
# {"detail": "Unauthorized: No auth header found"}
49+
# """
50+
51+
# Scenario: Check if LLM responds to sent question with error when not authorized
52+
# Given The system is in default state
53+
# And I modify the auth header so that the user is it authorized
54+
# When I use "query" to ask question "Say hello"
55+
# Then The status code of the response is 403
56+
# And The body of the response is the following
57+
# """
58+
# {"detail": "Forbidden: User is not authorized to access this resource"}
59+
# """
60+

0 commit comments

Comments
 (0)