\n",
+ " Patient Satisfaction Analytics using Large Language Models and Enteprise Vector Store\n",
+ "
\n",
+ " ![\"Teradata\"](\"https://storage.googleapis.com/clearscape_analytics_demo_data/DEMO_Logo/teradata.svg\"){kind=logo}
\n",
+ "
\n",
+ " Patient Satisfaction Analytics using Large Language Models and Enteprise Vector Store\n",
+ "
\n",
+ " \n",
+ "
Introduction:
\n", + "\n", + "What is Patient Satisfaction?\n", + "Patient satisfaction measures how happy a patient is with their healthcare experience, reflecting whether their expectations were met. It's a key indicator of quality and impacts various aspects of healthcare, including clinical outcomes, patient retention, and legal claims. Effective communication, efficient processes, and positive interactions all contribute to higher patient satisfaction.
\n", + "\n", + "Why is Patient Satisfaction so important?\n", + "\n", + "Patient satisfaction is crucial in healthcare because it's a key indicator of both the quality of care and the overall patient experience. Satisfied patients are more likely to adhere to treatment plans, have better health outcomes, and recommend their healthcare providers. Additionally, patient satisfaction can influence factors like patient retention, loyalty, and the financial well-being of healthcare organizations. And provider reimbursement in healthcare is increasingly linked to patient satisfaction through higher reimbursement rates.
\n", + "\n", + "Challenges with operationalizing AI
\n", + "\n", + "Healthcare organizations of all types are struggling with how to best leverage the promise and the power of new Large Language Model and other AI models, tools, and techniques. Some of the top challenges include:
\n", + "\n", + "Patient Sentiment Analysis\n", + "
\n", + " \n", + " |  |
Semantic Search for Targeted Patient Outreach\n", + "
\n", + " \n", + " |  |
Please restart the kernel after executing the above cell to include/update these libraries into memory for this kernel. The simplest way to restart the Kernel is by typing zero zero: 0 0 and then clicking Restart.
\n", + "2. Connect to VantageCloud Lake
\n", + "Connect to VantageCloud using `create_context` from the teradataml Python library. Input your connection details, including the host, username, password and Analytic Compute Group name.
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"Checking if this environment is ready to connect to VantageCloud Lake...\")\n", + "\n", + "if os.path.exists(\"/home/jovyan/JupyterLabRoot/VantageCloud_Lake/.config/.env\"):\n", + " print(\"Your environment parameter file exist. Please proceed with this use case.\")\n", + " # Load all the variables from the .env file into a dictionary\n", + " env_vars = dotenv_values(\"/home/jovyan/JupyterLabRoot/VantageCloud_Lake/.config/.env\")\n", + " # Create the Context\n", + " eng = create_context(host=env_vars.get(\"host\"), username=env_vars.get(\"username\"), password=env_vars.get(\"my_variable\"))\n", + " execute_sql('''SET query_band='DEMO=Patient_Satisfaction_Analyzer_EVS_VCL.ipynb;' UPDATE FOR SESSION;''')\n", + " print(\"Connected to VantageCloud Lake with:\", eng)\n", + "else:\n", + " print(\"Your environment has not been prepared for connecting to VantageCloud Lake.\")\n", + " print(\"Please contact the support team.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "3. Authenticate into User Environment Service (UES) for Container Management
\n", + "\n", + "UES authentication is required to create and manage the Python or R environments that we will be creating. A VantageCloud Lake user can easily create the authentication objects using the Console in a VantageCloud Lake environment. The step to create these authentication objects has already been performed for you.\n", + "
\n", + "\n", + " \n", + "
4. Sentiment Analysis
\n", + "\n", + "Extract the emotion (anger, joy, sadness, love) and strength of that emotion (0-1) based on analysis of unstructured call center transcripts. Inspect the distribution of how strongly patients are feeling these emotions - are we reaching a point where we need to change business practices or improve outreach?
\n", + "\n", + "\n",
+ "
\n", + " \n", + " | |
5. Load the Data
\n", + "\n", + "We have provided data for this demo in the lake environment. The data is available in the database \"DEMO_SalesForecasting\". Your user should have read access to the database. In case of any issues please write a mail to the support group (\"SC230208@teradata.com\").
\n", + "\n", + "**Note: The tables are available in DEMO_PatientSatisfaction_DB database and we have created views in DEMO_PatientSatisfaction database which are used in the cells below.
\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "tdf_transcripts = DataFrame(in_schema(\"DEMO_PatientSatisfaction\",\"patient_feedback\"))\n", + "tdf_transcripts" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "6. Sentiment Extraction
\n", + "\n", + "\n", + "6.1 Set up Sentiment Extraction Model
\n", + "\n", + "The teradatagenai python library can both connect to cloud-based LLM services as well as instantiate private models running at scale on local GPU compute. In this case we will use the AWS Bedrock implementation of Anthropic claude-v2 for a balance of price and performance. Enter a valid key/secret/region that has this model enabled, or use an auth object.
\n", + "\n", + "Configuring AWS CLI\n", + "The following cell will prompt us for the following information:
\n", + "6.2 Extract Patient Sentiment
\n", + "\n", + "A simple method call will extract the sentiment for patient comments in-database using the desired LLM and CSP provider.
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "tdf_transcripts_5=tdf_transcripts.iloc[0:100]\n", + "tdf_transcripts_5.shape" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "%%time\n", + "tdf_sentiment = obj.analyze_sentiment(column='Transcript', data=tdf_transcripts_5)\n", + "tdf_sentiment" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "6.3 Execute simple descriptive analytics
\n", + "\n", + "Now, we can begin asking increasingly more complex questions of the data; what's the distribution of sentiment? Or - does any individual patient have increasing feelings of anger, fear, or sadness? Native, in-database ClearScape Analytics functions can answer these questions at extreme speed and scale.
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "CategoricalSummary(data = tdf_sentiment, target_columns = 'Sentiment').result" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "df = CategoricalSummary(data = tdf_sentiment, target_columns = 'Sentiment').result.to_pandas()\n", + "fig = px.bar(df, x = 'DistinctValue', y = 'DistinctValueCount')\n", + "fig.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "7. Develop targeted patient outreach with Semantic Search
\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Match our patients with targeted outreach programs based on the Semantic Meaning of their comments history. This can allow our organization to provide tailored programs or assistance based on the intent of the commentary, allowing for more accurate targeting.
\n", + "\n", + "\n",
+ "
\n", + " \n", + " | |
7.1 Enterprise Vector Store
\n", + "\n", + "Vector Embedding is a numerical representation of data that captures semantic relationships and similarities, making it possible to perform mathematical operations and comparisons on the data for various tasks like text analysis and recommendation systems.
\n", + "\n", + "Generative Responses
\n", + "\n", + "Use Natural Language to inspect the patient complaints data, and leverage Private LLMs to generate rich, actionable responses.
\n", + "\n", + "Hyper-personalize the appropriate action based on AI-based semantic analysis and search.
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "vs = VectorStore(name = 'Transcript_Analysis')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# if desired, destroy and recreate the vector store - or to use an existing one\n", + "# skip to the \"similarity_search\" code below\n", + "# vs.destroy()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "# Python\n", + "vs.create(search_algorithm = 'VECTORDISTANCE',\n", + " object_names = DataFrame(in_schema(\"DEMO_PatientSatisfaction\",\"patient_detail_feedback\")),\n", + " key_columns = ['Call_ID', 'first_name', 'last_name'],\n", + " data_columns = ['Transcript'],\n", + " vector_column = 'Embedding')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "vs.status()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "vs.similarity_search(question = 'top 10 most angry unique callers').similar_objects[['score','Call_ID','Transcript']]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "from IPython.display import Markdown\n", + "Markdown(vs.ask(question = 'the top angry customers', prompt = 'suggest a targeted outreach program'))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "7.2 Construct a list of patient care programs
\n", + "\n", + "Create a small list of topics that will represent the outreach programs we want to target. This could represent existing outreach or marketing programs we want to more accurately target.\n", + "\n", + "
Sample patient care programs
\n", + "7.3 Identify patient and program match
\n", + "\n", + "Provide these results in real-time to call center agent dashboards, content management systems, or other marketing automation applications.
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "topics_list = l\n", + "\n", + "d = widgets.Dropdown(options=topics_list)\n", + "\n", + "output = widgets.Output()\n", + "\n", + "def on_change(change):\n", + " if change['type'] == 'change' and change['name'] == 'value':\n", + " with output:\n", + " clear_output()\n", + " local_df = vs.similarity_search(question = change['new']).similar_objects[['first_name', 'last_name','Transcript']].to_pandas()\n", + " ipydisplay(local_df)\n", + " text = \" \".join(comment for comment in local_df['Transcript'])\n", + " stopwords = set(STOPWORDS)\n", + " wordcloud = WordCloud(stopwords = stopwords, width = 800, height=400, \n", + " max_words=100, min_word_length=1, \n", + " collocations = True, background_color = 'white').generate(text)\n", + " plt.figure(figsize=(10, 5))\n", + " plt.imshow(wordcloud, interpolation='gaussian')\n", + " plt.axis('off')\n", + " plt.title(f'Word Cloud for Search Topic')\n", + " plt.show()\n", + " \n", + "d.observe(on_change)\n", + "\n", + "print('Select Topic to Search:')\n", + "ipydisplay(d, output)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Conclusion - Operationalizing AI-powered analytics
\n", + "\n", + "\n", + "\n", + "The preceding demo showed two real-world applications of AI-powered analytics that leverages private data securely, to allow for deeper insight into unstructured data that can improve patient satisfaction.
\n", + "\n", + "\n", + "8. Cleanup
\n", + "\n", + "\n", + "\n", + "Call the destroy() method of the VS object to clean up the objects created during this demo.
" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "vs.destroy()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "remove_context()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.10" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/VantageCloud_Lake/UseCases/Patient_Satisfaction_Analyzer_EVS/images/Semantic_Search_AWS.png b/VantageCloud_Lake/UseCases/Patient_Satisfaction_Analyzer_EVS/images/Semantic_Search_AWS.png new file mode 100644 index 00000000..3a40c256 Binary files /dev/null and b/VantageCloud_Lake/UseCases/Patient_Satisfaction_Analyzer_EVS/images/Semantic_Search_AWS.png differ diff --git a/VantageCloud_Lake/UseCases/Patient_Satisfaction_Analyzer_EVS/images/Sentiment_Analysis_AWS.png b/VantageCloud_Lake/UseCases/Patient_Satisfaction_Analyzer_EVS/images/Sentiment_Analysis_AWS.png new file mode 100644 index 00000000..f3724957 Binary files /dev/null and b/VantageCloud_Lake/UseCases/Patient_Satisfaction_Analyzer_EVS/images/Sentiment_Analysis_AWS.png differ diff --git a/VantageCloud_Lake/UseCases/Patient_Satisfaction_Analyzer_EVS/requirements.txt b/VantageCloud_Lake/UseCases/Patient_Satisfaction_Analyzer_EVS/requirements.txt new file mode 100644 index 00000000..5a32a3ef --- /dev/null +++ b/VantageCloud_Lake/UseCases/Patient_Satisfaction_Analyzer_EVS/requirements.txt @@ -0,0 +1,3 @@ +teradataml==20.0.0.5 +teradatagenai>=20.0.0.1 +wordcloud \ No newline at end of file