PromptSense Introduces Sentiment‑Aware Prompts for Safer Mental‑Health Chatbots

Current large‑language‑model (LLM) chatbots used for mental‑health support often fall short on empathy, context, and safety, which can make conversations ineffective or even risky. A team of researchers led by Sugi Lee proposes PromptSense, a framework that injects sentiment and depression analysis into the prompting stage, dynamically tailoring each response to the user’s emotional state and risk level.

How does PromptSense work?

PromptSense adds two preprocessing layers before the LLM generates a reply. First, the user’s input is run through a sentiment‑detection module and a depression‑risk classifier. The outputs feed a rule‑based filter that flags high‑risk language (e.g., self‑harm cues) and adjusts the prompt accordingly. The filtered information is then combined with a base instruction set to produce a dynamic prompt that guides the LLM toward an empathetic, context‑aware, and safety‑conscious answer. This differs from static prompting, where the same instruction set is used regardless of the user’s emotional tone.

What were the experimental results?

The authors evaluated four model strategies (including vanilla prompting) and two PromptSense‑engineered variants. They measured qualitative metrics such as empathy, helpfulness, clarity, relevance, and safety, as well as toxicity using the Perspective API. According to the arXiv paper, the PromptSense variants outperformed the strongest baseline, DSFT, by 1.8 % and RLAIF by 1.1 % on the composite quality score. The improvement was most pronounced in safety and empathy categories, where the dynamic prompts reduced toxic or dismissive language.

Why does dynamic prompting matter for mental‑health bots?

Static prompts cannot react to the nuanced shifts in a user’s mood that often occur within a single conversation. By feeding real‑time sentiment signals into the prompt, PromptSense enables the LLM to:

1. Adjust tone – adopt a calmer or more supportive voice when negative sentiment is detected.
2. Prioritize safety – insert warnings or escalation steps if depression risk crosses a threshold.
3. Maintain relevance – reference earlier emotional cues, improving continuity across turns.

These capabilities align with clinical best practices that stress active listening and risk assessment, making the chatbot a more trustworthy first‑line support tool.

How can developers experiment with PromptSense‑style prompting?

While the paper’s code is not bundled with Grafy, creators can prototype similar pipelines using Grafy’s Chat app, which lets you chain multiple models in a single session. For example, you could route a user’s message through a sentiment‑analysis model (available via OpenRouter or Vertex AI), feed the result into a rule‑based script, and then construct a custom prompt for a conversational LLM. The Graph app can visualize the branching workflow, showing how each analysis step influences the final prompt. For developers interested in embedding such a bot on a website, the Bots app provides a straightforward way to publish a support bot that runs the same preprocessing chain server‑side.

Where does PromptSense fit into the broader AI safety landscape?

Prompt engineering has become a practical lever for steering LLM behavior without retraining. PromptSense demonstrates that coupling sentiment analysis with rule‑based filters can produce measurable safety gains in a high‑stakes domain. The approach complements other safety techniques like RLHF (reinforcement learning from human feedback) and supervised fine‑tuning, offering a lightweight, modular alternative that can be added to existing chatbot deployments.

How to explore the research with Grafy tools?

If you want to read the full paper and experiment with its methods, start with Grafy’s Reader app (/reader). Upload the PDF, ask the AI to summarize sections, or query specific evaluation tables. Then switch to Chat (/chat) to test your own dynamic prompts against a variety of LLMs side by side. Finally, use Graph (/graph) to map the prompt‑construction logic and iterate on rule definitions.

Frequently asked questions

Does PromptSense replace existing safety layers like RLHF?

PromptSense is designed to complement, not replace, other safety mechanisms. It adds a preprocessing step that can work alongside RLHF or fine‑tuned models, providing an extra guardrail that reacts to real‑time sentiment.

Can the sentiment analysis be swapped for a different emotion detector?

Yes. The framework is modular; any model that outputs a sentiment or risk score can feed the rule‑based filter. Developers can experiment with open‑source classifiers or commercial APIs depending on accuracy and latency requirements.

Is PromptSense ready for production use in clinical settings?

The paper reports promising early results, but it remains a research prototype. Clinical deployment would require extensive validation, regulatory compliance, and possibly integration with professional oversight tools before it could be considered safe for real patients.

---

Source: arXiv AI — PromptSense: Sentiment-Aware Mental Health Chatbot with Dynamic Prompting