What is directional stimulus prompting (DSP)?

Directional stimulus prompting (DSP) is a new prompting methodology in natural language processing (NLP) wherein a model is presented with a directive or structured stimulus to generate wanted outputs.

Unlike standard prompting such as one-shot, zero-shot or few-shot prompting, this approach distinguishes itself by giving direct control over the model's output by establishing criteria or providing instruction. In this approach, a guiding stimulus acts as a control mechanism of the model's generative process along lines defined by a certain criterion.

Directive stimulus prompting (DSP) is useful when a task calls for a specific set of responses, very much context-sensitive, but still without labeled data.

For instance, in the case of summarization tasks, where retaining essential information is crucial, DSP provides a guiding stimulus that nudges the model to produce in a specific way. This leads to the overall generation of more accurate and contextually appropriate summaries.¹

Large language models (LLMs) such as GPT-3, 4 and PaLM are commonly referred to as "black box" models because users do not have access to their internals, such as parameters, tuning methods or decision-making processes.

Such interaction is essentially through text prompts that use application programming interface (API) calls as the major input and output mechanisms. While these models are quite excellent, their capability to produce precise task-specific outputs is often highly contingent on prompt quality.^{2, 3}

With this, prompt engineering to design targeted prompts to steer model behavior is relevant. Both manual and automated approaches to prompt engineering have yielded notable success. However, they do not come without bitter pills, especially for those tasks that call for strong control or much instance-specific output.

For example, tasks such as summarization or dialogue generation require the model to follow target behaviors systematically, such as including key details or adhering to a strict reasoning pattern or prescribed stylistic guidelines. Conventional techniques are often not enough to guarantee consistent compliance with these nuanced requirements.

Directional stimulus prompting (DSP) comes to fill this gap. DSP is a small auxiliary policy model and generates instance-specific directional stimulus prompts that guide the LLM toward its decisions.

The prompts issued serve a specific context for each instance and are seen to coax the LLM to yield more aligned and desirable outputs. By plugging DSP into the process, users have a powerful tool to correct the behavior of black box LLMs to greater consistency, relevance and accuracy in work that needs precision.¹

Training the policy model with supervised fine-tuning (SFT)

The process of training the policy model begins with supervised fine-tuning (SFT) on a pretrained model such as T5, GPT-2 or any other suitable LLM. The key idea is to fine-tune a smaller policy model on training data that generates directional stimuli rather than directly modifying the LLM.

This process is efficient because fine-tuning a smaller, task-specific policy model avoids the challenges and computational costs associated with training large, complex models directly.

To train this policy model, a small labeled dataset is created, where each input is paired with a pseudostimulus. These pseudo stimuli are designed to guide the LLM's responses in the wanted direction based on the task at hand.

For instance, in a summarization task, the pseudostimulus could consist of keywords or phrases drawn from a reference summary. Similarly, for dialogue generation tasks, dialogue acts such as requests, questions or statements can be used as pseudo stimuli.

These stimuli serve as signals that the policy model uses to generate task-specific inputs that effectively direct the LLM's output toward the target behavior.

The labeled dataset used for SFT might be relatively small, as the focus is on providing the policy language model with the necessary knowledge to generate stimuli, not training a massive LLM from scratch. This makes SFT a resource-efficient way to bootstrap the policy model with foundational knowledge about the task-specific requirements.⁴

Refinement through reinforcement learning (RL)

After the initial fine-tuning with SFT, the policy model is optimized through reinforcement learning (RL). RL enables the policy model to explore and refine its ability to generate stimuli that lead to higher-quality LLM outputs. The core idea in this phase is to use a reward function to evaluate the effectiveness of the generated stimuli.

For example, in summarization tasks, the reward function could be based on metrics such as ROUGE or BLEU scores, which measure the quality of the generated summary in comparison to refer to. 

By focusing on training the policy model instead of the LLM directly, DSP overcomes the challenges associated with fine-tuning black box models, leading to a more efficient and scalable method. 

Figure 1: Architecture of DSP framework 

Directional stimulus prompting has notable advantages and some challenges, making it an intriguing yet intricate technique. Here’s a closer examination of its merits and demerits.⁵

Pros:

Targeted attention mechanism: The targeted attention mechanism in DSP emphasizes relevant tokens or information, enhancing accuracy and efficiency by concentrating processing on essential components.

Optimized resource usage: By concentrating on pertinent stimuli, directional stimulus prompting reduces dataset requirements, resulting in faster processing times and lower computational costs.

Enhanced precision: By isolating and emphasizing the most relevant input tokens, directional stimulus prompting boosts the accuracy of language model responses and interpretations.

Adaptability: This approach can be customized for various language tasks, ranging from text generation to sentiment analysis, offering versatility across different natural language processing applications.

Cons:

Reliance on accurate cues: The success of directional stimulus prompting heavily relies on precise stimuli, which can be challenging to achieve in complex or noisy environments. If the context or stimuli undergo significant changes, the method's effectiveness might diminish, resulting in reduced reliability.

Configuration complexity: Setting up directional stimuli needs careful design and calibration, which can make the initial configuration process more complicated.

Limited generalization: Its capacity to generalize across different signal types or unexpected input variations is limited, restricting its applicability in wider contexts.

Directive stimulus prompting (DSP) shows great potential across various NLP tasks, effectively guiding models to enhance their performance.

Summarization: DSP is used to create wanted summaries that align more closely with reference summaries. In an experimental result, using a small dataset of just 4,000 samples from the CNN/Daily Mail dataset, DSP improved benchmark performances such as ROUGE and BLEU or other measures including human preferences scores by 4–13%, surpassing some fully supervised models.⁶

Dialogue response generation: In task-oriented dialogue generation, DSP assisted ChatGPT in producing more accurate and relevant responses. For example, with only 80 dialogues from the MultiWOZ dataset, DSP achieved a performance boost of 41.4%, outpacing several state-of-the-art models (such as ChatGPT, Codex and InstructGPT) trained on larger datasets.⁷

Chain-of-thought reasoning: DSP also enhances chain-of-thought reasoning by generating instance-specific prompts that outperformed human-designed and automatically generated task-specific prompts, leading to improved reasoning accuracy. These examples illustrate how DSP can offer targeted guidance, enhancing model performance across a range of NLP applications.⁸