Chain-of-Thought Prompting: What the Research Showed and How to Use It in Practical Instructions

Asking for the steps can raise accuracy on multi-step problems

Chain-of-thought prompting is a way of instructing an AI model to write out its intermediate reasoning, not just the final answer. Research reported that this improved accuracy on problems that take several steps, such as math word problems.

The models in that research are not the tools available today, so the findings should not be applied to current practice as they stand. It helps to separate what the study showed from how the tools have changed.

What the research showed

In their 2022 paper, "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," Jason Wei and coauthors tried a simple change. When showing a model a few example problems, they included the working and reasoning in each example, not only the answer.

On math word problems, commonsense questions, and symbolic tasks, examples that contained reasoning steps led to better results than examples with answers alone. On a math word problem benchmark, a large model of the time reached a new best reported result.

The effect did not show up clearly in small models. It appeared in larger ones. Smaller models sometimes wrote steps that sounded reasonable but were wrong.

Later in 2022, Takeshi Kojima and colleagues reported that even without examples, adding a line like "Let's think step by step" could help in some cases.

Limits of the findings

Some cautions apply:

• The tasks leaned toward arithmetic and logic, where there is one correct answer.
• The models of the time differ from today's models built with reasoning in mind.
• There are reports that the written steps do not always reflect how the model actually reached its answer.

The last point means a tidy explanation on the page may not be the real reason for the result.

Using it in practical instructions

It can help in situations like these:

• Calculations, or choosing among options that must satisfy several conditions: "Check each condition in turn, then give your conclusion."
• When you want to see the basis for a judgment: "List the reasons before the conclusion."
• Tasks with many stages: have the model work through them in numbered order.

On the other hand, some models now available are designed to reason internally before answering. With those, adding "think step by step" may do little or nothing. Check what your tool's documentation says, and compare results with and without the line.

For tasks without a single right answer, such as rewording or summarizing, asking for steps does not reliably improve quality. It can lengthen the output and add more to check.

Keep checking the result

One benefit of seeing the steps is that a person can spot mistakes more easily. But a convincing explanation does not guarantee a correct result. Verify numbers and proper names yourself.

An example of how to write the instruction

In practice, a structure like this works:

1. State the purpose of the task and the conditions to use.
2. List the criteria for judging, numbered.
3. Specify the order: "Check each criterion in turn, then write the conclusion."
4. Specify the output format, such as a table, bullets, or a length.

The benefit of specifying an order is that you can see which criterion went wrong. When only a conclusion appears, finding the error takes longer.

A caution when testing on your own work

A single trial on a few examples cannot be told apart from chance. Compare several runs of the same kind of task with the same inputs, and decide the measure first, such as the number of calculations done correctly or the time spent correcting.

Results from research vary by model and by task type. Rather than copying someone else's success, testing the effect on your own work gives you the best evidence for a decision.

What to do next

1. Pick one task that has many steps and often goes wrong.
2. Run five inputs with a line asking for reasoning first, and five of the same inputs without it.
3. Compare the results and note accuracy and how easy each was to check.
4. If there was no benefit, remove the line from that task's instructions.
5. Save the instruction for any task where it helped, as a standing procedure.