Chain-of-Thought Prompting: What It Is and When It Helps

“Let’s think step by step” might be the most famous sentence in prompt engineering. For a while, adding it to a prompt was close to a magic trick. Models that fumbled simple word problems suddenly got them right.

Then the models changed. Many of today’s AI systems reason internally before they answer, and the old trick can now slow them down, raise your bill and sometimes make answers worse. So is chain-of-thought prompting still worth using?

The short answer: yes, but selectively. This guide explains what chain-of-thought (CoT) prompting is, what the research found, and a simple way to decide when to use it.

What chain-of-thought prompting is

Chain-of-thought prompting means asking a language model to produce intermediate reasoning steps before its final answer. Instead of jumping straight from question to answer, the model writes out the steps in between, the way a student shows their work.

The idea was popularized in a 2022 Google research paper by Jason Wei and colleagues, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. They gave models a few worked examples that included reasoning steps, and found that this improved performance on arithmetic, commonsense and symbolic reasoning tasks. Their headline result: prompting a 540-billion-parameter model with just 8 chain-of-thought examples reached state-of-the-art accuracy on the GSM8K math word problem benchmark at the time.

A few months later, Takeshi Kojima and colleagues showed you didn’t even need examples. In Large Language Models are Zero-Shot Reasoners, simply adding “Let’s think step by step” raised InstructGPT’s accuracy on the MultiArith benchmark from 17.7% to 78.7%, and on GSM8K from 10.4% to 40.7%.

Those are dramatic jumps, and they’re why CoT became standard advice. But they were measured on models from 2022.

The 3 main types of chain-of-thought prompts

Anthropic’s prompting documentation describes a similar progression, from a basic “think step-by-step” instruction, to guided prompts that outline specific steps, to structured prompts that separate reasoning from the answer with tags. It also notes a key point: with a standard model, if the reasoning isn’t written out, it doesn’t happen.

Why it works (in plain English)

A language model generates text one token at a time, and each token is influenced by everything before it. When the model writes out intermediate steps, those steps become part of its context. Later tokens can “build on” earlier ones, rather than forcing the model to leap to a final number in one go.

Think of it as giving the model scratch paper. For a 4-step math problem, writing down step 1 makes step 2 easier to get right. Without the scratch paper, the model has to compress all of that work into the first few tokens of its answer.

That also explains when CoT doesn’t help. If a task doesn’t require multiple dependent steps, such as recalling a fact, classifying sentiment or rewriting a sentence, extra reasoning text adds cost without adding much accuracy.

What newer research says

The picture got more nuanced as models improved.

A 2024 study by Zayne Sprague and colleagues, To CoT or not to CoT?, combined a meta-analysis of over 100 papers with its own tests on 20 datasets across 14 models. The finding is right in the title: chain-of-thought helps mainly on math and symbolic reasoning, with much smaller gains on other task types. On the broad MMLU knowledge benchmark, CoT added little except for questions involving symbolic operations, such as those containing an equals sign. The authors suggest using CoT selectively to save inference costs.

A 2025 report from the Wharton Generative AI Labs, The Decreasing Value of Chain of Thought in Prompting by Lennart Meincke, Ethan Mollick, Lilach Mollick and Dan Shapiro, tested CoT on recent models. They found:

• Non-reasoning models generally gained a small average improvement, but answers became more variable, and CoT sometimes introduced errors on questions the model would otherwise get right.
• Many recent models already reason step by step without being asked, so the explicit request changed little.
• Reasoning models saw marginal accuracy gains at best, while response time and token use went up substantially.

Key takeaway: Chain-of-thought prompting still helps standard models on multi-step math and logic. For reasoning models and simple tasks, it mostly adds time and cost.

Chain of thought vs reasoning models

AI reasoning models are trained to produce their own internal chain of thought before answering. In effect, CoT is now built into the model rather than requested in the prompt.

That changes the advice. OpenAI’s reasoning best-practices guide says that prompting reasoning models to “think step by step” or “explain your reasoning” is unnecessary, because they reason internally, and that such instructions may not improve performance. Anthropic’s guidance for its extended thinking feature similarly recommends high-level instructions to think deeply rather than prescriptive step-by-step scripts, and suggests removing old chain-of-thought instructions when switching to extended thinking.

Here’s how the 2 approaches compare:

Both approaches spend extra tokens. If you’re watching costs, read up on AI tokens and context windows before you scale up.

When chain-of-thought prompting helps

Use CoT when all 3 of these are true:

1. The task has multiple dependent steps. Math word problems, pricing calculations, scheduling with constraints, logic puzzles, multi-condition eligibility checks.
2. You’re using a standard (non-reasoning) model. Smaller, cheaper or older models benefit most.
3. Accuracy matters more than speed. You can afford a longer, slower response.

It’s also useful when you want to audit the reasoning. Seeing the steps helps you spot where an answer went wrong, though written reasoning isn’t guaranteed to reflect exactly how the model reached its answer.

When it wastes tokens

Skip CoT for:

• Reasoning models, which already think before answering.
• Simple lookups and rewrites, such as summaries, translations, tone changes and formatting.
• Classification at scale, where CoT multiplies cost across thousands of calls.
• Real-time chat, where users notice the delay.
• Fact recall, where reasoning steps can make a confidently wrong answer look more convincing. CoT doesn’t fix AI hallucinations.

How to use chain-of-thought prompting well

If you decide CoT fits your task, these steps keep it effective and affordable.

Step 1: test without it first

Run 20 to 50 real examples with a plain prompt. Measure accuracy, response time and token count. This is your baseline.

Step 2: add the lightest version

Try a zero-shot instruction: “Work through this step by step, then give your final answer on the last line.” Rerun the same examples and compare.

Step 3: guide the steps if needed

If results are still weak, name the steps you want: “First list the constraints. Then check each option against them. Then choose.” This is especially useful for business rules.

Step 4: separate reasoning from the answer

Ask for the reasoning and the final answer in clearly labeled sections. Your app can show only the answer to users and log the reasoning for review.

Step 5: keep it only if it pays

If accuracy improves enough to justify the extra tokens and latency, keep it. If not, remove it or switch to a reasoning model for that specific task.

This approach fits into a broader practice of prompt engineering: test with real examples, measure, and keep only what earns its place.

Conclusion

Chain-of-thought prompting was a breakthrough in 2022, and it still has a job: helping standard models get multi-step math and logic right. But it’s no longer a default. Reasoning models do this work internally, and on simple tasks, CoT mostly adds cost. Test it on your own examples, keep it where it measurably helps, and drop it everywhere else. For more on choosing models, see our comparison of ChatGPT, Claude and Gemini.

Sources

• Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv 2201.11903) — original CoT paper and GSM8K result with 8 exemplars
• Kojima et al., Large Language Models are Zero-Shot Reasoners (arXiv 2205.11916) — “Let’s think step by step” accuracy figures on MultiArith and GSM8K
• Sprague et al., To CoT or not to CoT? (arXiv 2409.12183) — meta-analysis showing CoT helps mainly on math and symbolic tasks
• Meincke et al., The Decreasing Value of Chain of Thought in Prompting (arXiv 2506.07142) — CoT effects on recent standard and reasoning models
• OpenAI reasoning best practices — guidance not to prompt reasoning models to think step by step
• Anthropic extended thinking tips — high-level thinking instructions for extended thinking