Advanced Prompt Engineering Techniques: From Few-Shot to Chain-of-Thought
As Large Language Models (LLMs) grow in parameter size and training quality, prompt engineering has evolved from a simple art of querying to a structured methodology of programming in natural language. By structuring prompts carefully, we can steer model outputs, reduce hallucinations, and unlock reasoning capabilities.
In this guide, we will analyze advanced prompt engineering techniques: Zero-shot vs. Few-shot learning, Chain-of-Thought (CoT) prompting, and the Self-Consistency decoding strategy. We will also write a Python script simulating consistency voting.
Prompt Engineering Techniques: Flowchart mapping Few-Shot learning and Chain-of-Thought reasoning steps
1. Zero-Shot vs. Few-Shot Prompting
LLMs are trained on massive text corpora, giving them broad zero-shot capabilities. However, when you require a specific formatting style, output constraint, or domain-specific logic, Few-Shot Prompting is the standard approach.
Zero-Shot Prompting
You ask the model to perform a task without giving it any examples.
Classify the following review as Positive or Negative:
"The battery life is decent, but the screen is prone to scratches."Few-Shot Prompting
You provide one or more high-quality input-output examples (exemplars) in the prompt to demonstrate the desired behavior.
Classify the following reviews:
Review: "The camera is incredible, and the design is sleek."
Sentiment: Positive
Review: "It broke within two days of usage. Highly disappointed."
Sentiment: Negative
Review: "The battery life is decent, but the screen is prone to scratches."
Sentiment:Providing structured exemplars establishes context, helps the model understand your constraints, and leads to more reliable classification.
2. Chain-of-Thought (CoT) Prompting
Introduced by Wei et al. (2022), Chain-of-Thought prompting dramatically improves performance on complex reasoning, mathematical, and symbolic tasks. Instead of asking the model to output the final answer directly, CoT prompts the model to generate intermediate reasoning steps.
The simplest way to invoke CoT is to append the phrase "Let's think step by step" to the end of the prompt.
Why CoT Works
Autoregressive LLMs predict the next token based on all previous tokens. If you ask a model to solve a complex math puzzle and output only the final number, the model has to perform the entire reasoning pathway in a single forward pass. By prompting the model to write out its reasoning steps, we give it "scratchpad" space to break down the problem, calculate intermediate values, and base its final answer on those step-by-step calculations.
Standard Prompt Chain-of-Thought Prompt
[ Input Math Question ] [ Input Math Question ]
| |
v v
[ Instant Final Output (Error-prone) ] [ Step-by-Step Reasonings (Scratchpad) ]
|
v
[ Final Calculated Answer (Accurate) ]
3. Self-Consistency: Voting on the Best Path
Even with Chain-of-Thought prompting, a model can still make a minor mathematical or logical error in its reasoning path, leading to an incorrect final answer. Self-Consistency (Wang et al., 2022) mitigates this by:
- Generating multiple different reasoning paths (e.g., using a temperature
> 0). - Parsing the final answer from each reasoning path.
- Taking a majority vote to select the most consistent final answer.
+---> Path 1: "... So the answer is 42."
|
[ Input Query with CoT Prompt ] ----+---> Path 2: "... Therefore, we get 42." ---> [ Majority Vote: 42 ]
|
+---> Path 3: "... Thus, it yields 18."
4. Simulating Self-Consistency in Python
Below is a Python demonstration showing how to programmatically implement a self-consistency checker. We mock LLM outputs for a math question and run a consensus algorithm.
from collections import Counter
import re
# Mocking LLM outputs generated with temperature = 0.7 for the question:
# "A toy store has 15 blocks. They sell 4, then buy 8. How many blocks do they have?"
mock_llm_outputs = [
"Let's think step-by-step. The store starts with 15 blocks. Selling 4 means 15 - 4 = 11. Then they buy 8, which is 11 + 8 = 19. The answer is 19.",
"First, we subtract 4 from 15, leaving 11 blocks. Next, we add 8 blocks because they bought more. 11 + 8 = 19. The final answer is 19.",
"Starting with 15. Sold 4: 15 - 4 = 11. Bought 8: 11 + 8 = 18. So the answer is 18.", # Hallucination/calculation error
"We have 15 blocks. 15 - 4 = 11. They add 8 blocks: 11 + 8 = 19. Therefore, the answer is 19."
]
def extract_final_number(text):
"""Regex helper to extract the final number from the LLM text output."""
numbers = re.findall(r'\b\d+\b', text)
return int(numbers[-1]) if numbers else None
def run_self_consistency(outputs):
answers = []
for out in outputs:
answer = extract_final_number(out)
if answer is not None:
answers.append(answer)
print(f"Path reasoning extracted answer: {answer}")
# Count the frequencies of each answer
consensus = Counter(answers)
most_common_answer, count = consensus.most_common(1)[0]
print(f"\nConsensus Votes: {dict(consensus)}")
print(f"Final Selection: {most_common_answer} (Confidence: {count}/{len(outputs)})")
return most_common_answer
# Run the consistency voting pipeline
final_answer = run_self_consistency(mock_llm_outputs)Summary
Prompt engineering is a powerful way to guide LLMs. Moving from zero-shot to few-shot prompting establishes clear formatting patterns, and incorporating Chain-of-Thought gives models the intermediate reasoning steps needed to solve complex problems. By layering Self-Consistency checks on top, we can build robust, production-ready LLM pipelines that minimize errors and maximize accuracy.