Prompt & Done · RAGKIT
Why your AI makes things up, and the 30-line fix (RAG)
A language model only knows two things: what it learned in training and what's in your prompt. Your refund policy is in neither, so it either guesses or says it doesn't know. RAG (retrieval-augmented generation) fixes that: find the right piece of your documents and paste it into the prompt.
Tested 7 Nov 2026 with Claude Sonnet 5.5 (Claude Code CLI, no tools) on a demo policy with invented numbers.
1 How it works, in four steps
Chunk: split your documents into small pieces (here: one policy line each). Address: turn each piece into a list of numbers (an embedding) so similar meanings get nearby numbers. Retrieve: turn the question into numbers too and pick the closest piece(s). Paste: put those pieces into the prompt and tell the model to answer only from them and cite them.
The starter below uses TF-IDF word matching instead of real embeddings so it runs with no installs. It matches words, not meaning, which is exactly why it can grab the wrong chunk (see 3). Swap in an embeddings model for real use.
2 The starter (Python 3, no dependencies)
Save as rag_starter.py, save the demo policy below as policy.txt, then run python3 rag_starter.py policy.txt "What's the most you'll refund if I don't have a receipt?". It prints the exact prompt to send; uncomment the last line to send it with the Anthropic SDK (pip install anthropic, set ANTHROPIC_API_KEY).
# rag_starter.py - the smallest useful RAG: retrieve the right line, paste it, ask.
# Retrieval here is TF-IDF word matching (a simplified stand-in for embeddings).
import math, re, sys, collections
def words(t): return re.findall(r"[a-z0-9$]+", t.lower())
def retrieve(question, chunks, k=1):
docs = [collections.Counter(words(c)) for c in chunks]
df = collections.Counter(w for d in docs for w in d)
idf = lambda w: math.log((1 + len(docs)) / (1 + df[w])) + 1
def vec(c): return {w: n * idf(w) for w, n in c.items()}
q = vec(collections.Counter(words(question)))
def cos(a, b):
dot = sum(a[w] * b.get(w, 0) for w in a)
na = math.sqrt(sum(v * v for v in a.values())); nb = math.sqrt(sum(v * v for v in b.values()))
return dot / (na * nb or 1)
scored = sorted(((cos(q, vec(d)), c) for d, c in zip(docs, chunks)), reverse=True)
return scored[:k]
def build_prompt(question, hits):
ctx = "\n".join(c for _, c in hits) # chunks keep their own line numbers
return (f"Answer using ONLY these policy lines and cite the line number.\n"
f"If they don't contain the answer, say you don't know.\n\n{ctx}\n\nQuestion: {question}")
if __name__ == "__main__":
chunks = [l.strip() for l in open(sys.argv[1]) if l.strip()[:1].isdigit()] # one chunk per policy line
question = sys.argv[2]
hits = retrieve(question, chunks)
prompt = build_prompt(question, hits)
print(prompt)
# Send it: pip install anthropic, set ANTHROPIC_API_KEY, then uncomment:
# import anthropic; r = anthropic.Anthropic().messages.create(model="claude-sonnet-5-5", max_tokens=300, messages=[{"role": "user", "content": prompt}]); print(r.content[0].text)
Demo policy (invented numbers):
DEMO POLICY (invented numbers) - Northwind Gadgets customer help
1. Returns are accepted within 21 days of delivery.
2. Refunds over $50 need the original receipt.
3. Without a receipt, the most we refund is $50, as store credit.
4. Opened headphones can be returned only if faulty.
5. Shipping fees are refunded only when the item arrived damaged.
6. Gift cards cannot be refunded or exchanged for cash.
7. Price match: we match our own website price within 7 days.
8. Warranty repairs take 5 to 10 business days.
9. Support hours: 8 am to 6 pm, Monday to Saturday.
10. Damaged items: send a photo within 48 hours of delivery.
3 What our run showed
No policy attached (3/3 runs): "I don't have a confirmed no-receipt refund limit, so I can't give you a figure without guessing."
Line 3 retrieved and pasted (3/3 runs): "The most we refund without a receipt is $50, as store credit (policy line 3)."
Where it breaks: "Can I get my money back on a gift card?" retrieved line 4 (headphones) instead of line 6 (gift cards), because the words "can" and "be" overlapped more. Claude answered "I don't know", because the prompt told it to. Without that instruction a model may answer from the wrong chunk. Fixes: real embeddings, retrieve the top 3 instead of 1, and keep the "say you don't know" line.
Real-world case from the hook: Moffatt v. Air Canada, 2024 BCCRT 149 (an airline was held responsible for what its chatbot told a customer about refunds).