Introduction
Amazon Quick chat agents are powerful but getting them to respond exactly right for domain-expert users requires more than uploading documents and writing a one-line prompt. Agent instructions are not a “one size fits all” solution. They must be iteratively refined based on real user feedback until the agent consistently meets your specific accuracy, formatting, and scope requirements.
This article explains how to develop agent instructions that reliably meet user expectations — covering the common pitfalls, the feedback driven refinement process, and a reusable template you can adapt to any use case.
The Challenge
What Domain Specific Agents Must Do
A typical domain-specific chat agent needs to:
- Answer questions accurately — only from approved sources (uploaded documents, knowledge bases, approved websites via web crawlers)
- Cite every response — with document name, section, page, or clickable URL
- Refuse off topic questions — no partial answers, no “helpful” general knowledge
- Stay within scope boundaries — geographic, temporal, or domain-specific
- Never hallucinate — no fabricated data, amounts, or citations
Why This Is Harder Than It Sounds
Out of the box, LLM-powered agents want to be helpful. They will:
- Supplement retrieved content with general knowledge
- Answer questions outside their domain rather than refusing
- Provide data from the wrong scope
- Generate plausible-sounding citations that don’t exist
- Format responses inconsistently
The gap between “mostly works” and “trustworthy enough for expert users” requires precise, layered instructions.
The Refinement Process
Getting agent instructions right is an iterative process. Expect a few refinement cycles based on real user testing. Here are the common issues you’ll encounter and fix, roughly in order:
| Phase | Common Problem | Fix |
|---|---|---|
| 1. Hallucinations | Agent mixes general knowledge with document content | Add strict “ZERO general knowledge” rules + mandatory source citations |
| 2. Scope leakage | Agent answers questions outside its domain or geographic boundary | Add explicit refusal templates with exact response text |
| 3. Citation inconsistency | Vague citations (“See the manual”) or inconsistent formats | Prescriptive citation format: “Every answer MUST end with Source: [Doc, Section, Page]” |
| 4. Formatting issues | Non-clickable links, raw URLs, inconsistent structure | Explicit formatting rules: “All links must be clickable [text](URL)” |
| 5. Missing knowledge | Users ask questions the agent can’t answer from current sources | Expand knowledge bases — add documents, web crawlers, or additional approved sources |
| 6. Source confusion | Agent pulls from the wrong source when multiple exist | Add source routing: “If question is about X → check Source Y” |
| 7. Conflicting sources | Multiple documents give different answers | Add document priority hierarchy: “Amendments → Directives → Base Manual” |
Key insight: Don’t try to anticipate every problem upfront. Start with strong guardrails, then expand and refine based on what real users actually ask.
Key Principles for Agent Instruction Design
1. Be Prescriptive, Not Suggestive
 Vague |
 Prescriptive |
|---|---|
| “Cite your sources” | “Every answer MUST end with Source: [Document, Section, Page]” |
| “Stay on topic” | “Off-topic: respond ONLY with: ‘[exact refusal text]’” |
| “Use the documents” | “Answer ONLY from approved sources. Do NOT use general knowledge.” |
2. Define the Negative Space
The most important instructions tell the agent what NOT to do. Explicitly forbid: supplementing with general knowledge, answering off-topic (even partially), fabricating citations, providing out-of-scope data.
3. Give Exact Templates for Edge Cases
Don’t let the agent improvise responses for edge cases. Provide exact text for: off-topic refusal, information not found, out-of-scope requests, and ambiguous questions.
4. Add Source Routing for Multiple Knowledge Bases
When your agent draws from multiple sources, provide a decision tree:
- [Topic A] → check [Source 1]
- [Topic B] → check [Source 2]
- Questions spanning both → cite both, note precedence
5. Test with Domain Experts
General users might not catch subtle inaccuracies. Domain experts will immediately flag wrong amounts, outdated procedures, missing nuance, and scope violations.
Agent Instructions Template
Below is a generalized template. Adapt each section to your specific use case.
# [Agent Name] — Instructions
## Role
You are the [Agent Name] for [audience description] in [organization].
You answer:
1. [Category 1] — from [source description].
2. [Category 2] — from [source description].
3. [Category 3] — from [approved sources below].
## Scope Enforcement (MANDATORY)
Answer ONLY from approved sources. Refuse everything else.
- [Scope restriction]. Never provide [out-of-scope] data.
- If asked about [out-of-scope topic]: "[Specific refusal with redirect]"
- Do NOT use general knowledge or unapproved sources.
- Off-topic questions: DO NOT answer — not even partially. Respond only with:
"[Specific refusal message stating what you CAN help with]"
- In-scope but not found:
"[Fallback message directing user to appropriate human resource]"
## Approved Sources
### [Source Category 1: Internal Documents]
- [Document 1] (rev. YYYY-MM-DD)
- [Document 2] (rev. YYYY-MM-DD)
### [Source Category 2: External Authorities]
- [Source A] — [description]
- [Source B] — [description]
### [Source Category 3: Approved Websites (Web Crawlers)]
- [domain1.gov] — [what it provides]
- [domain2.org] — [what it provides]
### Source Routing
- [Topic A] → [which source to check]
- [Topic B] → [which source to check]
- Questions mixing sources → cite both
## Core Rules
1. Answer ONLY from approved sources.
2. NEVER supplement with general knowledge. If you cannot retrieve data
from an approved source, point the user to the tool/website and stop.
3. Every answer MUST end with a citation:
Source: [Document, Section, Page] or [Site — Page](clickable URL)
If you cannot cite an approved source, do not make the statement.
4. All links must be clickable — [text](URL). No raw URLs.
5. Never fabricate data points, amounts, or references.
6. [Priority rule]: [Newer documents] override [older documents].
Check newest guidance first.
7. When multiple sources apply, present both and note which takes
precedence.
8. Suggest follow-ups when appropriate.
## Audience
[Description of who uses this agent and their expertise level.
Note any jargon they use naturally.]
## Common Acronyms
[List domain acronyms so the agent responds naturally to shorthand.]
## Response Format
- Direct answer first, then supporting detail, then Source: citation.
- All links clickable: [text](URL).
- Bullet points for multi-part answers.
- Address sub-categories separately when both apply, each with its
own citation.
Tone: [Professional/Concise/Approachable — choose what fits.]
## Question Types
[For each common question type, specify HOW to answer:]
- [Type 1]: Exact figures and steps. Cite section and page.
- [Type 2]: Offer related options from approved sources.
- [Type 3]: Present info only — no [legal/medical/financial] advice.
- Scenario-based: Combine multiple sources. Cite each.
## Guardrails
- [Scope restriction] for [specific data types].
- Refuse off-topic questions without providing any off-topic info.
- ZERO general knowledge. Every factual statement must come from an
approved source.
- No [medical/legal/clinical/financial] advice.
- Protect privacy. Don't repeat PII. Remind users not to share it.
- Never fabricate data or citations.
## Document Lookup Priority
[Newest/Most authoritative] → [Next level] → [Base documents].
Never use [out-of-scope sources].
Summary
Building effective Amazon Quick agent instructions is an iterative craft:
- Start simple → discover gaps through user testing
- Add guardrails → eliminate hallucinations and off-topic responses
- Fix formatting → ensure citations and links are consistently useful
- Enforce scope → geographic, temporal, and domain boundaries
- Expand knowledge → add sources based on actual user needs
- Add routing → help the agent choose the right source
- Polish edge cases → priority hierarchies, dual citations, acronyms
The template above gives you a head start, but expect to iterate multiple times before your agent consistently meets domain-expert expectations. Agent instructions are a living document — not a one-time setup.
About the Author
Govind Palanisamy is a Solutions Architect at AWS, where he helps government agencies migrate and modernize their workloads to increase citizen experience. He is passionate about technology and transformation, and he helps customers transform their businesses using AI/ML and generative AI-based solutions.
Have questions? Share your agent instruction challenges in the comments below.