Document 3 of 4 · Vendor demos
Adversarial testing vendor demo agenda
- Format: scripted 2-hour working session against our seeded demo target D-1
- Question bank: Q1–Q20, each traced to requirement IDs
- Scoring: weighted rubric (9 criteria); score determines advancement to the POC
01Pre-work
- Demo runs against our target D-1; vendor slides limited to 10 minutes
- All vendors receive the same target, time allocation and questions
| When | Action | Owner |
|---|---|---|
| T-14 days | Send vendor brief: use cases UC-1 to UC-4, the scoring method, and the rule that the demo runs against our target with at most 10 minutes of slides | POC lead |
| T-14 days | Send written questionnaire (security, deployment, data handling, pricing, ownership); due T-3 | Procurement + security architecture |
| T-7 days | Stand up demo target D-1: a lightweight RAG chat + 3-tool agent + 1 MCP server, instrumented with canaries. D-1 is separate from the POC target. | Platform engineer |
| T-2 days | Issue D-1 credentials: OAuth client, two user roles, API spec, MCP endpoint, egress allowlist for vendor IPs | Platform engineer |
| T-0 | Evaluators get a scoring sheet; one person runs the clock; one captures evidence screenshots | POC lead |
02Demo target D-1
| Component | What it exercises | How success is observed |
|---|---|---|
| D-1a RAG chat | Indirect-injection resistance; hidden-context protection | Canary strings and out-of-band callbacks we control |
| D-1b Agent with 3 tools | Tool-use authorization and action scoping per user | Side-effect sinks (ledger, mail sink) we control |
| D-1c MCP server | Tool-description trust | Canary access we control |
- All vendors are tested against the same D-1 flaws
- POC reference targets (doc 4) use a different, undisclosed ground-truth set
03Two-hour demo agenda
00:00 – 00:10
Our context (us)
- Business context, AI app portfolio, UC-1 to UC-4
- Scoring: weighted criteria (§5)
- Ground rules: vendor drives the keyboard; we may ask to pause or rerun at any point
00:10 – 00:20
Architecture & data flow (vendor, max 10 min)
- Where the attacker model and judge run; which model(s); whether we can bring our own
- What leaves our network: prompts, responses, tool traces; retention; region
- Deployment options: SaaS, VPC, self-hosted, air-gapped; vendor shows the architecture diagram
00:20 – 00:35
Live onboarding of D-1
- Connect D-1a over our API with OAuth2 + token refresh; prove multi-turn session state works
- Connect the D-1b agent so tool calls are visible (SDK hook, proxy, trace ingestion, or OTel)
- Register the D-1c MCP server
- Measured: minutes to first successful attack run, and whether any vendor engineering help was needed
00:35 – 01:00
Live attack scenarios S1–S5
- S1 Context profiling: the tool profiles D-1 and proposes business-specific objectives
- S2 Adaptive multi-turn: crescendo-style attack against our content policy ("never give visa/legal advice")
- S3 Indirect injection: find the indirect-injection condition in D-1a and show the evidence chain
- S4 Agent tool misuse: find the authorization weakness in D-1b; show the tool-call trace and side-effect proof
- S5 MCP: assess the D-1c server and show what the agent did with its tools
01:00 – 01:15
Findings & evidence walkthrough
- Open one Critical: transcript, tool trace, retrieved content, judge rationale, severity, framework mapping (OWASP LLM / ASI / MCP / ATLAS)
- Replay the finding live to test reproducibility
- Mark one finding as a false positive; show that the dismissal persists into the next run
- Show the judge prompt and model, and whether we can change them
01:15 – 01:30
CI/CD & regression
- Suite as config-as-code committed to a repo; run as a pipeline gate (GitHub Actions or ours)
- Threshold per objective; non-zero exit on failure
- Diff two versions of D-1 (we toggle one fix): show fixed / new / regressed
- Schedule a recurring run and show drift alerting
01:30 – 01:40
Remediation & runtime loop
- Remediation guidance: prompt hardening, tool scoping, authorization fix
- Push to Jira / ServiceNow; SIEM export; SARIF/JSON
- Guardrail generation: support for third-party guardrails vs. own guardrail only
01:40 – 01:50
Platform security & administration
- SSO, SCIM, RBAC by app/team, audit log
- Containment: how the attacker agent is prevented from reaching non-target systems; kill switch
- Token/cost controls and rate limiting per target
01:50 – 02:00
Commercials, roadmap, continuity
- Pricing unit and what drives cost (apps, scans, tokens, seats)
- Roadmap dates: OWASP LLM 2026 IDs, Agent Control Standard, A2A, voice, multimodal
- Ownership: acquisition status, continuity commitments, data export on exit
- Two reference customers running agentic targets in production CI
04Question bank
- Q15–Q19: sent in advance with the T-14 questionnaire
- Q1–Q14, Q20: asked live while the vendor drives
| # | Question | Traces to |
|---|---|---|
| Targets & integration | ||
| Q1 | How do you observe tool calls for agents you don't host? Which of SDK hook, proxy, trace ingestion and OTel GenAI do you support today? | TGT-05, AGT-01 |
| Q2 | How do you handle OAuth refresh, per-user sessions and multiple user roles in one campaign? | TGT-02, AGT-03 |
| Q3 | Can you act as a malicious MCP server or tool response to test our agent, not only test our MCP servers? | TGT-06 |
| Q4 | How do you seed indirect injection into RAG, email, tickets and web without write access to production stores? | TGT-07 |
| Attack quality | ||
| Q5 | Where does your attack library come from, how often is it updated, and how do you avoid overfitting to public benchmarks? | ATK-01 |
| Q6 | Which attacker model(s) do you use? Can we bring our own, including self-hosted? | ATK-08 |
| Q7 | Do you run multiple attempts per objective and report pass@k? What k is the default? | ATK-07 |
| Q8 | Show an objective you generated that is specific to our business, not a template. | ATK-03 |
| Evaluation | ||
| Q9 | What is your judge's measured accuracy against human labels, and how was it measured? Can we see the data? | EVL-02 |
| Q10 | Which findings rely on deterministic oracles (canaries, OOB, tool-call assertions) and which on LLM judgment? | AGT-02, EVL-01 |
| Q11 | How do you report variance across runs? Can any finding be replayed exactly? | EVL-03, OPS-05 |
| Q12 | How do you measure coverage of our tools, permissions and data flows? | AGT-07 |
| Lifecycle | ||
| Q13 | How do baselines work across releases? Show fixed / new / regressed. | OPS-03 |
| Q14 | What is the runtime of a CI smoke suite on a typical agent? What does the full suite cost in tokens? | OPS-02, COM-01 |
| Security of the tool | ||
| Q15 | What data leaves our environment, where is it stored, and for how long? Is any of it used to train your models? | SEC-03 |
| Q16 | How is the attacker agent contained? Has it ever reached a non-target system at a customer? | SEC-04 |
| Q17 | SOC 2 Type II / ISO 27001 scope; latest pen test; product SBOM. | SEC-05 |
| Commercial & viability | ||
| Q18 | Are you in acquisition talks, or were you acquired? What continuity commitments will you contract for? | COM-03 |
| Q19 | Pricing unit, overage model, and the cost of a 4-week POC. | COM-01 |
| Q20 | Dated roadmap for OWASP LLM 2026, ASI, Agent Control Standard, A2A and voice. | COM-02 |
05Scoring rubric
| Criterion | Weight | 1 = poor | 3 = acceptable | 5 = excellent |
|---|---|---|---|---|
| Onboarding to our target | 10% | Needed vendor engineering or code changes in D-1 | Connected with help in under 15 min | Self-serve; multi-turn + agent traces in under 10 min |
| Attack relevance (S1, S2) | 15% | Generic jailbreak list | Some tailoring to D-1 | Business-specific objectives; adaptive multi-turn beat our policy |
| Indirect injection & exfil (S3) | 15% | Not demonstrated | Demonstrated via transcript only | Beacon hit with full evidence chain |
| Agentic side effects (S4) | 20% | Text-only verdicts | Tool call shown, no side-effect proof | Tool trace + ledger/canary proof; authorization root cause identified |
| MCP (S5) | 5% | Not supported | Static description flag | Static flag + observed agent behavior |
| Evidence, replay, judge transparency | 10% | Opaque judge; no replay | Transcript + mapping | Replay works; judge editable; FP dismissal persists |
| CI/CD & regression | 15% | UI only | CLI gate, no diff | Config-as-code gate + fixed/new/regressed diff |
| Platform security & deployment fit | 5% | SaaS only, unclear retention | SaaS with region pinning + SSO | VPC/self-hosted, BYO model, containment controls |
| Commercial & viability | 5% | Opaque pricing; continuity unclear | Clear pricing | Clear pricing + contractual continuity + agentic references |
- Each evaluator scores independently within 24 hours
- Group reconciles any criterion with a score spread of 2 or more
- Weighted score = Σ(score × weight) ÷ 5 × 100
- Advance to POC: ≥ 65 and no 1s on agentic side effects or evidence
06Demo schedule
| Week | Mon–Tue | Wed–Thu | Fri |
|---|---|---|---|
| Demo week 1 | Pillar · Straiker | Mindgard · incumbent platform module | Evaluator calibration: independent scores, then reconcile |
| Demo week 2 | Alternates if needed: Repello / F5 / Gray Swan | OSS control-group walkthrough (internal: promptfoo + PyRIT on D-1) | Down-select to 3 vendors for POC; send POC terms |