-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
356 lines (328 loc) · 21.1 KB
/
Copy pathindex.html
File metadata and controls
356 lines (328 loc) · 21.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" -->
<!-- Replace the content tag with appropriate information -->
<meta name="description" content="DESCRIPTION META TAG">
<meta property="og:title" content="SOCIAL MEDIA TITLE TAG"/>
<meta property="og:description" content="SOCIAL MEDIA DESCRIPTION TAG TAG"/>
<meta property="og:url" content="URL OF THE WEBSITE"/>
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630-->
<meta property="og:image" content="static/image/your_banner_image.png" />
<meta property="og:image:width" content="1200"/>
<meta property="og:image:height" content="630"/>
<meta name="twitter:title" content="TWITTER BANNER TITLE META TAG">
<meta name="twitter:description" content="TWITTER BANNER DESCRIPTION META TAG">
<!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600-->
<meta name="twitter:image" content="static/images/your_twitter_banner_image.png">
<meta name="twitter:card" content="summary_large_image">
<!-- Keywords for your paper to be indexed by-->
<meta name="keywords" content="KEYWORDS SHOULD BE PLACED HERE">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>GuardAgent Project Page</title>
<link rel="icon" type="image/x-icon" href="static/images/shield.png">
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro"
rel="stylesheet">
<link rel="stylesheet" href="static/css/bulma.min.css">
<link rel="stylesheet" href="static/css/bulma-carousel.min.css">
<link rel="stylesheet" href="static/css/bulma-slider.min.css">
<link rel="stylesheet" href="static/css/fontawesome.all.min.css">
<link rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<link rel="stylesheet" href="static/css/index.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="https://documentcloud.adobe.com/view-sdk/main.js"></script>
<script defer src="static/js/fontawesome.all.min.js"></script>
<script src="static/js/bulma-carousel.min.js"></script>
<script src="static/js/bulma-slider.min.js"></script>
<script src="static/js/index.js"></script>
</head>
<body>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-1 publication-title">GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning</h1>
<div class="is-size-5 publication-authors">
<!-- Paper authors -->
<span class="author-block">
<a href="https://zhenxianglance.github.io/" target="_blank">Zhen Xiang</a><sup>1</sup>,</span>
<span class="author-block">
<a href="SECOND AUTHOR PERSONAL LINK" target="_blank">Linzhi Zheng</a><sup>2</sup>,</span>
<span class="author-block">
<a href="https://scholar.google.com/citations?user=h3Chje4AAAAJ&hl=en" target="_blank">Yanjie Li</a><sup>3</sup>,</span>
<span class="author-block">
<a href="https://jyhong.gitlab.io/" target="_blank">Junyuan Hong</a><sup>4</sup>,</span>
<span class="author-block">
<a href="https://qinbinli.com/" target="_blank">Qinbin Li</a><sup>5</sup>,</span>
<span class="author-block">
<a href="https://oxfordblue7.github.io/" target="_blank">Han Xie</a><sup>6</sup>,</span>
<span class="author-block">
<a href="https://javyduck.github.io/" target="_blank">Jiawei Zhang</a><sup>1</sup>,</span>
<span class="author-block">
<a href="https://polaris-73.github.io/" target="_blank">Zidi Xiong</a><sup>1</sup>,</span>
<span class="author-block">
<a href="https://alphapav.github.io/" target="_blank">Chulin Xie</a><sup>1</sup>,</span>
<span class="author-block">
<a href="https://www.cs.emory.edu/~jyang71/" target="_blank">Carl Yang</a><sup>6</sup>,</span>
<span class="author-block">
<a href="https://dawnsong.io/" target="_blank">Dawn Song</a><sup>5</sup>,</span>
<span class="author-block">
<a href="https://aisecure.github.io/" target="_blank">Bo Li</a><sup>17</sup></span>
</div>
<div class="is-size-5 publication-authors">
<span class="author-block"><sup>1</sup>UIUC, <sup>2</sup>Tsinghua University, <sup>3</sup>Hong Kong Polytechnic University,
<sup>4</sup>UT Austin, <sup>5</sup>UC Berkeley, <sup>6</sup>Emory University, <sup>7</sup>University of Chicago</span>
</div>
<div class="column has-text-centered">
<div class="publication-links">
<!-- Arxiv PDF link -->
<span class="link-block">
<a href="https://arxiv.org/pdf/2406.09187.pdf" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fas fa-file-pdf"></i>
</span>
<span>Paper</span>
</a>
</span>
<!-- Github link -->
<span class="link-block">
<a href="https://github.com/guardagent/code" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fab fa-github"></i>
</span>
<span>Code</span>
</a>
</span>
<!-- Dataset Link -->
<span class="link-block">
<a href="https://github.com/guardagent/dataset" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fa fa-database"></i>
</span>
<span>Dataset</span>
</a>
</span>
<!-- API Link -->
<span class="link-block">
<a href="https://arxiv.org/abs/2406.09187.pdf" target="_blank"
class="external-link button is-normal is-rounded is-dark">
<span class="icon">
<i class="fa fa-wrench"></i>
</span>
<span>API</span>
</a>
</span>
</div>
</div>
</div>
</div>
</div>
</div>
</section>
<!-- Project Overview -->
<section class="section hero is-light">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Overview</h2>
<div class="content has-text-justified">
<p>
Welcome to the project page of GuardAgent! In this project, we aim to provide guardrails to LLM-powered agents (dubbed "target agents" below) by checking whether their inputs/outputs satisfy a set of guard requests (e.g., safety rules or privacy policies) defined by the users. This is fundamentally different from guardrails for LLMs (e.g. <a href="https://arxiv.org/abs/2312.06674" style="color:#36AE7C;">Llama Guard</a>) since the output of an LLM agent can be actions, codes, control signals, etc. <a href="https://arxiv.org/abs/2406.09187" style="color:#36AE7C;">GuardAgent</a> is designed with two major steps: 1) creating a task plan by analyzing the provided guard requests, and 2) generating guardrail code based on the task plan and executing the code by calling APIs or using external engines. In both steps, an LLM is utilized as the core reasoning component, supplemented by in-context demonstrations retrieved from a memory module. Such knowledge-enabled reasoning allows GuardAgent to understand various textual guard requests and accurately "translate" them into executable code that provides reliable guardrails. In addition to GuardAgent, we contribute two novel <a href="https://github.com/guardagent/dataset" style="color:#36AE7C;">benchmarks</a>: an EICU-AC benchmark for assessing privacy-related access control for healthcare agents and a Mind2Web-SC benchmark for safety evaluation for web agents. We show the effectiveness of GuardAgent on these two benchmarks with 98.7% and 90.0% guarding accuracy in moderating invalid inputs and outputs for the two types of agents, respectively. Finally, the GuardAgent <a href="https://arxiv.org/abs/2406.09187.pdf" style="color:#36AE7C;">API</a> that provides real-time guardrails based on user guard requests will be coming soon.
</p>
</div>
</div>
</div>
</div>
</section>
<!-- End paper abstract -->
<!-- Method -->
<section class="hero is-small">
<div class="hero-body">
<div class="container" style="width:1000px">
<h2 class="title is-3">Method</h2>
<div>
<p align="center">
<img src="static/images/figure1_v3.png" alt="figure1" width="800"/>
</p>
</div>
<br/><br/>
<div>
<p style="text-align: justify">
The key idea of GuardAgent is to leverage the logical reasoning capabilities of LLMs with knowledge retrieval to accurately ‘translate’ textual guard requests into executable code.<br/><br/>
<b>Inputs to GuardAgent:</b> 1) a set of user-defined guard requests (e.g. for privacy control), 2) a specification of the target agent (needed to inform the user requests), 3) inputs to the target agent, and 4) output (logs) of the target agent.<br/><br/>
<b>Outputs of GuardAgent:</b> 1) whether or not the outputs of the target agent (actions, responses, etc.) are denied, 2) the reasons if the outputs are denied.<br/><br/>
<b>Pipeline of GuardAgent:</b><br/><br/>
<ol style="PADDING-LEFT: 50px; text-align: justify">
<li><b>Task Planning:</b> Generate a step-by-step action plan from the inputs. The prompt to the core LLM contains: 1) planning instructions (fixed for all use cases), 2) demonstrations for task planning retrieved from memory, and 3) inputs to GuardAgent. </li>
<br/>
<li><b>Guardrail Code Generation and Execution:</b> Generate guardrail code based on the generated task plan and execute it. The prompt to the core LLM contains: 1) code generation instructions including all callable functions and APIs, 2) demonstrations for code generation retrieved from memory, and 3) generated action plan.</li>
</ol>
<br/><br/>
<b>Key features of GuardAgent:</b> 1) <b>generalizable</b> -- the memory and tools of \name can be easily extended to address new target agents with new guard requests, 2) <b>reliable</b> -- outputs of GuardAgent are obtained by successful code execution, and 3) <b>training-free</b> -- GuardAgent is in-context-learning-based and does not need any LLM training.
</p>
</div>
</div>
</div>
</section>
<!-- End Method -->
<!-- Benchmark -->
<section class="hero is-small">
<div class="hero-body">
<div class="container" style="width:1000px">
<h2 class="title is-3">Benchmark</h2>
<div>
<p style="text-align: justify">
We propose two novel benchmarks for different safety requests: 1) EICU-AC, which assesses access control for healthcare agents like <a href="https://arxiv.org/abs/2401.07128" style="color:#36AE7C;">EHRAgent</a>, and 2) Mind2Web-SC, which evaluates safety control for web agents like <a href="https://arxiv.org/abs/2401.01614" style="color:#36AE7C;">SeeAct</a>.<br/><br/>
</p>
</div>
<div>
<p align="center">
<img src="static/images/benchmark_example.png" alt="access" width="800"/><br/>
An example from EICU-AC (left) and an example from Mind2Web-SC (right)<br/><br/>
</p>
</div>
<div>
<p style="text-align: justify">
<b>EICU-AC</b> originates from an adapted version of the <a href="https://drive.google.com/file/d/1EE_g3kroKJW_2Op6T2PiZbDSrIQRMtps/view" style="color:#36AE7C;">EICU</a> dataset, which contains questions regarding the clinical care of ICU patients and 10 relevant databases with patient information needed for answering the questions.
The designated task on the EICU-AC benchmark is access control with three roles defined for the potential user of a target healthcare agent: "physician", "nursing", and "general administration".
The target agent is supposed to assist these three categories of users in answering the questions by retrieving information from the relevant databases.
However, each user role has access to only a subset of the databases and a subset of information categories in each accessible database (marked in green below).
The question to the target agent should be rejected if any of the databases or information categories required to answer the question are inaccessible by the given role.
<br/><br/>
</p>
</div>
<div>
<p align="center">
<img src="static/images/access_all.png" alt="access" width="1200"/>
</p>
</div>
<br/><br/>
<div>
<p style="text-align: justify">
Each example in EICU-AC contains the following key information: 1) a healthcare-related question and the correct answer (from EICU), 2) the databases and the information categories required to answer the question (correctly inferred by EHRAgent), 3) a user role (assigned by us), 4) a binary label '0' if all required databases and information categories are accessible to the given role or '1' otherwise, and 5) the required databases and information categories inaccessible to the identity if the label is '1'.
In summary, EICU-AC contains 52, 57, and 45 examples labeled to '0' for "physician", "nursing", and "general administration", respectively, and 46, 55, and 61 examples labeled to '1' for the three roles, respectively.
<br/><br/>
</p>
</div>
<div>
<p style="text-align: justify">
<b>Mind2Web-SC</b> is born out of <a href="https://osu-nlp-group.github.io/Mind2Web/" style="color:#36AE7C;">Mind2Web</a> which contains over 2,000 complex web tasks spanning 137 websites across 31 domains (e.g., car rental, shopping, entertainment, etc.)
The target web agent here is designed to solve each task by conducting a sequence of actions grounded on a provided webpage (e.g. clicking on a certain button).
Mind2Web-SC additionally considers a safety control request with a set of rules that prohibit certain users from engaging in specific web activities (see the figure below).
Each example in Mind2Web-SC includes 1) a task to be conducted, 2) an action step towards the completion of the task (correctly inferred by SeeAct), 3) a user profile containing 'age' in integer, and 'domestic', 'dr_license', 'vaccine', and 'membership', all boolean (created by us), 4) a binary label '1' if the action should be denied due to rule violations and '0' otherwise, and 5) the violated rule if the label is '1'.
In summary, Mind2Web-SC includes 100 examples per label.
<br/><br/>
</p>
</div>
<div>
<p align="center">
<img src="static/images/rules.png" alt="access" width="650"/><br/>
Safety rules of Mind2Web-SC and the number of examples (with label '1') for each rule violation.
</p>
</div>
<br/><br/>
</div>
</div>
</section>
<!-- End Benchmark -->
<!-- Experiment -->
<section class="hero is-small">
<div class="hero-body">
<div class="container" style="width:1000px">
<h2 class="title is-3">Experiment</h2>
<div>
<p style="text-align: justify">
<b>Setup:</b> We test GuardAgent on EICU-AC and Mind2Web-SC, with EHRAgent and SeeAct being the target agent, respectively.
We use GPT-4 version 2024-02-01 with temperature zero as the core LLM of GuardAgent.
For EICU-AC and Mind2Web-SC, we use 1 and 3 demonstrations, respectively.
The guard requests for the two benchmarks are shown below.
<br/><br/>
<div>
<p align="center">
<img src="static/images/guard_request.png" alt="access" width="1000"/><br/>
Guard requests for EICU-AC and Mind2Web-SC in our experiments. GuardAgent is designed to serve diverse guard requests for different target agents.
</p>
</div>
<br/><br/>
<b>Evaluation metrics:</b> 1) label prediction accuracy (<b>LPA</b>) -- the percentage of correct label prediction (i.e., reject inputs for examples labeled to '1' or permit output for examples labeled to '0') over all examples in each dataset,
2) label prediction precision (<b>LPP</b>), 3) label prediction recall (<b>LPR</b>), and (4) comprehensive control accuracy (<b>CCA</b>) -- the percentage of all examples with ground truth labeled '1' that are correctly predicted AND with correct reasoning (i.e., with all inaccessible databases and information categories (for EICU-AC) or all violated rules (for Mind2Web-SC) successfully detected).
<br/><br/>
<b>Baselines:</b> Since GuardAgent is the first LLM agent designed to safeguard other agents, we compare it with baselines using models to safeguard agents.
Approaches for model-guard-model, such as LlamaGuard designed to detect predefined unsafe categories, are not considered here due to their completely different objectives.
Here, we consider GPT-4 version 2024-02-01 and Llama3-70B as the guardrail models.
We create comprehensive prompts containing high-level guardrail task instructions and the same number of demonstrations as for GuardAgent, but without guardrail code generation or the utilization of long-term memory.
<br/><br/>
<div>
<p align="center">
<img src="static/images/results_main.png" alt="access" width="600"/><br/>
Performance of GuardAgent on EICU-AC and Mind2Web-SC compared with the model-guard-agent baselines.
</p>
</div>
<br/><br/>
<div>
<p align="center">
<img src="static/images/detailed_results.png" alt="access" width="640"/><br/>
Breakdown of GuardAgent results over the three roles in EICU-AC and the six rules in Mind2Web-SC.
</p>
</div>
<br/><br/>
<div>
<p align="center">
<img src="static/images/results_ablation.png" alt="access" width="800"/><br/>
Performance of GuardAgent with different numbers of demonstrations on EICU-AC and Mind2Web-SC.
</p>
</div>
<br/><br/>
<b>Results:</b><br/>
<ol style="PADDING-LEFT: 50px; text-align: justify">
<li>GuardAgent outperforms the model-guard-agent baselines. This is likely due to the code generation of GuardAgent that avoids ambiguities in the database names.
<br/>
<li>GuardAgent performs uniformly well for the three roles in EICU-AC and the six rules in Mind2Web-SC, except for rule 5 related to movies, music, and videos, which demonstrates relatively strong capabilities in handling complex guard requests with high diversity.
<br/>
<li>GuardAgent can achieve descent guardrail performance with very few shots of demonstrations.
</ol>
</p>
</div>
</div>
</div>
</section>
<!-- End Experiment -->
<!--BibTex citation -->
<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>
@misc{xiang2024guardagent,
title={GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning},
author={Zhen Xiang and Linzhi Zheng and Yanjie Li and Junyuan Hong and Qinbin Li and Han Xie and Jiawei Zhang and Zidi Xiong and Chulin Xie and Carl Yang and Dawn Song and Bo Li},
year={2024},
eprint={2406.09187},
archivePrefix={arXiv}}
</code></pre>
</div>
</section>
<!--End BibTex citation -->
<footer class="footer">
<div class="container">
<div class="columns is-centered">
<div class="column is-8">
<div class="content">
<p>
This page was built using the <a href="https://github.com/eliahuhorwitz/Academic-project-page-template" target="_blank">Academic Project Page Template</a> which was adopted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a> project page.
</p>
</div>
</div>
</div>
</div>
</footer>
<!-- Statcounter tracking code -->
<!-- You can add a tracker to track page visits by creating an account at statcounter.com -->
<!-- End of Statcounter Code -->
</body>
</html>