**Multi-Agent/MCP Parallel Running**
## Dataset Features
**Genuinue Users' Queries**: Most of the high quality datasets are collected from query logs of our live AI Agents, such as [MCP Tool Use Agent](https://agent.deepnlp.org/agent/mcp_tool_use), [Open OneKey Agent Router](https://www.deepnlp.org/agent/onekey-mcp-router) and [Open OneKey MCP Router](https://www.deepnlp.org/agent/onekey-mcp-router).
**Function Call and MCP Servers Support**: The datasets covers wide range of MCP servers from the Open MCP Marketplace() and Playgrounds.
**Users Action and Humans' Feedback**: Users' actual feedbacks are crucial in improving the AI Agents training process. We collects users' genuine actions, such as **ACCEPT/REJECT** in confirming the function call results, **Upvote/Downvote** action of the final responses, and many other users' feedback on clickable elements.
**Various Domains and Tasks**: We covers 40+ categories of AI agents' tool use scenarios, ranging from information seeking (AI search, map search, etc) to autonomous AI agents browser use, computer use, Data Analysis, Excel Spreadsheet and Powerpoint creation and generation, etc.
**Example AI Agent Dataset Dialogues**
| Domain | Related MCP Server | Demo |
| ---- |------------------------------------------------------------------------------------| ---- |
| Office File Agent | Excel Spreadsheet, Powerpoint, PDF, etc | [Example](https://agent.deepnlp.org/agent/mcp_tool_use/share/ee640008-6bc1-4c3a-832b-2557f985b540) [MCP]() |
| AI Search/Deep Research | Bing/Google Custom/Perplexity/Tavily/Firecrawl | [Demo](https://agent.deepnlp.org/agent/mcp_tool_use?server=tavily-ai/tavily-mcp) [MCP]() |
| Map Trip Planning | GoogleMap, Amap(Gaode), BaiduMap, etc. | [Example](https://agent.deepnlp.org/agent/mcp_tool_use/share/8ab0b25c-b72d-4cae-9c86-a852df8c6541) [MCP](https://agent.deepnlp.org/agent/mcp_tool_use?server=amap-mcp/amap-mcp-%E9%AB%98%E5%BE%B7%E5%9C%B0%E5%9B%BE-mcp) [Use MCP]() |
| Browser Usage | Playwright, Puppeteer, etc. | [Demo](https://agent.deepnlp.org/agent/mcp_tool_use?server=puppeteer/puppeteer) [MCP]() |
| Chart,Graph,Image | Gemini NanoBanana3,everart,mcp-server-charts(AntV),canva-mcp,bing-image-search-mcp | [Example](https://agent.deepnlp.org/agent/mcp_tool_use/share/1a6101e0-9afa-47c4-828c-f3d373dc96f1) [MCP]() |
| Multi-Agent Parallel Running | Deep Research/Trip Planning and more. | [Example](https://agent.deepnlp.org) |
## Dataset Introduction
We provide main below types of AI agents datasets in List of Messages Json Formats and scalar data such as rewards, etc.
| Dataset Name | Description | User Feedback | Example Dataset Download | Full DataSet Download |
|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ---- |-------|-----------------------------------------------------------------------------------------------------------------|
| Tool Use Multi-Turn Dialogue | The tool use multi-turn dialogue dataset is in the list of messages formats, Useful for AI Search/Deep Research/Map/Financial Data/etc | YES | 50 instances, [Download](https://www.deepnlp.org/store/dataset/dataset/pub-deepnlp/agent-tool-use-dialogue-open-dataset-example) | 1k, [Download](https://www.deepnlp.org/store/dataset/dataset/pub-deepnlp/agent-tool-use-dialogue-open-dataset) |
| Function Calling Tool Use | The dataset contains **messages** and **available tools** as input and output the choosen **tool_call** result indicating which tool to use and the arguments. The datasets are collected from calling SOTA LLM such as GPT, OpenAI o-series, Claude, Qwen, Kimi, etc. | No | 50 instances, [Download](https://deepnlp.org/store/dataset/dataset/pub-deepnlp/agent-function-calling-open-dataset-example) | 1k, [Download](https://deepnlp.org/store/ai-agent/ai-agent/pub-deepnlp/agent-function-calling-open-dataset) |
| Reinforcement Learning | Sessions of user and assistant' multi-dialogues, rewards from users' feedback in this session, such click of confirmation (Accept/Reject), Upvote, Downvote on the responses, etc. | YES | 50 instances, [Download](https://deepnlp.org/store/dataset/dataset/pub-deepnlp/agent-reinforcement-learning-open-dataset-example) | 1k, [Download](https://deepnlp.org/store/dataset/dataset/pub-deepnlp/agent-reinforcement-learning-open-dataset) |
### Using Datasets Training Scripts
How to use these datasets? We have also provided SFT/RL training demo scripts with packages such as `trl`, `verl` and
you can find the demo scripts in [GitHub](https://github.com/AI-Hub-Admin/tutorial_agent_training_sft_rl/tree/main)
### Dataset 1 Tool Use Multi-Turn Dialogue Dataset
**Dataset Description**
| KEY | Type | Description |
| ---- |---------------------|-----------------------------------------------------------------------------------------------------------------------------|
| trace_id | String | Identify each unique new user request or API calling |
| session_id | String | The identifier of each dialogue, which consists of multiple turns of dialogues and every user input produces a new trace_id |
| messages | List of Json Object | Dialogue Messages |
This data instances indicates a multi-turn dialogues of users' calling Google Maps **get_weather** tool to know
the recent weather in San Francisco. The dialogues contains three types of messages:
```
User: query, original question that user asks,
User: available_tools, List of Json that user provides to LLM,
Assistant: message, content.type='tool_use', LLM output which tool to use and its parameters,
User: message, content.type='tool_result', Users' actual function call running results.
```
```
[
{
"role": "user",
"content": "What is the weather like in San Francisco?"
},
{
"role": "assistant",
"content": [
{
"type": "text",
"text": "I need to use get_weather, and the user wants SF, which is likely San Francisco, CA."
},
{
"type": "tool_use",
"id": "toolu_01A09q90qw90lq917835lq9",
"name": "get_weather",
"input": {
"location": "San Francisco, CA",
"unit": "celsius"
}
}
]
},
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_01A09q90qw90lq917835lq9",
"content": "15 degrees"
}
]
}
]
```
Note that the function call comes in different formats when calling various models. We are mainly collecting in the OpenAI and anthroupic function calling formats.
We supported both and you can see the differences from the offical documentations.
**Multi-modal and Files** formats are also attached:
The images and raw descriptions of the files such as path are also attached for context variables.
**OpenAI/Qwen/etc Function Call Formats**
```
{
"tool_call": {
"id": "call_d6f4ed29ce614390b99a05",
"function": {
"arguments": "{\"url\": \"https://www.stackoverflow.com\", \"browserType\": \"chromium\"}",
"name": "playwright_navigate"
},
"type": "function",
"index": 0
}
}
```
**Anthroupic Tool Use Formats**
```
{
"type": "tool_use",
"id": "toolu_01A09q90qw90lq917835lq9",
"name": "get_weather",
"input": {
"location": "San Francisco, CA",
"unit": "celsius"
}
}
```
### Dataset 2 Function Calling Tool Use Dataset
The functions call tool use datasets contains both the List of **messages** (Users' original query) and **Available Tools** (List of Json) and **Tool Call Results** produced by SOTA LLM.
For example, the LLM chooses **puppeteer_navigate** tool from 7 other tools of MCP Server puppeteer. And set the parameters as **{\"url\": \"https://www.baidu.com\"}**
**Dataset Description**
| KEY | Description |
|--------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| trace_id | Identify each unique new user request or API calling |
| session_id | The identifier of each dialogue, which consists of multiple turns of dialogues and every user input produces a new trace_id |
| data | The Json Object of Each Function Call |
| data.function_calls.messages | List of messages in {"role":"", "content":""} formats. |
| data.function_calls.tools | List of available tools, including the schema of tools name, description and parameters |
| data.function_calls.tool_calls | Dict of tool_id, function, arguments |
| tool_calls_results | Note: Tool Call Result are tool running actual output, which can be found in the list of dialogue message in Dataset 1, but not needed for training function call models training dataset |
Format
```
{
"model": "claude-opus-4",
"session_id": "37e83ca0-d0d4-4204-b3e6-95be51bf57a6",
"trace_id": "5c6a8e17-e121-4786-955d-0a716ed733ae",
"function_calls": [{
"messages": [],
"tools": [],
"tool_calls": {
"id": "",
"function": {"arguments":"", "name":""},
"type": ""
}
},
{
"messages": [],
"tools": [],
"tool_calls": {
"id": "",
"function": {"arguments":"", "name":""},
"type": ""
}
}
]
}
```
Data Example of Puppeteer Navigate Open Website.
```
{
"trace_id": "72d51b6e-8c48-42ee-bfc9-05209d1444c8",
"session_id": "7ca7e84c-cee5-4ddd-be1e-748df2be481e",
"model": "claude-opus-4",
"data": {
"function_calls": [{
"messages": [{
"role": "user",
"content": "Visit website baidu.com and take a screenshot"
}, {
"role": "assistant",
"content": "",
"tool_calls": [{
"id": "call_f02ba9ee6b344dd0af9041",
"type": "tool_use",
"function": {
"name": "puppeteer_navigate",
"arguments": "{\"url\": \"https://www.baidu.com\"}"
}
}]
}, {
"role": "tool",
"tool_call_id": "call_f02ba9ee6b344dd0af9041",
"name": "puppeteer_navigate",
"content": ""
}, {
"role": "assistant",
"content": "",
"tool_calls": [{
"id": "call_c1e8658beda04f6881e8eb",
"type": "tool_use",
"function": {
"name": "puppeteer_screenshot",
"arguments": "{\"name\": \"baidu_homepage\"}"
}
}]
}, {
"role": "tool",
"tool_call_id": "call_c1e8658beda04f6881e8eb",
"name": "puppeteer_screenshot",
"content": ""
}],
"tools": [{
"type": "function",
"function": {
"name": "puppeteer_navigate",
"description": "Navigate to a URL",
"parameters": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "URL to navigate to"
},
"launchOptions": {
"type": "object",
"description": "PuppeteerJS LaunchOptions. Default null. If changed and not null, browser restarts. Example: { headless: true, args: ['--no-sandbox'] }"
},
"allowDangerous": {
"type": "boolean",
"description": "Allow dangerous LaunchOptions that reduce security. When false, dangerous args like --no-sandbox will throw errors. Default false."
}
},
"required": ["url"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_screenshot",
"description": "Take a screenshot of the current page or a specific element",
"parameters": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Name for the screenshot"
},
"selector": {
"type": "string",
"description": "CSS selector for element to screenshot"
},
"width": {
"type": "number",
"description": "Width in pixels (default: 800)"
},
"height": {
"type": "number",
"description": "Height in pixels (default: 600)"
},
"encoded": {
"type": "boolean",
"description": "If true, capture the screenshot as a base64-encoded data URI (as text) instead of binary image content. Default false."
}
},
"required": ["name"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_click",
"description": "Click an element on the page",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "CSS selector for element to click"
}
},
"required": ["selector"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_fill",
"description": "Fill out an input field",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "CSS selector for input field"
},
"value": {
"type": "string",
"description": "Value to fill"
}
},
"required": ["selector", "value"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_select",
"description": "Select an element on the page with Select tag",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "CSS selector for element to select"
},
"value": {
"type": "string",
"description": "Value to select"
}
},
"required": ["selector", "value"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_hover",
"description": "Hover an element on the page",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "CSS selector for element to hover"
}
},
"required": ["selector"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_evaluate",
"description": "Execute JavaScript in the browser console",
"parameters": {
"type": "object",
"properties": {
"script": {
"type": "string",
"description": "JavaScript code to execute"
}
},
"required": ["script"]
}
}
}],
"tool_calls": {
"id": "call_f02ba9ee6b344dd0af9041",
"function": {
"arguments": "{\"url\": \"https://www.baidu.com\"}",
"name": "puppeteer_navigate"
},
"type": "function"
}
}, {
"messages": [{
"role": "user",
"content": "Visit website baidu.com and take a screenshot"
}, {
"role": "assistant",
"content": "",
"tool_calls": [{
"id": "call_f02ba9ee6b344dd0af9041",
"type": "tool_use",
"function": {
"name": "puppeteer_navigate",
"arguments": "{\"url\": \"https://www.baidu.com\"}"
}
}]
}, {
"role": "tool",
"tool_call_id": "call_f02ba9ee6b344dd0af9041",
"name": "puppeteer_navigate",
"content": ""
}, {
"role": "assistant",
"content": "",
"tool_calls": [{
"id": "call_c1e8658beda04f6881e8eb",
"type": "tool_use",
"function": {
"name": "puppeteer_screenshot",
"arguments": "{\"name\": \"baidu_homepage\"}"
}
}]
}, {
"role": "tool",
"tool_call_id": "call_c1e8658beda04f6881e8eb",
"name": "puppeteer_screenshot",
"content": ""
}],
"tools": [{
"type": "function",
"function": {
"name": "puppeteer_navigate",
"description": "Navigate to a URL",
"parameters": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "URL to navigate to"
},
"launchOptions": {
"type": "object",
"description": "PuppeteerJS LaunchOptions. Default null. If changed and not null, browser restarts. Example: { headless: true, args: ['--no-sandbox'] }"
},
"allowDangerous": {
"type": "boolean",
"description": "Allow dangerous LaunchOptions that reduce security. When false, dangerous args like --no-sandbox will throw errors. Default false."
}
},
"required": ["url"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_screenshot",
"description": "Take a screenshot of the current page or a specific element",
"parameters": {
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "Name for the screenshot"
},
"selector": {
"type": "string",
"description": "CSS selector for element to screenshot"
},
"width": {
"type": "number",
"description": "Width in pixels (default: 800)"
},
"height": {
"type": "number",
"description": "Height in pixels (default: 600)"
},
"encoded": {
"type": "boolean",
"description": "If true, capture the screenshot as a base64-encoded data URI (as text) instead of binary image content. Default false."
}
},
"required": ["name"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_click",
"description": "Click an element on the page",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "CSS selector for element to click"
}
},
"required": ["selector"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_fill",
"description": "Fill out an input field",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "CSS selector for input field"
},
"value": {
"type": "string",
"description": "Value to fill"
}
},
"required": ["selector", "value"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_select",
"description": "Select an element on the page with Select tag",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "CSS selector for element to select"
},
"value": {
"type": "string",
"description": "Value to select"
}
},
"required": ["selector", "value"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_hover",
"description": "Hover an element on the page",
"parameters": {
"type": "object",
"properties": {
"selector": {
"type": "string",
"description": "CSS selector for element to hover"
}
},
"required": ["selector"]
}
}
}, {
"type": "function",
"function": {
"name": "puppeteer_evaluate",
"description": "Execute JavaScript in the browser console",
"parameters": {
"type": "object",
"properties": {
"script": {
"type": "string",
"description": "JavaScript code to execute"
}
},
"required": ["script"]
}
}
}],
"tool_calls": {
"id": "call_c1e8658beda04f6881e8eb",
"function": {
"arguments": "{\"name\": \"baidu_homepage\"}",
"name": "puppeteer_screenshot"
},
"type": "function"
}
}]
},
}
```
### Dataset 3 Reinforcement Learning
We collect users' positive and negative feedbacks on the AI Agent workflow.
**Positive feedback** include the click on the **ACCEPT** button of the function call results and **Upvote** button. We set rewards as 1.0, and use "reward_description" field to include detailed introduction of their actions.
**Negative feedback** include the click on the **REJECT** button of the function call results and **Downvote** button. We set rewards as -1.0.
**Dataset Description**
| KEY | Type | Description |
|----------------|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
| trace_id | String | Identify each unique new user request or API calling |
| session_id | String | The identifier of each dialogue, which consists of multiple turns of dialogues and every user input produces a new trace_id |
| messages | List of Json Object | Dialogue Messages |
| message.reward | scalar | Users' feedback on each individual message or function call, rool result level. ACCEPT/REJECT, 1.0 for positive feedback, -1.0 for negative feedback |
| message.reward_description | string | Detailed description of actions of users. |
| reward | scalar | Overall feedback of whole session, Buttom Bar Action, 1.0 for positive feedback, -1.0 for negative feedback |
| reward_description | string | Details description of the action, such as ACCEPT/REJECT,UPVOTE,DOWNVOTE, etc |
** Note that: The rewards are on two folds: each function call or message level, whole dialogue session level.
Each message or function call level: Json $.message.reward ,e.g. Users may not be satisfied with LLM choose the wrong tools, they might click rejects or after autorun they may click feedback.
Dialogue session level: Json: $reward, Bottom Bar Buttons, Upvote, Downvote, etc.
Auto-Run mode, function calls doesn't need users' confirmation and default to Accept.
**Positive Reward Example**
In this financial data checking scenarios, users' clicks on the ACCEPT to confirm the function calls of tool 'get_stock_price_global_market' and parameters "{\"symbol_list\": [\"MSFT\", \"NVDA\"], \"market\": \"US\"}" form server finance-agent-mcp-server.
```
{
"messages": [{
"role": "user",
"content": "Find the Microsoft and Nvidia Stock Price and Market Cap. Who has higher market cap today?"
}, {
"role": "assistant",
"content": "Finding Relevant Tools Using Queries:
Relevant MCP Servers:
\rRelevant Tools:
Finding Relevant Tools Using Queries:
Relevant MCP Servers:
\rRelevant Tools:
Finding Relevant Tools Using Queries:
Relevant MCP Servers:
\rRelevant Tools: