MindSearch is an open-source LLM-based search engine that mimics the human cognitive processes in problem-solving. It uses LLMs to understand the queries, breaks them down into sub-queries and performs separate web searches for each sub-query.
For instance, if a user asks: “Summarize the best paper from major ML conferences in 2023 (NeurIPS, ICML, ICLR), and provide an overview of the trends in several popular fields.“, the model will generate the following sub-tasks:
“First, I need to obtain the winners of the Best Paper in 2023 at the three major ML conferences: NeurIPS, ICML, and ICLR, along with the research topics of these papers. Then, we will analyze the research content of these award-winning papers to determine the trends in future research directions. Let’s start building the search chart.“
This strategy helps the model to gather information from numerous sources, offering more contextual search results than many private AI search engines, such as Perplexity.ai Pro and ChatGPT-Web.
MindSearch can allegedly process and integrate information from over 300 pages in just 3 minutes, a task that would take a human expert around 3 hours.
It’s important to note that Google processes thousands of search queries every second, prioritizing speed and relevance in its search results. In contrast, MindSearch focuses on understanding complex human intentions and providing detailed, context-aware responses.
MindSearch is open-source and you can find the code and models here, along with a comprehensive user guide.
The model
Its framework is made of two main agents: the WebPlanner and a group of WebSearchers (see the picture below).
The WebPlanner acts as the high-level coordinator, managing the reasoning steps and directing the WebSearchers. It simulates the human mind by dynamically constructing a Directed Acyclic Graph (DAG), with nodes representing the independent web searches and edges showing the reasoning relationships.
In this process, known as code as planning, the WebPlanner employs an LLM to iteratively create thoughts and their corresponding Python code. The code is then executed to generate search results. In the final iteration, the WebPlanner directly delivers the response without producing any additional code.
The following figure demonstrates how the WebPlanner systematically tackles a question through a step-by-step planning process.
The WebSearcher performs detailed web searches and summarizes valuable information back to the WebPlanner. It functions as an advanced Retrieve-and-Generate (RAG) agent with internet access, providing summarized and valuable responses.
MindSearch is powered by Large Language Models (LLMs), which have the ability to understand the questions and generate human-like responses. The model supports both closed-source LLMs, such as GPT and Claude, and open-source alternatives, with the InternLM2.5 series being the most optimized for this specific environment.
Pipeline
MindSearch handles complex queries like a human expert, performing the following actions:
- User query input: The user provides a query or question to the system.
- Query decomposition: The WebPlanner breaks down the user’s query into multiple sub-queries using natural language processing (NLP) models.
- Dynamic graph construction: The WebPlanner constructs a reasoning path (graph) that is progressively extended based on the search results from the WebSearcher.
- Parallel processing: The WebSearchers perform detailed web searches for each sub-question.
- Results aggregation: The WebSearchers collect and aggregate relevant information from multiple web pages.
- Answer generation: The WebPlanner integrates the collected information and provides a comprehensive response to the user.
Evaluation
The experiments conducted for evaluating MindSearch focus on two main types of Question Answering (QA) tasks:
- Open-Set QA: This tests the model’s subjective capabilities. MindSearch excelled in terms of depth and breadth, but faced challenges with factual accuracy. This could be due to processing too much information, which might distract the model from the core problem.
- Close-Set QA: This tests the model’s objective capabilities on specific QA datasets like Bamboogle and Musique. MindSearch significantly outperformed LLMs that didn’t use a search engine or simply treated it as a tool. The improvement was even more significant when using an open-source LLM compared to a closed-source one.
The following figure illustrates human preferences for responses generated by MindSearch, Perplexity.ai (Pro), and ChatGPT-Web on open-set QA tasks. These results are based on 100 real-world questions crafted by humans and evaluated by 5 experts.
MindSearch outperforms Perplexity.ai Pro and ChatGPT-Web in terms of depth, breadth, and accuracy (factuality).
How to use it
Visit the repository and follow the “⚽️ Build Your Own MindSearch” section. There are 3 steps:
- Clone the repository and install the dependencies.
- Start the FastAPI server.
- Run one of the frontends: React, Gradio, or Streamlit (the React one doesn’t work for us currently).
When running the server at step 2, you can choose an LLM, for example InternLM2.5-7b-chat (default, optimized for Chinese) or GPT4 (you need an API key). You can also choose the search engine, between DuckDuckGo (default), Bing, Brave, Google (most require an API key).
You can also try it online on HuggingFace. The official one doesn’t seem to work currently, but there are some unofficial Spaces for MindSearch.
Conclusion
MindSearch represents a groundbreaking advancement in AI-driven search technologies. By mimicking the cognitive processes of the human mind, it offers a sophisticated and efficient approach to information retrieval. The multi-agent system, dynamic reasoning paths, and open-source nature make MindSearch a valuable tool for addressing complex queries and integrating extensive information.
MindSearch enhances the capabilities of LLMs and provides accurate, efficient, and scalable solutions, setting a new benchmark for the future of AI-driven search engines.
