Awesome

<p align=center>Natural Language-Based Societies of Mind</p><!-- omit in toc -->

What magical trick makes us intelligent? The trick is that there is no trick. The power of intelligence stems from our vast diversity, not from any single, perfect principle. — Marvin Minsky, The Society of Mind, p. 308

✨ Introduction

We introduce the Natural Language-Based Societies of Mind (NLSOM) concept, which contains societies and communities of agents.

🔥 News:

• NLSOM got Best Paper Award in NeurIPS 2023 Ro-FoMo Workshop!!
• Dylan R. Ashley will give a presentation of NLSOM in NeurIPS RO-FoMo workshop. See our poster.
• This position paper marks the beginning. Our vision continues to unfold and grow stronger!
• We finished this repo in early May but was released 7 months later.

1. Concepts:

• Agents can be either LLMs, NN-based experts, APIs and role-players. They all communicate in natural language.
• To solve tasks, these agents use a collaborative "Mindstorm" process involving mutual interviews.
• Additional components for NLSOM can be easily added in a modular way.
• <details> <summary>More insights 👈 <b>[CLICK]</b></summary> <p> <ul> <li><b> Concepts of NLSOMs:</b> Both Minsky’s “Society of Mind” and Schmidhuber’s “Learning to Think” inspire diverse societies of large multimodal neural networks (NNs) that solve problems by interviewing each other in a “Mindstorm.” Recent implementations of NN-based societies of minds consist of large language models (LLMs) and other NN-based experts communicating through a natural language interface. In doing so, they overcome the limitations of single LLMs, improving multimodal zero-shot reasoning. In these natural language-based societies of mind (NLSOMs), new agents—all communicating through the same universal symbolic language—are easily added in a modular fashion. </li> </ul> </p> <p> <ul> <li><b> Process:</b> In this GitHub project, 1) our first step is to recommend relevant communities and agents aligned with the users' goals. These recommendations encompass tools, plugins, models, and role-players, which can then be loaded accordingly. 2) Additionally, we employ a Mindstorm approach, setting ourselves apart from previous models like VisualChatGPT and HuggingGPT, which solely rely on a single model for a specific function. Within our framework, a community of agents shares a common function, such as "search", while each agent possesses unique strengths and capabilities. For instance, agents like "Bing Search," "Wikipedia," "arXiv," and "WolframAlpha" collaborate to provide a more comprehensive understanding. 3) Furthermore, we emphasize the importance of reward mechanisms. In our current implementation, we reward different models for their contributions to task completion, which serves as a valuable means of evaluating a model's usefulness for specific tasks. </li> </ul> </p> <p> <ul> <li><b> Outlook:</b> The concept behind this project is to provide a preliminary NLSOM solution. To illustrate, let's consider a scenario where a non-technical individual lacks expertise in computer science and has limited knowledge about various AI plugins. In such cases, the NLSOM system can automatically recommend different communities and agents to automate tasks based on user input in natural language. Even without understanding the performance of a specific model, the user can benefit from multiple models with the same functionality working together to offer more comprehensive answers. These AI communities can collaborate and compete with each other. Furthermore, the NLSOM system implements a reward mechanism that grants AI models a higher status based on their performance. This reward mechanism serves the purpose of optimizing the end-to-end recommendation/Mindstorm process in the future. It establishes a system akin to a Darwinian meritocracy within the AI community, where models compete and succeed based on their demonstrated excellence. </li> </ul> </p>

2. About this repo:

This project is the technical extension for the original NLSOM paper, including:

• 🧰 Recommendation: Autonomously select communities and agents to form a self-organized NLSOM for solving the specified task.
• 🧠 Mindstorm: Multiple agents (models or APIs) can collaborate to solve tasks together more efficiently.
• 💰 Reward: Rewards are given to all agents involved.

3. Features:

• Manage Easily: Simply change the template to organize your NLSOM in different areas.
• Easy to extend: customize your own community and agents (Now we have 16 communities and 34 agents, see society).
• Reward Design: provide a reward mechanism (albeit rough). You can easily upgrade to a more refined version.
• Elegant UI: has an interface and support for diverse file sources (image, text, audio, video, etc).

💾 Usage

1. Install

Choose from three different installation methods to find the one that best fits your needs.

1. CONDA: conda env create -n nlsom -f nlsom.yaml

2. PIP: conda create -n nlsom python=3.8 and then pip install -r requirements.txt

# [Set Conda Env] 
conda create -n nlsom python=3.8
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 -c pytorch
pip install pandas==1.4.3
# [Set LangChain, OpenAI]
pip install langchain==0.0.158
pip install sqlalchemy==2.0.12
pip install openai
pip install colorama
# [Set Streamlit]
cd assets && unzip validators-0.20.0.zip
cd validators-0.20.0
python setup.py build
python setup.py install
pip install streamlit==1.22.0
pip install streamlit_chat==0.0.2.2
pip install soundfile
# [Set Huggingface/transformers]
pip install transformers==4.29.2
pip install accelerate==0.19.0
# [Set Search]
pip install wolframalpha
pip install wikipedia
pip install arxiv
# [Set Modelscope]
pip install modelscope==1.6.0
python3 -m pip install nvidia-cudnn-cu11==8.6.0.163 tensorflow==2.12.*
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
pip install modelscope[multi-modal]
pip install decord==0.6.0
pip install fairseq
pip install librosa
pip install setuptools==59.5.0
pip install tensorboardX
pip install open_clip_torch
# [Set OCR]
pip install easyocr
# [Set Text-to-Video]
pip install replicate==0.8.3
# [Set Image-to-3D]
pip install trimesh
pip3 install pymcubes
# [Set TTS] - not recommended due to environmental conflicts
pip install TTS
pip install protobuf==3.20.3

• Create the checkpoints dir

mkdir checkpoints && cd checkpoints
mkdir huggingface
mkdir modelscope

• Change Huggingface's setting

>>> import transformers
>>> print(transformers.__file__)
# Get the path: {YOUR_ANACONDA_PATH}/envs/nlsom/lib/python3.8/site-packages/transformers/__init__.py

Open the {YOUR_ANACONDA_PATH}/envs/nlsom/lib/python3.8/site-packages/transformers/utils/hub.py and change the line:

torch_cache_home = os.getenv("TORCH_HOME", os.path.join(os.getenv("XDG_CACHE_HOME", "{YOUR_NLSOM_PATH}/checkpoints"), "torch"))
hf_cache_home = os.path.expanduser(
   os.getenv("HF_HOME", os.path.join(os.getenv("XDG_CACHE_HOME", "{YOUR_NLSOM_PATH}/checkpoints"), "huggingface"))
)

• Similarly, the modelscope's setting

>>> import modelscope
>>> print(modelscope.__file__)
# Get the path: ${YOUR_ANACONDA_PATH}/envs/nlsom/lib/python3.8/site-packages/modelscope/__init__.py

Open {YOUR_ANACONDA_PATH}/envs/nlsom/lib/python3.8/site-packages/modelscope/utils/file_utils.py and change the line:

default_cache_dir = Path.home().joinpath('{YOUR_NLSOM_PATH}/checkpoints', 'modelscope')

2. APIs

Please complete the API keys in .env.template. The OpenAI API key is mandatory, while the others depend on your specific requirements. Then, mv .env.template .env

3. App

streamlit run app.py

🧸 Demo

1. Focus more on Mindstorm

2. Focus more on NLSOM

☑️ TODO?

We adopt two ways to conduct NLSOM and Mindstorm:

v1.0: 📋 Preliminary Experiments: In the original paper, NLSOM and Mindstorm is driven by hardcodes.

v2.0: 📋 In this version, NLSOM is self-organized, and Mindstorm happens automatically.

v3.0: 🎯 Future Work: 1) introducing RL; 2) Economy of Minds; 3) Self-Improvement; etc.

💌 Acknowledgments

This project utilizes parts of code from the following open-source repositories: langchain, BabyAGI, TaskMatrix, DataChad, streamlit. We also thank great AI platforms and all the used models or APIs: huggingface, modelscope.

:black_nib: Citation

References to cite:

@article{zhuge2023mindstorms,
  title={Mindstorms in Natural Language-Based Societies of Mind},
  author={Zhuge, Mingchen and Liu, Haozhe and Faccio, Francesco and Ashley, Dylan R and Csord{\'a}s, R{\'o}bert and Gopalakrishnan, Anand and Hamdi, Abdullah and Hammoud, Hasan and Herrmann, Vincent and Irie, Kazuki and Kirsch, Louis and Li, Bing and Li, Guohao and Liu, Shuming and Mai, Jinjie and Pi{\k{e}}kos, Piotr and Ramesh, Aditya and Schlag, Imanol and Shi, Weimin and Stani{\'c}, Aleksandar and Wang, Wenyi and Wang, Yuhui and Xu, Mengmeng and Fan, Deng-Ping and Ghanem, Bernard and Schmidhuber, J{\"u}rgen},
  journal={arXiv preprint arXiv:2305.17066},
  year={2023}
}