Home

Awesome

<!--- Copyright 2020 The AdapterHub Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

IMPORTANT NOTE

This is the legacy adapter-transformers library, which has been replaced by the new Adapters library, found here: https://github.com/adapter-hub/adapters.

Install the new library via pip: pip install adapters.

This repository is kept for archival purposes, and will not be updated in the future. Please use the new library for all active projects.

The documentation of this library can be found at https://docs-legacy.adapterhub.ml. The documentation of the new Adapters library can be found at https://docs.adapterhub.ml. For transitioning, please read: https://docs.adapterhub.ml/transitioning.html.


<p align="center"> <img style="vertical-align:middle" src="https://raw.githubusercontent.com/Adapter-Hub/adapter-transformers/master/adapter_docs/logo.png" /> </p> <h1 align="center"> <span>adapter-transformers</span> </h1> <h3 align="center"> A friendly fork of HuggingFace's <i>Transformers</i>, adding Adapters to PyTorch language models </h3>

Tests GitHub PyPI

adapter-transformers is an extension of HuggingFace's Transformers library, integrating adapters into state-of-the-art language models by incorporating AdapterHub, a central repository for pre-trained adapter modules.

💡 Important: This library can be used as a drop-in replacement for HuggingFace Transformers and regularly synchronizes new upstream changes. Thus, most files in this repository are direct copies from the HuggingFace Transformers source, modified only with changes required for the adapter implementations.

Installation

adapter-transformers currently supports Python 3.8+ and PyTorch 1.12.1+. After installing PyTorch, you can install adapter-transformers from PyPI ...

pip install -U adapter-transformers

... or from source by cloning the repository:

git clone https://github.com/adapter-hub/adapter-transformers.git
cd adapter-transformers
pip install .

Getting Started

HuggingFace's great documentation on getting started with Transformers can be found here. adapter-transformers is fully compatible with Transformers.

To get started with adapters, refer to these locations:

Implemented Methods

Currently, adapter-transformers integrates all architectures and methods listed below:

MethodPaper(s)Quick Links
Bottleneck adaptersHoulsby et al. (2019)<br> Bapna and Firat (2019)Quickstart, Notebook
AdapterFusionPfeiffer et al. (2021)Docs: Training, Notebook
MAD-X,<br> Invertible adaptersPfeiffer et al. (2020)Notebook
AdapterDropRücklé et al. (2021)Notebook
MAD-X 2.0,<br> Embedding trainingPfeiffer et al. (2021)Docs: Embeddings, Notebook
Prefix TuningLi and Liang (2021)Docs
Parallel adapters,<br> Mix-and-Match adaptersHe et al. (2021)Docs
CompacterMahabadi et al. (2021)Docs
LoRAHu et al. (2021)Docs
(IA)^3Liu et al. (2022)Docs
UniPELTMao et al. (2022)Docs

Supported Models

We currently support the PyTorch versions of all models listed on the Model Overview page in our documentation.

Citation

If you use this library for your work, please consider citing our paper AdapterHub: A Framework for Adapting Transformers:

@inproceedings{pfeiffer2020AdapterHub,
    title={AdapterHub: A Framework for Adapting Transformers},
    author={Pfeiffer, Jonas and
            R{\"u}ckl{\'e}, Andreas and
            Poth, Clifton and
            Kamath, Aishwarya and
            Vuli{\'c}, Ivan and
            Ruder, Sebastian and
            Cho, Kyunghyun and
            Gurevych, Iryna},
    booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
    pages={46--54},
    year={2020}
}