Home

Awesome

<img src="https://github.com/Helsinki-NLP/Opus-MT/blob/master/img/opus_mt.png" width="250" alt="OPUS-MT"/>

Tools and resources for open translation services

<p> <a href="https://console.tiyaro.ai/explore?q=opus-mt&pub=Helsinki-NLP"> <img src="https://tiyaro-public-docs.s3.us-west-2.amazonaws.com/assets/try_on_tiyaro_badge.svg"></a> </p>

This repository includes two setups:

There are also scripts for training models, but those are currently only useful in the computing environment used by the University of Helsinki and CSC as the IT service provider.

Please cite the following papers if you use OPUS-MT software and models:

@article{tiedemann2023democratizing,
  title={Democratizing neural machine translation with {OPUS-MT}},
  author={Tiedemann, J{\"o}rg and Aulamo, Mikko and Bakshandaeva, Daria and Boggia, Michele and Gr{\"o}nroos, Stig-Arne and Nieminen, Tommi and Raganato\
, Alessandro and Scherrer, Yves and Vazquez, Raul and Virpioja, Sami},
  journal={Language Resources and Evaluation},
  number={58},
  pages={713--755},
  year={2023},
  publisher={Springer Nature},
  issn={1574-0218},
  doi={10.1007/s10579-023-09704-w}
}

@InProceedings{TiedemannThottingal:EAMT2020,
  author = {J{\"o}rg Tiedemann and Santhosh Thottingal},
  title = {{OPUS-MT} — {B}uilding open translation services for the {W}orld},
  booktitle = {Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT)},
  year = {2020},
  address = {Lisbon, Portugal}
 }

Installation of the Tornado-based Web-App

Download the latest version from github:

git clone https://github.com/Helsinki-NLP/Opus-MT.git

Option 1: Manual setup

Install Marian MT. Follow the documentation at https://marian-nmt.github.io/docs/ (don't forget to include the cmake option for compiling the server binary -DCOMPILE_SERVER=ON) After the installation, marian-server is expected to be present in path. If not, place it in /usr/local/bin

Install pre-requisites. Using a virtual environment is recommended.

pip install -r requirements.txt

Download the translation models from https://github.com/Helsinki-NLP/Opus-MT-train/tree/master/models and place them in models directory.

Then edit the services.json to point to those models.

And start the webserver.

python server.py

By default, it will use port 8888. Launch your browser to localhost:8888 to get the web interface. The languages configured in services.json will be available.

Option 2: Using Docker

docker-compose up

or

docker build . -t opus-mt
docker run -p 8888:8888 opus-mt:latest

And launch your browser to localhost:8888

Option 2.1: Using Docker with CUDA GPU

docker build -f Dockerfile.gpu . -t opus-mt-gpu
nvidia-docker run -p 8888:8888 opus-mt-gpu:latest

And launch your browser to localhost:8888

Configuration

The server.py program accepts a configuration file in json format. By default it try to use services.json in the current directory. But you can give a custom one using -c flag.

An example configuration file looks like this:

{
    "en": {
        "es": {
            "configuration": "./models/en-es/decoder.yml",
            "host": "localhost",
            "port": "10001"
        },
        "fi": {
            "configuration": "./models/en-fi/decoder.yml",
            "host": "localhost",
            "port": "10002"
        },
    }
}

This example configuration can provide MT service for en->es and en->fi language pairs.

Installation of a websocket service on Ubuntu

There is another option of setting up translation services using WebSockets and Linux services. Detailed information is available from doc/WebSocketServer.md.

Public MT models

We store public models (CC-BY 4.0 License) at https://github.com/Helsinki-NLP/Opus-MT-train/tree/master/models They should all be compatible with the OPUS-MT services, and you can install them by specifying the language pair. The installation script takes the latest model in that directory. For additional customisation you need to adjust the installation procedures (in the Makefile or elsewhere).

There are also development versions of models, which are often a bit more experimental and of low quality. But there are additional language pairs and they can be downloaded from https://github.com/Helsinki-NLP/Opus-MT-train/tree/master/work-spm/models

Train MT models

There is a Makefile for training new models from OPUS data in the Opus-MT-train repository, but this is heavily customized for the work environment at CSC and the University of Helsinki projects. This will (hopefully) be more generic in the future to be able to run in different environments and setups as well.

Known issues

To-Do and wish list

Links and related work

Acknowledgements

The work is supported by the European Language Grid as pilot project 2866, by the FoTran project, funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 771113), and the MeMAD project, funded by the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement No 780069. We are also grateful for the generous computational resources and IT infrastructure provided by CSC -- IT Center for Science, Finland.