Awesome
Qlib-Server
is the data server system for Qlib
. It enable Qlib
to run in online
mode. Under online mode, the data will be deployed as a shared data service. The data and their cache will be shared by all the clients. The data retrieval performance is expected to be improved due to a higher rate of cache hits. It will consume less disk space, too.
To sum up, Qlib-Server
is designed to solve the following problems:
- Manage the data in a centralized way, which makes data management (including cache management, date updating) much easier.
- Reduce the amount of cache to be generated.
- Make the client light-weighted and leverage the powerful computing resources of remote server
Framework of Qlib-Server
<div style="align: center"> <img src="docs/_static/img/framework.png" /> </div>The Client/Server
framework of Qlib
is based on WebSocket
considering its capability of bidirectional communication between client and server in async mode.
Qlib-Server
is based on Flask, which is a micro-framework for Python and here Flask-SocketIO is used for websocket connection.
Quick start
Deployment
One-click Deployment
One-click deployment of Qlib-Server
is supported, you can choose either of the following two methods for one-click deployment:
- Deployment with
docker-compose
- Deployment in
Azure
One-click Deployment with docker-compose
Deploy Qlib-Server
with docker-compose
according to the following processes:
-
Install
docker
, please refer to Docker Installation. -
Install
docker-compose
, please refer to Docker-compose Installation. -
Run the following command to deploy
Qlib-Server
:git clone https://github.com/microsoft/qlib-server cd qlib-server sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env build sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env up -d # Use the following command to track the log sudo docker-compose -f docker_support/docker-compose.yaml --env-file docker_support/docker-compose.env logs -f
One-click Deployment in Azure
Firstly, You need to have an Azure
account to deploy Qlib-Server
in Azure
. Then you can deploy Qlib-Server
in Azure
according to the following processes:
-
Install
azure-cli
, please refer to install-azure-cli. -
Add the
Azure
account to the configuration fileazure_conf.yaml
sub_id: Your Subscription ID username: azure user name password: azure password # The resource group where the VM is located resource_group: Resource group name
-
Execute the deployment script by running the following command:
git clone https://github.com/microsoft/qlib-server cd qlib-server/scripts python azure_manager.py create_qlib_cs_vm \ --qlib_server_name test_server01 \ --qlib_client_names test_client01 \ --admin_username test_user \ --ssh_key_value ~/.ssh/id_rsa.pub \ --size standard_NV6_Promo\ --conf_path azure_conf.yaml
To know more about one-click Deployment, please refer to Qlib-Server One-click Deployment.
Step-by-step Deployment
To know more about step-by-step Deployment, please refer to [Qlib-Server Step-by-step Deplyment]https://qlib-server.readthedocs.io/en/latest/build.html#step-by-step-deployment).
Using Qlib
in Online
Mode
In the Qlib Document, the Offline
mode has been introduced.
With Qlib-Server
, you can use Qlib
in Online
mode, please initialize Qlib
with the following code:
import qlib
ONLINE_CONFIG = {
# data provider config
"calendar_provider": {"class": "LocalCalendarProvider", "kwargs": {"remote": True}},
"instrument_provider": "ClientInstrumentProvider",
"feature_provider": {"class": "LocalFeatureProvider", "kwargs": {"remote": True}},
"expression_provider": "LocalExpressionProvider",
"dataset_provider": "ClientDatasetProvider",
"provider": "ClientProvider",
# config it in user's own code
"provider_uri": "127.0.0.1:/",
# cache
# Using parameter 'remote' to announce the client is using server_cache, and the writing access will be disabled.
"expression_cache": None,
"dataset_cache": None,
"calendar_cache": None,
"mount_path": "/data/stock_data/qlib_data",
"auto_mount": True, # The nfs is already mounted on our server[auto_mount: False].
"flask_server": "127.0.0.1",
"flask_port": 9710,
"region": "cn",
}
qlib.init(**client_config)
ins = D.list_instruments(D.instruments("all"), as_list=True)
For more details, please refer to Qlib-Server Client.
More About Qlib-Server
The detailed documents are organized in docs. Sphinx and the readthedocs theme is required to build the documentation in html formats.
cd docs/
conda install sphinx sphinx_rtd_theme -y
# Otherwise, you can install them with pip
# pip install sphinx sphinx_rtd_theme
make html
You can also view the latest document online directly.
Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.