Home

Awesome

Kubespider

<p align="center"> <img src="./docs/images/logo.png" width="200" alt="Kubespider logo"> </p>

💥 A global resource download orchestration system

Python GitHub release Total Downloads GitHub stars GitHub license

English | 简体中文

Installation | Configuration | Developer Group

We are a community of individuals who share a passion for life and have come together based on shared interests and needs. In our free time, we collaborated to develop Kubespider.

Kubespider is developed to utilize an idle server in a local area network as a NAS, enabling automatic downloads of TV series, triggering downloads from a local laptop, and adapting to various websites such as YouTube and BiliBili, as well as different types of resources such as TV series, movies, music and more.

After being exposed to Terraform and its great versatility, we were inspired to create Kubespider as a general download orchestration system that is compatible with various resource platforms and download software. Kubespider supports multiple download methods, including request trigger, cycle trigger, and update trigger, making it the most comprehensive and unified solution for resource downloads.

⛵ System architecture

To realize a general download orchestration system, it is necessary to adapt various resource websites and download software, so these two parts are abstracted from the core functions: image

💽 Installation

To adapt to multiple resource platforms, Kubespider provides many adapters. Enable or disable these adapters according to your needs. The configuration file is in the .config folder.

Premise

  1. The operating computer and your server are on the same LAN. (I haven't tried the installation not in the same LAN yet)
  2. The server is a Linux system.
  3. Docker is installed on the server.

Default installation (with docker)

Installing with built-in command can download and config anything needed automatically.

1.Download the repository and install

# Define KUBESPIDER_HOME to specify the installation path
# export KUBESPIDER_HOME=xxx
git clone https://github.com/opennaslab/kubespider.git
cd kubespider
bash hack/install_kubespider.sh

Output like:

root@cesign [09:01:34 PM] [+22.0°C] [~/git/kubespider] [main *]
-> # bash hack/install_kubespider.sh
 _          _                     _     _
| | ___   _| |__   ___  ___ _ __ (_) __| | ___ _ __
| |/ / | | | '_ \ / _ \/ __| '_ \| |/ _` |/ _ \ '__|
|   <| |_| | |_) |  __/\__ \ |_) | | (_| |  __/ |
|_|\_\\__,_|_.__/ \___||___/ .__/|_|\__,_|\___|_|
                           |_|
[INFO] Start to deploy with default configuration...
b13820946878c4f00635e7fa3db64ea83506850ebb7d07ff77b62814db1d894a
9c0aa1059f1546a4f1a2accac445ce1389f1c400b96328f3e18c8af03f0bbc70
[INFO] Deploy successful, check the information:
*******************************************
Kubespider config path: /root/kubespider/.config/
Download file path: /root/kubespider/nas/
Kubespider webhook address: http://<server_ip>:3080
Aria2 server address: http://<server_ip>:6800/jsonrpc, you can use any gui or webui to connect it
Aria2 default secret is:kubespider
*******************************************

2.Connect to aria2

For checking the download tasks on your server conveniently, go to the chrome store to download AriaNg plugin. Then connect, the plugin configuration is as follows(shown at the end of the script install_kubespider.sh execution output): img

Of course, it is not necessary, you can also use the desktop program to connect: AriaNg

3.Install Chrome plugin

With Kubespider Chrome plugin, you can get a better experience.

After step 1 has been done, open chrome, then install Kubespider plugin, and configure it as follows(The server address is http://<server_ip>:3080): img

For example, if I want to download a torrent-related file, I can right click and send the download task to Kubespider(demo website): img

Then with aria2 chrome plugin, you can see the download task starts: img

4.More (Optional)

Other installation

📝 Configuration

The global configuration file is located at .config/kubespider.yaml, which is installed under ${HOME}/kubespider/.config/kubespider.yaml by default. The description of each configuration item is as follows:

Configuration ItemRequiredFunction
proxyNoThe request proxy, such as http://192.168.1.5:1087
auth_tokenNoAuthentication token, used for authentication when triggering download requests. If public network access is required, it can be configured to increase security.
auto_change_download_providerNoSwith to control whether to change download provoider for failed/long-time-pending download tasks automatically.
server_portNoThe port used to provider API service

🔧 Feature list

Source provider(Configured as needed)

As mentioned before, the source provider is used to adapt to different websites, and download related resources based on the conditions:

Source ProviderSupportPurposeConfiguration
mikanani source providerYesAutomatically download updated animelink
btbtt12 source providerYesTrigger download resource in btbtt12 websitelink
meijutt source providerYesDownload updated TV serieslink
YouTube source providerYesTrigger download YouTube videolink
Bilibili source providerYesTrigger download bilibili videolink
Bilibili vlogger subscribe source providerYesAutomatically download updated Bilibili vlogger's videolink
general rss source providerYesAutomatically download rss sourcelink
Magic source providerYesTrigger download resource via xpath configurationlink
Douyin source providerYesTrigger download Douyin video with watermarklink
alist source providerYesAutomatically download newly added or updated files from the cloud storagelink
ANi source providerYesFully automated anime download from ANilink

Download provider(Configured as needed)

In order to make full use of the existing download software in the open-source community(For example, YouTube has dedicated download software), the following software is currently supported:

Download SoftwareSupportConfiguration
aria2Yeslink
xunleiYeslink
qBittorrentYeslink
you-getYeslink
yt-dlpYeslink
transmissionYeslink
yuttoYeslink

Notification provider(Configured as needed)

In order to make full use of the existing notification software in the open-source community, the following software is currently supported:

Notification SoftwareSupportConfiguration
PushdeerYeslink
TelegramYeslink
QQYeslink
BarkYeslink

Basic feature

Currently, the following file types are supported to trigger downloading:

Download TypesSupport
magnetYes
torrentYes
general(jpg, mp4, etc.)Yes

Please refer to link for how to trigger the download.

Other

🙋 FAQ

See the FAQ for more details, and feel free to contribute your questions and solutions. Your valuable input will collectively improve the documentation and benefit a wider audience. Join us in making a difference!

🤝 Contribution PRs Welcome

If you are interested in participating in joint development, welcome to STAR/FORK/PR, please check link for Roadmap.

If you want to participate more in the community, welcome to join our Telegram developer group.

❤️ Sponsor

Thanks to the following companies for sponsoring development tools, the ranking is not in any particular order:

<table> <thead> <tr> <th align="center" height="60px"> <a href="https://www.jetbrains.com/"> <img align="center" src="./docs/images/sponsor/jetbrains.png" height="40px"><br> </a> </th> <th align="center" height="60px"> <a href="https://www.synology.cn/"> <img align="center" src="./docs/images/sponsor/synology.png" height="40px"><br> </a> </th> <th align="center" height="60px"> <a href="https://www.terra-master.com/"> <img align="center" src="./docs/images/sponsor/terramaster.svg" height="40px"><br> </a> </th> <th align="center" height="60px"> <a href="https://www.qnap.com.cn/"> <img align="center" src="./docs/images/sponsor/qnap.svg" height="40px"><br> </a> </th> <th align="center" height="60px"> <a href="https://www.asustor.com/"> <img align="center" src="./docs/images/sponsor/asustor.png" height="40px"><br> </a> </th> <th align="center" height="60px"> <a href="https://www.ugnas.com/"> <img align="center" src="./docs/images/sponsor/ugnas.png" height="40px"><br> </a> </th> <th align="center" height="60px"> <a href="https://www.jonsbo.com/"> <img align="center" src="./docs/images/sponsor/jonsbo.png" height="40px"><br> </a> </th> </thead> </table>

📃 Disclaimer

All downloading and use of the software are deemed to have carefully read and fully agreed to the following terms: