Home

Awesome

<h1 align="center"> <div> <a href="https://maxun-website.vercel.app/"> <img src="/src/assets/maxunlogo.png" width="50" /> <br> Maxun </a> </div> Open-Source No-Code Web Data Extraction Platform <br> </h1> <p align="center"> Maxun lets you train a robot in 2 minutes and scrape the web on auto-pilot. Web data extraction doesn't get easier than this! </p> <p align="center"> <a href="https://maxun-website.vercel.app/"><b>Website</b></a> | <a href="https://discord.gg/5GbPjBUkws"><b>Discord</b></a> | <a href="https://x.com/maxun_io"><b>Twitter</b></a> | <a href="https://docs.google.com/forms/d/e/1FAIpQLSdbD2uhqC4sbg4eLZ9qrFbyrfkXZ2XsI6dQ0USRCQNZNn5pzg/viewform"><b>Join Maxun Cloud</b></a> | <a href="https://www.youtube.com/@MaxunOSS"><b>Watch Tutorials</b></a> <br /> <br /> <a href="https://trendshift.io/repositories/12113" target="_blank"><img src="https://trendshift.io/api/badge/repositories/12113" alt="getmaxun%2Fmaxun | Trendshift" style="width: 250px; height: 55px; margin-top: 10px;" width="250" height="55"/></a> </p>

maxun_demo

<img src="https://static.scarf.sh/a.png?x-pxid=c12a77cc-855e-4602-8a0f-614b2d0da56a" />

Installation

  1. Create a root folder for your project (e.g. 'maxun')
  2. Create a file named .env in the root folder of the project
  3. Example env file can be viewed here. Copy all content of example env to your .env file.
  4. Choose your installation method below

Docker Compose

  1. Copy paste the docker-compose.yml file into your root folder
  2. Ensure you have setup the .env file in that same folder
  3. Run the command below from a terminal
docker-compose up -d

You can access the frontend at http://localhost:5173/ and backend at http://localhost:8080/

Without Docker

  1. Ensure you have Node.js, PostgreSQL, MinIO and Redis installed on your system.
  2. Run the commands below
git clone https://github.com/getmaxun/maxun

# change directory to the project root
cd maxun

# install dependencies
npm install

# change directory to maxun-core to install dependencies
cd maxun-core 
npm install

# get back to the root directory
cd ..

# make sure playwright is properly initialized
npx playwright install
npx playwright install-deps

# get back to the root directory
cd ..

# start frontend and backend together
npm run start

You can access the frontend at http://localhost:5173/ and backend at http://localhost:8080/

Environment Variables

  1. Create a file named .env in the root folder of the project
  2. Example env file can be viewed here.
VariableMandatoryDescriptionIf Not Set
BACKEND_PORTYesPort to run backend on. Needed for Docker setupDefault value: 8080
FRONTEND_PORTYesPort to run frontend on. Needed for Docker setupDefault value: 5173
BACKEND_URLYesURL to run backend on.Default value: http://localhost:8080
VITE_BACKEND_URLYesURL used by frontend to connect to backendDefault value: http://localhost:8080
PUBLIC_URLYesURL to run frontend on.Default value: http://localhost:5173
VITE_PUBLIC_URLYesURL used by backend to connect to frontendDefault value: http://localhost:5173
JWT_SECRETYesSecret key used to sign and verify JSON Web Tokens (JWTs) for authentication.JWT authentication will not work.
DB_NAMEYesName of the Postgres database to connect to.Database connection will fail.
DB_USERYesUsername for Postgres database authentication.Database connection will fail.
DB_PASSWORDYesPassword for Postgres database authentication.Database connection will fail.
DB_HOSTYesHost address where the Postgres database server is running.Database connection will fail.
DB_PORTYesPort number used to connect to the Postgres database server.Database connection will fail.
ENCRYPTION_KEYYesKey used for encrypting sensitive data (proxies, passwords).Encryption functionality will not work.
MINIO_ENDPOINTYesEndpoint URL for MinIO, to store Robot Run Screenshots.Connection to MinIO storage will fail.
MINIO_PORTYesPort number for MinIO service.Connection to MinIO storage will fail.
MINIO_CONSOLE_PORTNoPort number for MinIO WebUI service. Needed for Docker setup.Cannot access MinIO Web UI.
MINIO_ACCESS_KEYYesAccess key for authenticating with MinIO.MinIO authentication will fail.
GOOGLE_CLIENT_IDNoClient ID for Google OAuth, used for Google Sheet integration authentication.Google login will not work.
GOOGLE_CLIENT_SECRETNoClient Secret for Google OAuth.Google login will not work.
GOOGLE_REDIRECT_URINoRedirect URI for handling Google OAuth responses.Google login will not work.
REDIS_HOSTYesHost address of the Redis server, used by BullMQ for scheduling robots.Redis connection will fail.
REDIS_PORTYesPort number for the Redis server.Redis connection will fail.
MAXUN_TELEMETRYNoDisables telemetry to stop sending anonymous usage data. Keeping it enabled helps us understand how the product is used and assess the impact of any new changes. Please keep it enabled.Telemetry data will not be collected.

How Does It Work?

Maxun lets you create custom robots which emulate user actions and extract data. A robot can perform any of the actions: <b>Capture List, Capture Text or Capture Screenshot. Once a robot is created, it will keep extracting data for you without manual intervention</b>

Screenshot 2024-10-23 222138

1. Robot Actions

  1. Capture List: Useful to extract structured and bulk items from the website. Example: Scrape products from Amazon etc.
  2. Capture Text: Useful to extract individual text content from the website.
  3. Capture Screenshot: Get fullpage or visible section screenshots of the website.

2. BYOP

BYOP (Bring Your Own Proxy) lets you connect external proxies to bypass anti-bot protection. Currently, the proxies are per user. Soon you'll be able to configure proxy per robot.

Features

Cloud

We offer a managed cloud version to run Maxun without having to manage the infrastructure and extract data at scale. Maxun cloud also deals with anti-bot detection, huge proxy network with automatic proxy rotation, and CAPTCHA solving. If this interests you, join the cloud waitlist as we launch soon.

Screenshots

Maxun PH Launch (1)-1-1 Maxun PH Launch (1)-2-1 Maxun PH Launch (1)-3-1 Maxun PH Launch (1)-4-1 Maxun PH Launch (1)-5-1 Maxun PH Launch (1)-6-1 Maxun PH Launch (1)-7-1 Maxun PH Launch (1)-8-1 Maxun PH Launch (1)-9-1

Note

This project is in early stages of development. Your feedback is very important for us - we're actively working to improve the product. <a href="https://forms.gle/E8vRMVB7bUbsSktPA">Drop anonymous feedback here.</a>

License

<p> This project is licensed under <a href="./LICENSE">AGPLv3</a>. </p>

Contributors

Thank you to the combined efforts of everyone who contributes!

<a href="https://github.com/getmaxun/maxun/graphs/contributors"> <img src="https://contrib.rocks/image?repo=getmaxun/maxun" /> </a>