Home

Awesome

REST Commander: Parallel Async HTTP Client as a Service Travis status

Formerly known as REST Superman. Fire thousands of HTTP requests and aggregate responses in a couple of clicks in seconds. Please check detail instructions, screenshots, documentations, sample code, REST APIs, and demos at www.restcommander.com and its demo video. What's new? Check related work review on efficient HTTP clients and concurrency and throttling model in Akka at eBay tech blog.

REST Commander is a fast parallel async HTTP/REST/SOAP client as a service to monitor and manage 10,000s of web servers. Sends requests to 1000 servers with response aggregation in 10 seconds. or 10,000 servers in 50 seconds.

Need standalone library alternative? Try Parallec.io (released 2015.11). With the feedbacks, lessons, and improvements from the past year of internal usage and open source of REST Commander, we now made its core as an easy to use standalone library. We added 15+ new features, rewritten 70%+ of the code, with 90%+ test coverage for confident usage and contribution. The key benefits of Parallec are flexible response aggregation and ease to send the results anywhere.

REST Commander serves as the agent master of cronus-agent (open sourced) for scalable software deployment, script execution, config push, and monitoring.

Version 2.0.0 : faster than faster, by restructuring via AKKA remoting and clustering, we are able to make REST Commander distributed and horizontally scalable. Distributed REST Commander can send request to 100K+ machines in eBay's cloud and gather result back in just 100s using 5 VM.

Version 2.0.0 | What and Why | Highlights | Performance | Run Instructions

REST Commander has been in top 10 trending out of 10 millions+ projects in Github in all languages on 01/21/2014 and 01/22/2014. It has been recommended and listed in top 20 trending out of 28K+ software in oschina, the largest open source community in China. It has also been featured and front-paged at InfoQ.

Structure Overview

<a name="a_V2"></a>

Version 2.0.0 : distributed REST Commander

We restructure REST Commander via AKKA remoting and clustering. Detailed information and API document at here. Workflow and architecture design details at here. See git branch distributed_commander.

New Features

Workflow & Architecture

The main workflow of distributed REST Commander is shown as below.

Work Flow

Distributed REST Commander is based on AKKA. Each functionality component in the pictrue above is implemented as an AKKA actor (except Job Manager). The whole system is based on message passing model.

<a name="a_whatAndWhy"></a>

What is REST Commander and Why I need it?

Commander is Postman at scale: a fast parallel async http client as a service with aggregated response and regular expression based string extraction. It is in Java (with Akka and Play Framework).

So what can Commander do? It speaks HTTP at scale, thus is powerful with many use cases. Here are some basic ones for automation on managing and monitoring tens of thousands web servers (See Sample Code). Commander itself is also "as a service": with its powerful REST API, you can define ad-hoc target servers, an HTTP request template, variable replacement, and a regular expression all in a single call.

Whenever comes to sending multiple HTTP requests in parallel, federated data aggregation or scalable task executions on HTTP, Think Commander First.

Highlights

Commander is powerful to send (1) the same request to different servers; (2) different requests to different servers; (3) different requests to the same server. Why we need them? Check out these live examples on (1) monitor websites; (2) poll job status (3) call the same weather WSDL web service with different zip codes.

<a name="a_performance"></a>

Performance (SLA)

Run Instructions<a name="a_runInstructions"></a>

Directly Under Windows/Linux With Zero Installation:

WINDOWS
LINUX or MAC

Run/Debug With Eclipse:

Settings

Key files are under conf folder

Troubleshooting

About REST Commander

Motivation:

REST and SOAP API (HTTP GET/POST/PUT/DELETE) has become the dominant approach in current platform and services. However, efficient REST / SOAP calls to multiple servers in parallel with server-specific requests and aggregated response analysis are still challenging.

Problem Statement:

Design and implement a user friendly and generic HTTP client able to conduct efficient HTTP calls to a large amount of servers in parallel with uniform or server-specific requests and aggregated response analysis.

Impact

REST API Example

REST Commander supports both intuitive step-by-step wizards and REST APIs. Here is an simple example of uniform request to 3 target servers. In this example, the command and aggregation rule have been pre-defined. More complex API examples using none pre-defined command or aggregation rules can be found here.

Request: (assuming Commander runs on localhost:9000)

HTTP POST to: http://localhost:9000/commands/genUpdateSendCommandWithReplaceVarMapAdhocJson POST Body:

{
   "targetNodes":[
	  "www.restcommander.com",
	  "www.jeffpei.com",
	  "www.yangli907.com"
   ],
  
  "useNewAgentCommand":"false",
  "agentCommandType":"GET_VALIDATE_INTERNALS",
   "willAggregateResponse":true,
   "useNewAggregation":false,
   "aggregationType":"PATTERN_VI_SERVER_CPU",
   "replacementVarMap":{}
}

Response:

{
	"aggregationMap": {
		"23.54": "1",
		"27.08": "1",
		"7.08": "1"
	},
	"aggregationValueToNodesList": [
		{
			"value": "23.54",
			"nodeList": [
				"www.yangli907.com"
			],
			"isError": false
		},
		{
			"value": "27.08",
			"nodeList": [
				"www.jeffpei.com"
			],
			"isError": false
		},
		{
			"value": "7.08",
			"nodeList": [
				"www.restcommander.com"
			],
			"isError": false
		}
	]
}