Awesome
Manticore Buddy
Manticore Buddy is a sidecar for Manticore Search, written in PHP, which helps with various tasks. The typical workflow is that before returning an error to the user, Manticore Search asks Buddy if it can handle it for the daemon. Buddy is written in PHP, which makes it easy to implement high-level functionalities that do not require extra-high performance.
Read articles about Buddy to get a deep dive:
- Introducing Buddy: the PHP sidecar for Manticore Search
- Manticore Buddy: challenges and solutions
- Manticore Buddy: pluggable design
SQL processing
Introduction
To control different SQL commands, we use the following architecture. Each command consists of a prefix and the rest of the parameters. For example, we have "backup to /path/to/folder". In this case, backup
is the command, and to /path/to/folder
is the rest.
Execution flow
Each request is handled by ReactPHP, parsed in the main loop, and forwarded to the QueryProcessor
, which is the main entry point for command detection.
Each command that Buddy can process consists of implementing two interfaces: CommandRequestInterface
and CommandExecutorInterface
.
The CommandRequestInterface
represents the parsing of raw network data from JSON and its preparation for future request processing by the command. Its primary purpose is to parse the data into a CommandRequest
object and throw exceptions in case of any errors.
There is a base class – CommandRequestBase
, that implements the required interface and adds some base logic to all requests we support. You should extend this class when you create a new command request.
The CommandExecutorInterface
contains the logic for the command that must be executed. It uses the Task
class, which is run in parallel in a separate thread to make the process non-blocking.
Exceptions can be thrown when implementing a new command because they are all caught in a loop and processed.
There is a GenericError
class that implements the setResponseError
and getResponseError
methods. If you want to provide a user-friendly error message, you can use the setResponseError
method or create an exception with GenericError::create('User-friendly error message goes here')
. It's important to note that the default exception message will not be included in the user response but in the Manticore log file.
Helper tool to start new command development
We offer a tool that simplifies the process of adding a new command. Using the tool, you can create a new command with a single command-line instruction like this:
bin/create-command NameOfTheCommand
Once you execute this command, all the necessary structures for the NameOfTheCommand
command will be created automatically. All that's left for you to do is to review the files and add your code where necessary. For further information on how this tool works, please refer to the next section.
Steps for creating a new command
Let's take a closer look at an example of how to create an abstract RESTORE command:
- Start by creating a directory with our command namespace
src/Restore
and implementing the Request and Executor classes. - Then, write code that implements the
fromRequest
method in theRequest
class. This method should parse the input network request and return a new Request instance with all the necessary data loaded and ready to be used in theExecutor
. - In the
Executor
, write code that implements the run method and contains all the logic for the command. This method should return an instance of theTask
class. - With the
Task
instance, you can check the status, get the result, and have full control over the asynchronous execution of the command. 5.Finally, add a case for our new command to theextractCommandFromRequest
method of theQueryProcessor
class. This ensures that theQueryProcessor
can recognize and handle requests for theRESTORE
command.
Debug
To debug the command flow, you can use the bin/query
script. To run it, pass the query as an argument. For example:
$ bin/query "BACKUP"
Running query: BACKUP
Manticore config
endpoint = 127.0.0.1:9308
Manticore versions:
manticore: 5.0.3 129438c1c@221013 dev
columnar: 0.0.0
secondary: 0.0.0
2022-10-20 14:26:51 [Info] Starting the backup...
Status code: Finished
Result: Backup directory is not writable
done
This will execute the BACKUP
command and display the results. The output will present the Manticore configuration, versions, and the status and outcome of the command.
Running from CLI
To run a Buddy instance from the command line interface (CLI), use the following command:
$ manticore-executor src/main.php [ARGUMENTS]
Copyright (c) 2024, Manticore Software LTD (https://manticoresearch.com)
Usage: manticore-executor src/main.php [ARGUMENTS]
Arguments are:
--bind Which IP to bind, default is 127.0.0.1
--listen HTTP endpoint to accept Manticore requests
--version display the current version of Buddy
--help display this help message
--telemetry-period=[N] set period for telemetry when we do snapshots
--disable-telemetry disables telemetry for Buddy
--threads=[N] start N threads on launch, default is 4
--debug enable debug mode for testing
--debugv enable verbose debug mode with periodic messages
Examples:
manticore-executor src/main.php --debug
manticore-executor src/main.php --disable-telemetry
You can find more detailed information on the Manticore executor here.
Development
If you want to contribute to the project and develop extra features, we have prepared a particular docker image.
Just go to your "buddy" folder on a host machine and run the following instructions.
docker run --privileged --entrypoint bash -v $(pwd):/workdir --name manticore-buddy -it ghcr.io/manticoresoftware/manticoresearch:test-kit-latest
ghcr.io/manticoresoftware/manticoresearch:test-kit-latest
After that, you can go into the container and work as normal. It has pre-installed Manticoresearchd with the columnar library, production, and development version of executor.
The image is built from Alpine Linux. Please note that you should also run composer install
before running Buddy from the source code with manticore-executor
.
If you want to test Buddy somewhere else (not just Alpine), the easiest way to build it as a PHAR archive.
Ensure you are in the directory where the Buddy repository is cloned. And follow the instructions:
git clone https://github.com/manticoresoftware/phar_builder.git
docker run -rm --entrypoint bash -v $(pwd):/workdir --workdir /workdir --entrypoint bash -it ghcr.io/manticoresoftware/manticoresearch:test-kit-latest
ghcr.io/manticoresoftware/manticoresearch:test-kit-latest -c './phar_builder/bin/build --name="Manticore Buddy" --package="manticore-buddy"
Check the build directory and get the built version of Buddy from there and replace it in your another OS in the "modules" directory.
Run custom process inside the Plugin
To run the process that can maintain some logic and communicate you need to create the Processor
class and add getProcessors
method to the Payload
Here is the example that explain how to do So
Create Processor
plugin and implement required logic
<?php declare(strict_types=1);
… your NS and other copyright here …
use Manticoresearch\Buddy\Core\Process\BaseProcessor;
use Manticoresearch\Buddy\Core\Process\Process;
final class Processor extends BaseProcessor {
public function start(): void {
var_dump('starting');
parent::start();
$this->execute('test', ['simple message']);
}
public function stop(): void {
var_dump('stopping');
parent::stop();
}
public static function test(string $text): void {
var_dump($text);
}
}
Add to the Payload
info that your plugin has processors
public static function getProcessors(): array {
static $processors;
// To ensure the object reference remains unchanged,
// create it as static and keep track of it.
if (!$processors) {
$processors = [new Processor()];
}
return $processors;
}
Communication Protocol v3
This is the protocol description used for communication between Manticore Search and Buddy.
You can find communication protocol v1 here. You can find communication protocol v2 here.
Request from Manticore Search to Buddy
The request from Manticore Search to Buddy is made in JSON format no matter how the original query is made (JSON/SQL/binary). The fields are:
Key | Description |
---|---|
type | Either unknown json request when the original request is made via JSON over HTTP or unknown sql request for SQL over HTTP/mysql. |
error | An object containg information about error(error message, etc.) to be returned to the user, if any. |
message | An object containing details such as path_query (specific to JSON over HTTP requests), http_method (HEAD , GET , etc) and body which holds the main content of the request. For JSON over HTTP, path_query can include specific endpoints like _doc , _create , etc., while for SQL over HTTP/mysql, it remains empty ("" ). http_method is set to "" for SQL over HTTP/mysql |
version | The maximum protocol version supported by the sender. |
Example of the request:
{
"type":"unknown json request",
"error": {
"message":"unknown option 'fuzzy'",
"body":{"error":"unknown option 'fuzzy'"}
},
"message":{
"path_query":"/search",
"body":"{\"index\":\"name\",\"query\":{\"bool\":{\"must\":[{\"match\":{\"*\":\"RICH\"}}]}},\"options\":{\"fuzzy\":true}}",
"http_method":"POST"
},
"version":3
}
Response from Buddy to Manticore Search
The response JSON structure:
Key | Description |
---|---|
type | Set to json response if the request type was unknown json request and sql response for unknown sql request . |
message | A JSON object potentially containing an error message for displaying and/or logging. This is what Manticore Search will forward to the end-user. |
error_code | An integer representing the HTTP error code which will be a part of the HTTP response to the user making a JSON over HTTP request. For SQL over HTTP/mysql communications, this field is ignored. |
version | Indicates the current protocol version being used. Currently version is 2. |
Example of HTTP Response:
{
"type": "json response",
"message": {
"a": 123,
"b": "abc"
},
"error_code": 0,
"version": 3
}
Example of HTTP Response:
{
"type": "json response",
"message": {
"a": 123,
"b": "abc"
},
"error_code": 0,
"version": 3
}
Example of MySQL Response:
{
"type": "sql response",
"message": [
{
"columns": [
{
"Field": {
"type": "string"
}
},
{
"Type": {
"type": "string"
}
},
{
"Properties": {
"type": "string"
}
}
],
"data": [
{
"Field": "id",
"Type": "bigint",
"Properties": ""
},
{
"Field": "title",
"Type": "text",
"Properties": "indexed"
},
{
"Field": "gid",
"Type": "uint",
"Properties": ""
},
{
"Field": "title",
"Type": "string",
"Properties": ""
},
{
"Field": "j",
"Type": "json",
"Properties": ""
},
{
"Field": "new1",
"Type": "uint",
"Properties": ""
}
],
"total": 6,
"error": "",
"warning": ""
}
],
"error_code": 0,
"version": 3
}
Note: The structure for error responses in the message
field follows the guidelines specified in the Manticore Search documentation for JSON and similarly structured documentation for SQL interactions.