Awesome

Name

lua-rds-parser - Resty-DBD-Stream (RDS) parser for Lua written in C

Status

This module is production ready.

Synopsis

local parser = require "rds.parser"

local res, err = parser.parse(rds)

if res == nil then
    error("failed to parse: " .. err)
end

print(res.errcode)
print(res.errstr)
print(res.insert_id)
print(res.affected_rows)

local rows = res.resultset
if rows then
    for i, row in ipairs(rows) do
        for col, val in pairs(row) do
            if val ~= parser.null then
                print(col .. ": " .. val)
            end
        end
    end
end

Description

This Lua library can be used to parse the Resty-DBD-Stream formatted data generated by ngx_drizzle and ngx_postgres into Lua data structures. In the past, we have to use JSON as the intermediate data format which is quite inefficient in terms of both memory and CPU time.

To maximize speed and minimize memory footprint, this library is implemented in pure C.

Null values in RDS are turned into the light user data parser.null (or ngx.null) where "parser" is the module object returned by Lua's "require".

JSON Serialization

If you want to serialize the parsed result into JSON, please use the lua-cjson library instead of lua-yajl, because lua-cjson is faster than lua-yajl in many common cases, and more importantly,

parser.null == cjson.null ~= yajl.null

Using with HttpDrizzleModule

To use with ngx_drizzle, here is a small example:

upstream backend {
    drizzle_server 127.0.0.1:3306 protocol=mysql
                   dbname=ngx_test user=ngx_test password=ngx_test;
    drizzle_keepalive max=10 overflow=ignore mode=single;
}

server {
    ...

    location /mysql {
       drizzle_query $echo_request_body;
       drizzle_pass backend;
    }

    location /api {
       content_by_lua '
           local sql = "select * from cats"
           local resp = ngx.location.capture("/mysql", {
               method = ngx.HTTP_POST, body = sql
           })
           if resp.status ~= ngx.HTTP_OK or not resp.body then
               error("failed to query mysql")
           end

           local parser = require "rds.parser"
           local res, err = parser.parse(resp.body)
           if res == nil then
               error("failed to parse RDS: " .. err)
           end

           local rows = res.resultset
           if not rows or #rows == 0 then
               ngx.say("empty resultset")
               ngx.exit(0)
           end

           for i, row in ipairs(rows) do
               ngx.print("row ", i, ": ")
               for col, val in pairs(row) do
                   if val ~= parser.null then
                       ngx.print(col, "=", val, " ")
                   else
                       ngx.print(col, "=null ")
                   end
               end
               ngx.say()
           end
       ';
    }
}

On my machine, GET /api will yield

row 1: id=2 name=null
row 2: id=3 name=bob

of course, the actual output depends on the structure and contents of the "cats" table in the mysql database.

You can use this Lua library with the ngx_postgres module in a similar way.

Installation

Build requirements

Lua (http://www.lua.org/),
or LuaJIT (http://www.luajit.org/)

Gnu make and gcc is required to build this module.

Linux/BSD/Solaris

gmake CC=gcc
gmake install CC=gcc

Mac OS X

make LDFLAGS='-bundle -undefined dynamic_lookup' CC=gcc
make install

If your Lua or LuaJIT is not installed into the system, specify its include directory like this:

make LUA_INCLUDE_DIR=/opt/luajit/include/luajit-2.0

You can specify a custom path for the installation target:

make install LUA_LIB_DIR=/opt/lualib

The DESTDIR variable is also supported, to ease RPM packaging.

TODO

add support for option "compact" to generate a compact Lua table for the "resultset" field.

Known Issues

The endianness flag in RDS is not supported yet in this library, and it will assume it's of the host's endian. So do not try parsing the RDS stream that is generated by another machine of a different endian.

Author

Yichun "agentzh" Zhang agentzh@gmail.com, CloudFlare Inc.

Copyright & License

This module is licenced under the BSD license.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.