GZIP from backend

[MekelCon]

Hello,

i wrote a plugin to tranform request from xml -> json and the response from json -> xml

Here is the code and it is working, add an xml-to-json.lua :

-- Licensed to the Apache Software Foundation (ASF) under one or more
-- contributor license agreements.  See the NOTICE file distributed with
-- this work for additional information regarding copyright ownership.
-- The ASF licenses this file to You under the Apache License, Version 2.0
-- (the "License"); you may not use this file except in compliance with
-- the License.  You may obtain a copy of the License at
--
--     http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software
-- distributed under the License is distributed on an "AS IS" BASIS,
-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-- See the License for the specific language governing permissions and
-- limitations under the License.
--
local zlib        = require('zlib')
local core        = require("apisix.core")
local ngx         = ngx
local ngx_header = ngx.header
local request     = require("apisix.core.request")
local pairs       = pairs
local xml2lua     = require("xml2lua")
local handler     = require("xmlhandler.tree")
local str_lower   = string.lower

local plugin_name = "xml-to-json"


local schema = {
    type = "object",
    maxProperties = 0
}

local _M = {
    version  = 0.1,
    priority = 700,
    name     = plugin_name,
    schema   = schema,
}


function _M.check_schema(conf)
    local ok, err = core.schema.check(schema, conf)
    if not ok then
        return false, err
    end
    return true
end

do

function _M.rewrite(conf, ctx)
    local content_type = request.header(ctx, "Content-Type")
    if not content_type then
        return 
    end

    content_type = content_type:lower()
    if content_type ~= "application/xml" and content_type ~= "text/xml" then
        return
    end
    
    local body = core.request.get_body()
    if not body then
        return
    end

    local parser = xml2lua.parser(handler)
    parser:parse(body)
    -- Second '.root' should be dynamically based on the request root body, 
    -- here we expect a body like <root>...the data...</root>
    local request_body, err = core.json.encode(handler.root.root)
    if err then
        return
    end

    core.request.set_header(ctx, "Content-Type", "Application/Json")
    ngx.req.set_body_data(request_body)    
end

function _M.body_filter(conf, ctx)
    local content_type = request.header(ctx, "Content-Type")
    if not content_type then
        return 
    end

    content_type = content_type:lower()
    if content_type ~= "application/json" then
        return
    end

    local buffer = core.response.hold_body_chunk(ctx)
    if not buffer then
        return
    end

    -- when body has already been read by other plugin
    -- the buffer is an empty string
    if buffer == "" and ctx.resp_body then
        buffer = ctx.resp_body
    end

    local gzip = false
    local content_encoding = ngx_header["Content-Encoding"]
    if content_encoding then
        content_encoding = content_encoding:lower()
        if content_encoding == "gzip" then
            gzip = true
            buffer = zlib.inflate()(buffer)
        end
    end

    local data, err = core.json.decode(buffer)
    if err then
        core.log.error("Error while decoding json response, buffer was : " .. buffer)
        return
    end    
    local xml_response = xml2lua.toXml(data, "root")
    if gzip then
        xml_response = zlib.deflate(1, 31)(xml_response, "finish")
    end
    ngx.arg[1] = xml_response
end


function _M.header_filter(conf, ctx)
    core.response.set_header("Content-Type", "application/xml")
end

end  -- do


return _M

As i use a 3rd party lib i had to modify the rockspec

...
    "lua-resty-ldap = 0.1.0-0",
    "lua-zlib = 1.2-2"
... 

And enable the plugin in the config.yaml.

It works well, I just had one issue : My backend after Apisix is answering gzip encoded content.
So before doing the transformation json -> xml , I had to do :

1. Gzip decode
2. tranform json to xml
3. Gzip encode

My question is : where is the right place to manage well this Gzip encoded content, i believe for other plugin, like logger or response-rewrite we will have gzip. So it won't be possible to log or tranform the body.
Should I make this logic of unGzip and reGzip more global in a pull request ?

[juzhiyuan]

1. @kingluo, can you please have a look at the Gzip question?
2. @MekelCon, would you like to contribute this xml-to-json plugin to apache/apisix community? 😄

[MekelCon]

@juzhiyuan yes for sure !
I plan to propose the pull request next week :)

[moonming]

@MekelCon great work 👍

[kingluo]

@MekelCon You need to unzip the body from your plugin. No other place you could place the unzip logic. Another option is clearing the Accept-Encoding header before sending it to the backend. In fact, no need to zip the JSON body because normally it's small in size.
FYI, no Nginx built-in body filters could be used to unzip the body in your case. Note that nginx gunzip filter is for a different purpose, which is used to unzip the body only when the downstream doesn't support gzip encoding. And it's even run after lua body filter.

[MekelCon]

Thank you all for you reply !
I think then I will keep it more or less like that in my pull request to submit the xml-to-json plugin and see with the reviewers if something should be changed.