Handling real-time events with Nginx, Lua and Redis

In this post, we will explore options to handle lots of HTTP events asynchronously using Nginx, Lua for the frontend and Redis as the backend. Although there are plenty of options out there to deal with this, we will check these technologies out in a bit more detail and leaving lots of options open for customisation and tuning, while using tried and true building blocks.

I have been thinking about realtime event handling for quite a while now. For instance, I have been wondering and learning about Twitter’s architecture, and about event-driven behaviour and systems, which are fairly common. There is plenty of stuff available online but in many cases, we do not need to go that low level or have that much load. From an engineering perspective, Twitter is one giant big realtime monster that handles events and event notifications really quickly and efficiently. It makes for an interesting exercise to work on similar lines as there are quite a few computing problems that can be reduced to event-handling and most are realtime at that. As an added bonus, any solution nowadays should have scalability and ease of deployment in mind, specially to be able to be deployed on the ‘Cloud’, be it Amazon Web Services,  your very own Nimbus cloud or any of the other good cloud solutions out there.

One such common use-case is for instance counting events on a website platform (user visits, user actions, user sessions, etc.) so they can be gathered and analysed in detail using Big Data techniques to improve on the website, learn user behaviour, etc. Many of those events can be gleaned from log files and there are plenty of tools for that, but usually they are not realtime and are many geared towards  technical log analysis instead of higher-level user activity study.

Let’s get started and name the project. I have called it ‘Meteorit‘, after which means ‘meteorite’ in Catalan  (not related to the good folks from the Meteor project). We will be starting by designing and deploying a system composed by three layers, using Nginx, LUA and Redis. As a deployment server we will be using a CentOS 6.x flavour even tough any modern *nix system would do. Packages will be built to RPMs for ease of deployment.

HTTP frontend: building Lua

Functionally, we need only one thing, which is a restricted and secure HTTP REST interface to register events coming from our users. We also require the system to handle high loads and the activity spikes that are so common in popular websites. To that effect, we will use an asynchronous event-handling queue to minimise risks of system overload during high event traffic.

Nginx is a high performance HTTP frontend which provides great flexibility and performance. For greater efficiency, Nginx only allows for compiled-in extensions and modules, which means we will probably have to compile it to configure it to our needs.

Lua is a flexible and lightweight modern programming language, often used for general scripting, gaming AI, etc. We will be using its existing integration tools with Nginx to handle frontend validation and logic.

To handle testing and building the different components, we will be having two strategies:

• Wherever there is a readily available Centos 6.x RPM package that is suitable, we will be using that package instead of building our own.
• On the other hand, when available packages do not suit our needs, are configured differently or they just need to be compiled, we will be using Maven to perform all necessary tasks, including generating the custom RPM.

Each Maven job will be generating an instantly deployable RPM package, complete with any runtime dependencies needed for operation declared within the package for easier installation and practical deployment to cloud systems.

One of the advantages of using Maven to do our builds is the ability to hold common logic in an inheritance hierarchy. In the case of building the frontend, we have two sub components that we want to compile ourselves, one is a LUA runtime and the other is Nginx with all the required modules so they are compiled-in. These two projects share the need of compiling and testing the necessary code so we move the common building logic on a common Maven parent.

The resulting Maven structure is described in this diagram:

The meteorit-compile-parent Maven project holds common build logic, namely build information, licensing, target platform, install prefixes, etc. It also holds simple executions of the exec-maven-plugin that invoke two bash scriptlets in the compile phase. If ‘${project.artifactId}-compile.sh’ and ‘${project.artifactId}-test.sh’ scripts exist in ‘${project.build.directory}’ they will be made executable and invoked from that folder. This means that any children Maven projects that have any of these files will run them, presumably to compile and test any required binaries. It is really useful to think of Maven as an implementation of the Gang of Four ‘Template’ behavioural design pattern, which establishes a common algorithm (the build lifecycle) where steps are predefined but the actual implementation of each step is either common (on a Maven parent project) or specific (on a child project). If done well, applying this design pattern strategy makes Maven very powerful and worthwhile.

In that way, the meteorit-frontend-lua project first downloads the speedy LuaJIT as well as the LuaRedisParser, then the compile bash script builds both packages using make and ‘installs’ them into a folder in ‘${project.build.directory}’ so we can add them to the installable RPM package. Compilation is pretty straightforward, LuaJIT pretty much builds itself and the LuaRedisParser library needs to take into account the include folders and libs resulting from LuatJIT.

To build the RPM, we will be using the dependable rpm-maven-plugin, cherry picking stuff from our ‘${project.build.directory}’ and mapping it to the destination installation folders, taking care of using softlinks when needed and assigning the correct permissions (rwxr-xr-x in the case of public executables) and so on. For example, to link the executable to the specific compiled version we create the following mapping:

 
  
   ${lua.tempfolder_}/${lua.installfolder_}/bin
   
    luajit
   
  
  
   luajit
   luajit-2.0.2
  
 
 ${lua.username_}
 ${lua.groupname_}
 755
 ${lua.installfolder_}/bin
 false

Which will result in the following filesystem structure:

/opt/lj2/bin/luajit -> luajit-2.0.2
/opt/lj2/bin/luajit-2.0.2*

Which means invoking ‘/opt/lj2/bin/luajit’ will always run the latest binary.

Included in the RPM we add a simple testing script that makes use of the excellent shUnit2 bash library to run a simple Lua script and therefore test the LuaJIT installation once it is installed. As the RPM is setup the script needs to be run manually, but it can come in handy to test all is OK. Test code is straightforward and just checks that a hello world Lua script returns ‘Hello World’:

testLua() {
	hello_=`$LUA_ -e 'print("Hello World")'`
	assertEquals 'LUAJit does not work' 'Hello World' "$hello_"
}

HTTP frontend: building Nginx with Lua integration

We use the same technique to build Nginx, the meteorit-frontend-nginx Maven project downloads the Nginx source and all appropriate modules:

• HttpLuaModule: to allow Lua scripts to run within Nginx, which will be the ones actually connecting to Redis
• lua-resty-redis: a non-blocking Lua Redis library using the cosocket API, provides a connection pool so the interface to Redis is reused between HTTP requests.

The compilation script is also quite straightforward, we just need to statically add the Lua module and specify where to find the LuaJIT library that we installed in the meteorit-frontend-lua package. The relevant flags are:

--with-ld-opt="-Wl,-rpath,PATH OF INSTALLED LuaJIT /lib"
--add-module='PATH of HttpLuaModule source folder'

To create the RPM we also use the ‘rpm-maven-plugin’ to specify where all files go, using the user ‘nginx’ for all Nginx-related files. As that user is not necessarily in the system, we specify a preinstall scriptlet:

${project.build.directory}/preinstall.sh
utf-8

Which will conditionally add the relevant system user if it does not exist:

id '${nginx.username_}'
if [ $? -eq 1 ]; then
	echo 'Adding nginx daemon user'
	 /usr/sbin/useradd -c 'Nginx daemon' -M -r -s /sbin/nologin ${nginx.username_}
fi

The Maven build also patches the main nginx.conf file to include our custom application configuration (meteorit.conf), adds an init.d script and adds the Lua logic files.
Having the frontend logic split up between Nginx configuration and Lua allows for a clearer separation of concerns and greater efficiency as Lua code is only ran when needed. The following diagram illustrates the concept:

We add another lua script to test the system which, like we did in the meteorit-frontend-lua package will be called from a testing shell script, to make sure the RPM deployment went well. The testing script makes a request which ultimately adds a testing value to redis, reads it and gives it back as the HTTP body, so it can be tested using an assertion.

Redis asynchronous queue backend

Redis is an excellent and versatile choice to handle a high-performance in-memory queue. There are plenty of clients for many programming languages and it has a wide range of flexible commands. Even though there are more specialised queing systems its efficiency and versatility make for an excellent choice as queue backend.

As there is a ready-made redis RPM package in the Epel repo, there is no need to build it ourselves and we just add it as a RPM dependency in the meteorit-frontend-nginx package.

Building and deploying the whole Meteorit system

I have setup a GitHub project with the code here and it should build out of the box on a CentOS 6.x system with no problems. The meteorit-env subproject has two shell scripts that install the deployment (Epel and rpmforge repos, java, pcre, zlib) and compilation (gcc, make, Maven…)  prerequisites. Both should be run as root.

Conclusions

We have demonstrated a robust build system that creates a ready-to-deploy high-performance HTTP event handling system (Meteorit) that sports a lot of flexibility and uses mature and stable technologies overall. Comments on the blog welcome and pull requests on GitHub are doubly welcome. In future installments I will be expanding on Meteorit a bit.