- Get Involved
- About Us
OpenERP 6.1 comes with a load of new features, and one of them provides a much greater ability to scale up on modern hardware. Until now, OpenERP offered only one option: a multi-threaded HTTP layer, with a limited ability to use available computing resources. F or the 6.1 release, one of the goal was to make it easy to run the OpenERP server in multiple processes, harvesting big performance gains. Doing so introduces nice deployment choices and development opportunities.
By running a Python application such as OpenERP 6.1 in multiple processes instead of multiple threads, one can avoid Python's Global Interpreter Lock (GIL) and take advantage of the multiple cores of today's machines.
This (rather technical) post explains how the upcoming OpenERP version runs more efficiently on multi-core systems by using the excellent Gunicorn HTTP server.
This subject will also be covered during the 2012 OpenDays. The slides for the talk are already available at http://bit.ly/IkOYyq
Dance around the GIL
To create and manage processes, we first thought to use the `multiprocessing` module. When the time to finally implement a multi-process solution arrived, we quickly thought it was better to handle the added complexity in the Unix way: make a specific piece of code to manage processes. As it happens, such specific piece of code already exists and we didn't write anything (and thus didn't use `multiprocessing`): we simply turned the server into a WSGI-compliant application, leaving the responsibility to manage it to someone else. That someone else is Gunicorn.
Gunicorn is a Python HTTP server with support for WSGI. It uses the pre-fork model to spawn a WSGI-compliant application into different processes. In our case, the WSGI application is the OpenERP server. The server has its WSGI entry point located in the `openerp.wsgi` module. It is simply named `application`. In our repository, we also provide a sample `gunicorn.conf.py` configuration file. Assembling the pieces together, launching the server with multiple processes is a simple as:
> gunicorn openerp:wsgi.core.application -c /path/to/gunicorn.conf.py
You can modify the configuration to your liking. Gunicorn is well documented and the comments in the sample configuration file should prove enough to get you started. Just note that it is not possible to pass arguments to OpenERP on the command line (i.e. the way you would do it with `openerp-server`). Instead, you can directly set OpenERP's configuration values from within the Gunicorn configuration file (as it is done in the example file).
Awesomeness provided by the beast
It is still possible to start the server with the regular `openerp-server` script. Doing so uses a multi-threaded HTTP layer (this is not the 6.0 HTTP layer: we also use the WSGI entry point, this time serving it with `werkzeug`). But serving OpenERP with Gunicorn is great! When handling two concurrent CPU-bound requests with two workers (on at least two cores), you can expect a nearly 2x speed-up. Of course, if the two requests lock the same rows in database and don't spend much of their time running Python code, you might achieve no speed-up at all.
Beside taking advantage of a multi-core setup, Gunicorn provides a few hooks that we use to limit the resources made available to each request. It is also possible to automatically kill and restart processes after they have served a few thousands of requests, to mitigate memory waste, if any. We have added three new options -- although they are documented as command-line options, they really are used only with Gunicorn:
* `virtual-memory-limit` limits how much memory a process can allocate. When the limit is reached, a `MemoryError` is raised.
* `virtual-memory-reset` is a similar limit: when the amount of memory used exceeds that limit, the process will gracefully die after the current request and Gunicorn will re-spawn a new process. This is again a safety net against memory leak.
* `cpu-time-limit` limits the amount of CPU time a request can use, also raising an exception when the limit is reached.
WSGI and statelessness
To ensure we could run multiple OpenERP processes safely, we had to modify the server to make it stateless, because any request can be handled by any process. For this reason, we changed the implementation (and the name) of the `osv_memory` class. Instead of being held in memory, a `TransientModel` is stored in database, just like a regular `Model` (the new name for `osv`). The difference with a `Model` is that `TransientModel` rows are automatically deleted after a while.
Server-side caching is another issue. It's useful for improving performance in some situations, but makes the server partially stateable and thus requires synchronization. Fortunately, most of OpenERP caches are of minor importance and read-only, so the relatively fast process recycling will take care of refreshing The only cache that really required an update is the login cache; because an authentication check is done for each request, if you change your password (causing only one process cache to be updated) you will immediately be locked out. The trivial way we fixed it was to ignore the login cache whenever an authentication fails, causing a refresh of the cache on that process. After a change of password all caches will thus be refreshed transparently one by one.
Still, for the situation that needs it (i.e. it is really necessary to run multiple processes while still allowing configuration changes), we implemented a signalling scheme using the PostgreSQL database. Whenever caches are invalidated on a process, or a new module is installed, the process signals the change to other processes (managed by the same Gunicorn instance, or running on a different machine). The solution will be part of a next 6.1 release.
As mentioned above, the OpenERP server is now a library exposing a WSGI entry point. It is also a kind of WSGI middleware as it can dispatch requests to other, registered entry points. This is indeed the way we have now embedded the web client in the server: the `openerp-web` project provides its own addons directory, which is put in the server's addons path. The server loads the web addons at startup because it is the default value in the new `server_wide_modules` option (exposed on the command line as `--load`). When being loaded, the web addons registers itself as a WSGI entry point: the server serves XML-RPC and regular browser requests on the same port (8069 by default). Of course you can use the same principle for your own modules.
Please note the web client is storing its sessions on disk. If you plan to deploy multiple web clients, embedded in the server or not, you have to make sure the sessions can be accessed by any of them.
Embracing existing (and great) tools allow us to be leaner and meaner. This is true with WSGI and Gunicorn but we hope to continue in this direction. One important question is left unanswered: how many processes must Gunicorn spawn on a given machine to be as efficient as possible? We don't know yet the answer but we should have it quite soon: we are assembling benchmarks in the `openerp-command` repository.
 My use of the word 'speed-up' may not be completely appropriate: speed-up is normally used for parallel computation. In this post a 2x speed-up means you can run a second request with no impact on another one.
 Server-wide modules are not tied to a particular database. For instance, the web client can serve a page to create a new database; obviously the web client has to run even if a database is not yet loaded.