Understanding Intelligent Caching Engine in Translator Plugin Pro / Gold 5.0

We have designed a new caching engine for Translator Plugin Pro 5.0 & Gold 1.0. Previously we used WordPress’ Object Caching Engine (bundled with WordPress) which doesn’t work so well in resource constrained sites. Also it doesn’t serve our need to have semi-permanent cache for translated pages and provide infrastructure for Translator Pro & Gold moving forward. Our primary design objectives in designing the new caching engine are as follows:

Translation speedup is the primary goal of any caching effort but there is more. Translating a page requires us to submit the content, after pre-processing, to a translation engine like Google, who then sends us the translated version of the page. The content is then post-processed and delivered to the client / requester. This process involves database queries and network traffic, each of which can add up if your site becomes / is popular. The goal of caching is not only significantly speedup delivery of translated pages but also to reduce network traffic and database queries.

Availability of service is a key goal. Frequent queries to translation engines can cause overload of their services and they in turn can disable your site for a time period or more, which is clearly undesirable. So not only do we need to speedup translation on your site we also need to keep the translation engines happy by managing their load within their acceptable limits. We have done extensive tests to understand and satisfy the limits of each translation engines we use. We also provide automatic failover support so if one engine fails we move over transparently to another and so on till we are able to satisfy the request or run out of engines (rarely). We have as much provided 4 translation engines for redundancy. It is possible that they may change their limits down the road which is why we have made the engine very much configurable from the options panel.

Intelligent caching makes caching more relevant and effective. In WordPress the traditional model of caching is as used in wp-cache 2 plugin. It invalidates the cache on first sign of change and creates it from scratch again. It is very inefficient and ineffective for sites with frequent changes and /or frequent comments. Each new post / updated post / new comment / updated comment etc. will invalidate the cache and start all over again. It is surely an overkill. Most of the posts do not change at all when a new post is created or even modified. We developed various ways for optimizing this scenario. The primary objective is to ensure that translated pages are available for all requested content, either by translation (see availability issue above) or from cache, even if the translated page is not always from the latest revision. Certain pages like index pages are prioritized for faster refresh. The cache is continuously refreshed in the background. Our intelligent caching engine achieves over 90% cache hits on average as you can see from the sidebar of this blog. There are other technologies in place to ensure higher priority is given for humans than for robots / spiders etc.

Manual Control is provided to individually delete the translated pages from cache or to delete the whole cache (not recommended).

The cached pages are compressed in database to minimize space requirements. However since you will have a maximum of 14 times (you can reduce the number by reducing the number of translations you provide) the amount of content, compressed for space reduction, as the original number of pages you need to have space allocated for the database to hold it. If you have limited space for the database then you should either switch hosting provider (I recommend WestHost) or you should reduce the number of translations your provide.