Sharding

ZODB supports both sharding, where the database is split across multiple file storages, as well as multi-database, where where the database is split among multiple databases. Indivudial databases can be FileStorage, RelStorage, NEO, or any other format supported by the ZODB. A multiple-database is pretty much what it sounds like: it ties together multiple DBs into one grand unified DB: each persistent object is stored in only one DB but can be transparently referenced as normal from any DB in the collection.

There are several parts to multiple-database.  Transaction management uses two phase commit to ensure that the transaction is successfully completed across all databases.  Garbage collection cleans up the garbage across all databases.  You also need a scheme to figure which database a particular object id is located in.   

Configuring Multiple Databases

Here is the documentation on how to configure multiple ZODB databases. 

ZODB Distributed Garbage Collection

This product collects gargage across a collection of intertwined zodb databases.  It marks objects as deleted, for later compaction, as well as provided debugging tools.  It can also be used to create a list of back pointers.  

Distributed Transactions

Here is the ZODB book chapter about how to manage transactions across multiple databases. 

Sharding Issues

The pros and cons are complex. It all depends on your application architecture. Reads may be faster, but writes will be slower because of the two phase commit. Load Balancing may be better. Complexity Separate caches may be good or bad. The great news is that you do have this option.



I invite you to Register and then link to your own blog postings and software packages..

Powered by Zopache, Grok, Zope and ZODB

Robots Crawl This