A distributed lock manager for OpenStack? [LWN.net]

A distributed lock manager for OpenStack?

Posted Oct 31, 2015 20:21 UTC (Sat) by harlowja (guest, #97846)
In reply to: A distributed lock manager for OpenStack? by isotopp
Parent article: A distributed lock manager for OpenStack?

As one of the authors of tooz. I agree, some should probably be deprecated. Redis and kazoo should likely be kept (kazoo == zookeeper, redis-py == redis), the odd one there is redis. It has been gaining properties that make it more of what I call the 'poor mans zookeeper', although it has become more of a swiss army knife than a surgical tool (just look at http://redis.io/commands/)...

to post comments

A distributed lock manager for OpenStack?

Posted Nov 1, 2015 19:50 UTC (Sun) by isotopp (subscriber, #99763) [Link] (3 responses)

You want to keep stuff? Run Jepsen on it. If it fails, in distributed mode, it should probably die. Redis does not survive Jepsen testing (http://aphyr.com/tags/jepsen)

A distributed lock manager for OpenStack?

Posted Nov 2, 2015 18:43 UTC (Mon) by harlowja (guest, #97846) [Link] (2 responses)

Understood, but then what really has survived jepsen (besides zookeeper)?

I'm more along the line of listing the various capabilities (strengths and weaknesses) of the underlying implementations and letting the smart user select one that is appropriate, jepsen testing is like ensuring you have a battleship, but sometimes you only need a powerboat and/or canoe (for lack of better analogy)...

A distributed lock manager for OpenStack?

Posted Nov 2, 2015 21:36 UTC (Mon) by isotopp (subscriber, #99763) [Link] (1 responses)

ZK, etcd and consul have (consul on 2nd try). And that's actually quite the point here - these things are built to manage a cluster properly by building the inner cluster (ZK calls it the "ensemble") and once that is stable, to provide a platform and service for the outer cluster.

Services provided usually include at least discovery (things such as hypervisors, running VMs, and *-api instances could check in with the ensemble and leave their endpoint data with it, making things such as keystone.endpoints and similar tables superfluous and much more accurate for degraded states. Also, HA for free). Other services that are common are locking (the original reason for the project), and with locking global order/sequence numbers/counters. Also, a kind of highly available, distributed, small key/value store, in consuls case with config file generator attached.

In the end, rigorous application of the tool to the entire openstack project would greatly benefit the project in handling degraded cases, HA, locking, and turning from a web system into a proper control system/state engine. That is, it would become much more reliable. For this the openstack project would have had to commit itself to a single of these system in order to be able to reliably use the feature set of that system.

With an abstraction layer inbetween you will only get what the weakest of these systems can give you (hence the Redis needs to die), and you would need to Jepsen the shit of out every possible configuration and service provided in order to make sure the abstraction on top of the underlying consensus system doesn't break stuff.

A distributed lock manager for OpenStack?

Posted Nov 2, 2015 23:49 UTC (Mon) by harlowja (guest, #97846) [Link]

Totally agree with the one solution; it was what I (and others) was thinking was going to be an outcome of that session (which I was the other driver of) but it didn't turn out this way in the end (even though I tried to say we should really be opinionated). As far as the other drivers, yes, the point is well taken, and jepsen is a good base case and it might be the way we should handle new drivers (if they have not been proven jepsen safe, then they aren't allowed); redis would then get kicked out due to this (although for small enough deployments perhaps having a single redis that uses persistent storage and can *manually* failover to a secondary master will be fine). Overall we (as a tooz group) are figuring out the way to go here and there seems to be a few ideas (like this) that are being bounced around.