How now, Stephen?

Running multiple local Riak instances on OSX

Wed, 24 Sep 2014 09:07:28 -0400

I recently had the need to set up a couple of different Riak servers locally for development purposes. Since there was no documentation for doing this, here we go!
Assumptions:

You have Homebrew installed
You’re comfortable in the command line
You don’t have an existing installation of Riak on your OSX device

Firstly, you’re going to need to pull the latest Riak source from http://docs.basho.com/riak/latest/downloads/ At the time of writing, it is v2.0.0


curl -O http://s3.amazonaws.com/downloads.basho.com/riak/2.0/2.0.0/riak-2.0.0.tar.gz
tar -xvf riak-2.0.0.tar.gz
cd riak-2.0.0

Next up you need the right version of Erlang/OTP. This means installing from a back version of Erlang. Woo!

brew install homebrew/versions/erlang-r16

That should install to /usr/local/Cellar/erlang-r16/, so now it’s time to build our development versions of Riak. First you need to alter your path so that OTP R16 is the first version of erlang that bash will find, then you can build the development instances


export PATH=/usr/local/Cellar/erlang-r16/R16B03-1_1/bin/:$PATH
make devrel

This should create 8 databases for you in ./dev

By default these are configured as a cluster - which, for my purposes was not so useful. Thankfully they also each act as an independent database, so if you edit the line in devN/etc/riak.conf that looks like

distributed_cookie = riak

And set a distinct cookie for each database you want, you can run multiple distinct clusters locally (up to 8 with the default configuration). For my purposes I set up two clusters of 4 machines:

# dev1, dev3, dev5, dev7
distributed_cookie = riak-odds

# dev2, dev4, dev6, dev8
distributed_cookie = riak-evens

How to bypass "/bin/rm: Argument list too long"

Fri, 04 Oct 2013 09:09:00 -0400

The following passes files to the rm program 100 at a time, ensuring that it never gets overwhelmed.

find . -name "REGEX" | xargs -n 100 rm

If you’re looking to get the most out of rm (though for all my use cases, 100 at a time seems sufficient), check out this StackExchange thread to work out what your absolute argument limit is Protip: it’s in bytes and it includes your environment data, so better to stick with small-ish number

Getting past the NoSQL curve with MongoDB

Sat, 28 Sep 2013 13:14:22 -0400

A few months back a friend of a friend came to me asking whether MongoDB was actually worth switching to from a fairly large-scale MySQL setup. My short answer was not unless your project is about to blow up (or is being rebuilt anyway), but my longer answer brought me to a list of the things I’ve learned about Mongo in the last couple of years that could at least help him to inform himself…

Firstly, it’s a good time

As Luigi Montanez discusses in Mongodb 2.0 Should Have Been 1.0 - the database didn’t really reach a point of proper stability until at least their v2.0 release in September of 2011 (later for those of us waiting on some of the 2.2+ features to really make it worthwhile to have abandoned SQL). Up until that release, brave souls using Mongo in production were really risking a lot of headaches for the (admittedly awesome) benefits of this simple NoSQL database. Thankfully the team at 10gen have done some very awesome work with the software to bring it to now be a production-grade database, most notably with the introduction of the aggregation framework in v2.2.

Structuring your collections

Mongo has no simple way of performing SQL-like JOIN queries - so you’re going to have to be much more clever about how you organize things. Remember database normalization? Forget it. Duplication of data comes with the beast, and (at least for now) it appears to be a necessary evil. When it comes to querying one versus three collections, you’ll start to see the value.

We’re schemaless (or we pretend to be). One of the unfortunate byproducts is that every one of your keys is taking up precious space. If you’re in it for big data (we’re talking tens of millions of documents) - then consider that by using single character keys, you stand to save on storage and RAM.


{"a": "1234 Fake Street", "p": "1-800-com-pany", "e": "email@company.com"}

Be aware that documents can only be 16mb. This can be a pain, though you may think *who stores 16 megabytes in a single “row” anyway. It’ll happen, and you’ll suddenly not understand what is up with your application. If you’re adding fields on the fly - be aware of what is going in them and react accordingly. Also be aware that your keys factor into the 16mb limit, see the above about keys.

Less worrisome, but worth noting from the limits documentation is that you can only nest BSON documents up to 100 levels. But if you’re doing more than that, you may need to rethink your life choices anyway.

If you’re storing files, mongo actually has a built in Distributed Filesystem (GridFS) that is made for that, and if you’re storing text, be aware of what the text sizes are going to be. But for example, I work with documents in which I store lists of thousands of urls, and still stay well within document size limits. This also applies to the result of aggregate framework queries, so be advised (esp. if you’re writing ones that you expect to take awhile - it’s always disappointing to get to the end and discover the result is too big). One advantage of the AF is that intermediate steps do not carry this same constraint.

While on the subject of RAM…

You’re going to use a lot of memory really poorly to start with. And this is because you get the wonderful task of writing your own indexes. Some things to know right off the bat:

Indexes are prefix driven, so in this situation: db.test.ensureIndex({field1: 1, field2: 1}) and db.test.ensureIndex({field1: 1, field2: 1, field3: 1}) - the first index is actually redundant, as queries looking for field1 and field2 can use the second index. But as mentioned above, order still matters.
Indexes should (as much as possible) be designed to fit in memory. This means you’re going to need your thinking cap on when designing your data structures to make as many of your queries fit as few indexes as possible.
Indexing starts to be important when you hit tens of thousands of documents in a collection, and takes awhile to rebuild when you’re in the hundreds of thousands. Building the right indexes has very literally improved my various application’s performance by up to 100x
Mongo still doesn’t support compounding of separate indexes (though I believe they’re working on it) - so if you have two indexes and query on fields in both of them, it will just choose one of them, instead of self-optimizing. Hopefully this will just come as a nice bonus in the near future!
Order matters for indexes, and affects their usage within queries:


// A table with these indexes
{"_id": 1}
{"name": 1, "address": -1}

// Will not use either of them for this query
{"address": "1234", "name": "fake", "_id": {"$gt": "somevalue"}}

// And will use ONLY the second index for
{"name": 1, "_id": {"$gt": "somevalue"}}

And about those ODMs

Despite how trivial a task it seems - mapping Mongo documents to (your language here) objects must be easy, it’s all an object notation! So far everything I’ve worked with is a lot messier than you’d expect. In the Python world, MongoEngine seems to be the go-to ODM. It has clean Django-like syntax, but is unfortunately a little too tied to the SQL concepts of a traditional ORM. Also, the aggregate framework is pretty much the way to go for any moderately complex queries anyway, so I tend to lean on the side of a client-side validator (like JSONSchema) and working directly with the BSON.

Would love to know if any ODMs really have gotten it right though!

But at least it’s all performant!

Until it isnt. Mongo database performance is generally somewhat like a cliff: it will perform amazingly up until a certain critical point (memory/cpu/bad database structuring) at which point it peaks out. Usually this is from user error or insufficient server resources, but it’s worthwhile to note, as it can and will make you pull your hair out.

What is this aggregate framework you keep mentioning

In short: it is map-reduce on steroids. In long, it is the best thing to come out of the MongoDB project to date. By allowing the building of unix-y data pipelines, you can pull out very sophisticated metrics about your large datasets painlessly. In a lot of cases I’ve seen it replace large blocks of client-side logic – and since it runs directly within the database itself, increase their performance massively (my first big win with the aggregate framework was more than 600x faster). It’s easier to just read the documentation and work through the examples, than it is to explain!

There is also often talk of “eventual consistency”. What’s the deal?

This is the idea that data reads after writing to the cluster may be slightly behind - which only really applies to reading from slave instances, which is disabled by default. Speaking from my own experience writing high frequency web crawlers (millions of webpages/day across hundreds of servers, performing tens of millions of database ops) this is only a problem you’ll encounter at unbelievably high load (i.e. telecom, high frequency trading, etc) and is not grounds for giving up on Mongo.

Other thoughts

Adding a .pretty() to the end of a find query makes things a lot easier to look at. .findOne is going to upset you if you want to extend the same query
You can add additional features to your Mongo shell, it’s just a Javascript prompt. Try adding some useful libraries, like underscore.js
BSON is not JSON. If you want to send a BSON object across the wire, the BSON library (usually installed with your mongo driver) can handle seralizing objects
facebook.com is a valid ObjectID - so don’t naively try to verify ObjectID strings by casting them to an ObjectID type (I speak from experience).
ObjectIds carry useful date information
Just because it’s schemaless, doesn’t mean you can pretend not to need a schema. You can be much more flexible about how you implement the schema, but if you don’t make it somewhat rigid, you’re going to have a bad time.

Am I wrong? did I miss anything? Let me know!

How do I Gevent?

Sat, 18 May 2013 13:51:00 -0400

So, you think you’re going to sneak one past the GIL and turn your single-threaded snorefest (or your multi-threaded headache) into a den of slick greenlets?

Not so fast, buster. It may not be worth it.

tl;dr Gevent is IO-bound, don’t bother trying it out if you’re doing heavy data processing or anything that spends most of its time on in-memory operations. The pattern in the last code block is a practical basis for multiprocessing systems that depend on IO calls.

Gevent is IO-bound

In particular, as you’ll see in a moment, it monkey-patches the os, select, socket, ssl, thread and time modules to conform to gevent’s cooperative scheduler. As a general rule, if you’re spending all of your time waiting on any of these, gevent will make your application awesome. A good way to gauge is to run your program using the awesome cProfile module

python -m cProfile my_gevent_candidate_program.py

And then look for the time spent on the socket and other IO based libraries

Gevent locks on CPU-bound tasks

This may come back to bite you in the ass on occasion, but it is the expected functionality. If the beautiful routines your application is running suck up CPU time, gevent will not be able to automagically flop between greenlets and chances are, you won’t want it to. There are, of course, exceptions to this rule, but that’s not worth getting into right now.

Gevent Monkey-Patches the python stdlib

This is where things can get really hairy with your dependencies - in general most libraries will play nice with the changes, but there will always be certain ones that just explode in your face when you try to use them with gevent. As far as I know, there’s no definitive list out there yet for these, but based on my present knowledge you’re fairly safe to use the following:

python-requests: if you don’t know it yet, then you’re in for a treat.
pymongo: the official MongoDB driver, also officially supports greenlets (even does some nice pooling of connections to improve your usage)
amqplib: one of two AMQP libraries I’ve tested on gevent, so far no issues
haigha: and here’s the other, this one is a bit more powerful but can be kind of finnicky when it comes to debugging - when in doubt, it’s always turned out to be me doing something stupid though, not haigha. protip: using a second channel for publishing messages seems to fix a lot of socket contention issues.

note: anyone who has tested gevent with other IO libraries: let me know so I can update this list

Gevent requires libevent

Chances are you’re not going to care about this, but you can’t necessarily assume your sysadmin/cloud host/botnet/custom hardware/toaster will support libevent.

Ok ok, just get to the good stuff already…

Alright, so you’ve decided your application needs some gevented magic, or just want to play around with it. Let’s talk a bit about what is happening in the background, so you have a clue when it comes to the foreground.

Welcome to cooperative scheduling

Like in any modern IO system, gevent works by the notion of scheduling. This can be done with a promise-like system (asking things to be done after an async request completes), in a manner that will pretty much hide itself in your code, thanks to those monkey patches from before. The scheduler is built to switch between greenlet contexts quickly and frequently to allow as much airtime to each of your greenlets as they deserve (this is important to remember). Whenever one of these greenlets hits an IO bound job, it sends that through to libevent and then yields to the scheduler to allow for a context switch. Beyond that, the internals are interesting, but not necessary to understand at this point.

Getting started

Let’s start with a simple first-try at a web crawler - we’ll take a seed url and keep following links from there:

import sys
import re
import requests

# Track the urls we've found
crawled = 0

def crawler(u):
    '''A very simple web crawler'''
    global crawled

    # Crawl the page, print the status
    response = requests.get(u)
    print response.status_code, u

    # Extract some links to follow using a *really* bad regular expression
    for link in re.findall('<a href="(http.*?)"', response.content):

        # Limit to 10 pages
        if crawled < 10:
            crawled += 1
            crawler(link)

# Read the seed url from stdin
crawler(sys.argv[1])

The first thing to notice is that this crawler is depth-first recursive. That is an extremely bad idea, but also the easiest to implement in a single-threaded system. We’ve got a time complexity of O(n), since we will load each webpage sequentially. Now that we’ve got our base, let’s see some gevent goodness. Every gevent application should start with a monkey:

import gevent.monkey
gevent.monkey.patch_all()

That’s all you need to set up the gevent scheduler - once the stdlib (or a subset, if that’s what you’re into) has been patched, everything will start piping itself through gevent. This will amount to, well, absolutely no benefit to start off! Your application will run as single threadedly as before, and you may start to feel like reading this far was time that should have been spent finally starting that knitting business you’ve been dreaming about. calm down, the magic is on its way

Let’s get restructuring

Our first basic approach for the gevent crawler looks like this

# monkey-patch
import gevent.monkey
gevent.monkey.patch_all()

import gevent
import sys
import re
import requests

# List for holding our greenlets
greenlets = []

def crawler(u):
    '''A very simple gevented web crawler'''
    global crawled

    # Crawl the page, print the status
    response = requests.get(u)
    print response.status_code, u

    # Extract some links to follow
    for link in re.findall('<a href="(http.*?)"', response.content):

        # Limit to 10 pages
        if len(greenlets) < 10:
            greenlets.append(gevent.spawn(crawler, link))

# Read the seed url from stdin
greenlets.append(gevent.spawn(crawler, sys.argv[1]))

# Wait until we've spawned enough url requests
while len(greenlets) < 10:
    gevent.sleep(1)

# Wait for everything to complete
gevent.joinall(greenlets)

Already looking much better. We’re now breadth-first (a much better idea when it comes to web crawling), and our theoretical time complexity has been brought down to O(2).

This crawler shows off the nice feature that greenlets are cheap. In general, you can parallelize by just adding a greenlets per IO task, and the scheduler will take care of organizing them. Unfortunately, this is far from a useful crawler - we can’t control much about it, and since we’re doing one HTTP request per greenlet we can expect that opening up the tap a bit wider may cause some network issues. So deeper down the rabbit hole we go!

Introducing worker pools

Now, instead of spawning a greenlet per action, we can just tell a pool of greenlets to churn through our actions.

# monkey-patch
import gevent.monkey
gevent.monkey.patch_all()

import gevent.pool
import sys
import re
import requests

# Prepare a pool for 5 workers
pool = gevent.pool.Pool(5)

# Crawl tracker is back
crawled = 0

def crawler(u):
    '''A very simple pooled gevent web crawler'''
    global crawled

    # Crawl the page, print the status
    response = requests.get(u)
    print response.status_code, u

    # Extract some links to follow
    for link in re.findall('<a href="(http.*?)"', response.content):

        # Limit to 10 pages (ignores links when the pool is already full)
        if crawled < 10 and not pool.full():
            crawled += 1
            pool.spawn(crawler, link)

# Read the seed url from stdin
pool.spawn(crawler, sys.argv[1])

# Wait for everything to complete
pool.join()

Pools are joinable, meaning that we can tell our application to wait until all the requested actions have been processed (in gevent-speak, the pool is in a ready state).

In practice this can sometimes cause some headaches, as individual pool actions might not reach the ready state inexplicably (infinte loop, anyone?) and as a result the entire pool may never be safely joined. Nevertheless, our pooling code feels cleaner, except for one block that looks very, very wrong:

# Limit to 10 pages (ignores links when the pool is already full)
if crawled < 10 and not pool.full():
    crawled += 1
    pool.spawn(crawler, link)

Considering how our gevent.spawn example allowed us to do a full breadth-first search, we’re now throwing away results whenever our pool is busy. This is forcing us to do a lossy breadth-first search, and will throw away urls based on parent page load speed. Wacky. But why?

Pool spawning is a blocking action

When you attempt to spawn a thread on a gevent pool, gevent will check if the pool is full, and wait for an availability if it isn’t. This causes some problems - since each of the crawler greenlets is calling pool.spawn, if at any point we already have 5 (our pool size) crawlers active and they all find links, they will all call pool.spawn at the same time, leaving you with some very hungry philosophers.

If you take out the not pool.full() line, your application will happily consume ~5 urls, and then wait patiently for all eternity for a free greenlet, without realizing every greenlet is doing the same!

But we want to crawl everything!

Thankfully, despite the hiccups with using pooling, there’s one more data structure we’ve been missing that will turn our crawler from a silly toy into a powerhouse web crawling application: queueing.

# monkey-patch
import gevent.monkey
gevent.monkey.patch_all()

import gevent.pool
import gevent.queue

import sys
import re
import requests

# Prepare a pool for 5 workers and a messaging queue
pool = gevent.pool.Pool(5)
queue = gevent.queue.Queue()
crawled = 0

def crawler():
    '''A very simple queued gevent web crawler'''
    global crawled

    while 1:
        try:
            u = queue.get(timeout=1)
            response = requests.get(u)
            print response.status_code, u

            # Extract some links to follow
            for link in re.findall('<a href="(http.*?)"', response.content):
                # Limit to 10 pages (ignores links when the pool is already full)
                if crawled < 10:
                    crawled += 1
                    queue.put(link)

        except gevent.queue.Empty:
            break

queue.put(sys.argv[1])

# Read the seed url from stdin
for x in xrange(0, 5):
    pool.spawn(crawler)

# Wait for everything to complete
pool.join()

Looks good, right? It will work, but only under very specific circumstances - specifically we’re depending on the seed url taking less than one second to load. Any more and the rest of the workers are going to exit, leaving us with exactly one worker. But, otherwise, not a bad first attempt - so lets look at what’s happening:

Workers are now their own loops

Previously we were spawning new workers every time we wanted to crawl a url. This fits the greenlets are cheap maxim, but in that same system, each worker would consume exactly one queue message, then exit. Not the most efficient system in the world (you can actually mimic that by just removing the while loop and adding in another small block, we’ll visit that in a minute).

Instead, we spawn each of our greenlets and have it wait for queue messages until no more exist (or until the queue has been empty for longer than one second)

We spawn all of our workers upfront

Previously, we were spawning workers on-demand as we had pool availability and new urls to crawl. Now we’re spawning them all upfront and assuming we’ll fill in the queue as needed. This is not a particularly bad practice (if you’ll recall, greenlets are cheap), but it still feels kind of dirty…

Let’s do one better

In order to optimize our usage of gevent (context switches don’t cost much, but they still have a cost), we want to:

consume every message in the queue or
fill the pool with workers

Whichever is possible, with the obvious caveat that our pool is only going to allow us to fulfill #2 if the queue has already more messages than we are allowing workers.

# monkey-patch
import gevent.monkey
gevent.monkey.patch_all()

import gevent.pool
import gevent.queue

import sys
import re
import requests

# Prepare a pool for 5 workers and a messaging queue
pool = gevent.pool.Pool(5)
queue = gevent.queue.Queue()
crawled = 0

def crawler():
    '''A very simple queued gevent web crawler'''

    print 'starting crawler...'
    global crawled

    while 1:
        try:
            u = queue.get(timeout=0)
            response = requests.get(u)
            print response.status_code, u

            # Extract some links to follow
            for link in re.findall('<a href="(http.*?)"', response.content):
                # Limit to 10 pages (ignores links when the pool is already full)
                if crawled < 10:
                    crawled += 1
                    queue.put(link)

        except gevent.queue.Empty:
            break

    print 'stopping crawler...'

queue.put(sys.argv[1])
pool.spawn(crawler)

while not queue.empty() and not pool.free_count() == 5:
    gevent.sleep(0.1)
    for x in xrange(0, min(queue.qsize(), pool.free_count())):
        pool.spawn(crawler)

# Wait for everything to complete
pool.join()

Running this will show that we spawn exactly one worker to start, which then loads the seed url and pushes the rest of our messages into the queue - after which the other 4 workers start up and we’re crawling!

And now you can Gevent

These pattern form the basis of an IO-bound multiprocessing system, and can be applied to anything that waits on system IO calls (many reads/writes to the local filesystem), local threading and any network IO.

Intro to the gevent.queue module

Mon, 13 May 2013 18:00:00 -0400

A very unofficial overview of all the awesomeness that can be accomplished with the gevent.queue module

import gevent.queue
queue = gevent.queue.Queue()

Putting messages into queues

From examples and my own experience, it is most consistent with good pythonic practice to communicate across queues using tuples. This allows simple packing and unpacking of arguments, and also cascades down to some of the more interesting queue implementations (in particular, priority queues), so as an example, I might do something like the following:

from datetime import datetime
priority = 1
queue.put((priority, datetime.utcnow(), 'here is a message',))

Some specific notes about some of the different queue environments. If you’re working with capped queues (queues with a fixed size, set using the maxsize keyword argument), the queue.put command is going to be blocking - can be quite a surprise when you go from an uncapped to capped queue that your application just locks up. But they thought of that, so you can do either of the following:


try:
    # The preferred method, simple and clear
    queue.put_nowait((priority, datetime.utcnow(), 'here is a message',))

    # queue.put_nowait is actually just an alias of this function
    queue.put((priority, datetime.utcnow(), 'here is a message'), block=False)

except gevent.queue.Queue.Full:
    print 'Could not put to the queue'

Using a timeout on the blocking put allows you to simulate nonblocking, but react accordingly when no result is added

try:
    queue.put((priority, datetime.utcnow(), 'here is a message'), timeout=1)
except gevent.queue.Queue.Full:
    print 'Could not put to the queue'

Getting messages from queues

Most basic applications will use a blocking get, waiting until a new message is in the queue to be processed. Note the tuple unpacking makes the returned data structure very clear (protip: for any situation where you’re going to expect more complex data, I suggest still using a tuple but tailing it with a dictionary and using LINKTHIS JSONSchema to validate messaging)

priority, date_sent, message = queue.get()

By default queue get is a blocking command, but you may need to make it nonblocking - it mirrors the same idioms as the put command

try:
    # Best-practice nonblocking get
    priority, date_sent, message = queue.get_nowait()

    # Which is an alias of this:
    priority, date_sent, message = queue.get(block=False)

    # And can be done with a timeout as well
    priority, date_sent, message = queue.get(timeout=1)

except gevent.queue.Queue.Empty:
    print 'Could not get from the queue'

except ValueError:
    print 'Your tuple unpack was bad, and you should feel bad'

Queue peeking (new in gevent 1.0)

Sometimes, you don’t want to take a message from the queue, just check to see what’s going on on top. The kind developers of gevent have your back on this, with the queue.peek commands. They work identically to the queue.get commands, but are non-destructive.

try:
    # Blocking peek
    priority, date_sent, message = queue.peek()

    # Best-practice nonblocking peek
    priority, date_sent, message = queue.peek_nowait()

    # Which is an alias of this:
    priority, date_sent, message = queue.peek(block=False)

    # And can be done with a timeout as well
    priority, date_sent, message = queue.peek(timeout=1)

except gevent.queue.Queue.Empty:
    print 'Could not peek at the queue'

except ValueError:
    print 'Your tuple unpack was bad, and you should feel bad'

Iterating over queues

We also have the ability to iterate over messages in a queue. This is a blocking action, so could be used to have a keep-alive consumer of a queue while other greenlets are feeding new messages in.

for priority, date, message in queue:
  print 'Got', message, 'from', date, 'with priority', priority

Types of queues and implementations

Out of the box, gevent supports several useful queue implementations: priority queues, joinable queues, lifo queues, as well as a very simplistic interface for defining your own queue subclasses.

Priority queueing (AKA: why we use tuples)

priority_queue = gevent.queue.PriorityQueue()

Priority queues, internally, are built ontop of the amazing LINKTHIS heapq module in the python standard library. Any logic you would use in implementations of lists using heapq applies to gevent priority queues as well, so in a situation like the following:

Gevent priority queues 
priority_queue.put((1, datetime.now(), 'countdown 3'))
priority_queue.put((1, datetime.now() - timedelta(minutes=1), 'countdown 2'))
priority_queue.put((0, datetime.now(), 'countdown 1'))
priority_queue.put((0, datetime.now() - timedelta(minutes=1), 'countdown - blastoff!'))

You can analytically determine that pulling all messages from the queue will yield the following:

(0, datetime.datetime(...), 'countdown - blastoff!')
(0, datetime.datetime(...), 'countdown 1')
(1, datetime.datetime(...), 'countdown 2')
(1, datetime.datetime(...), 'countdown 3')

I would highly recommend reading through the heapq documentation a bit more to ensure you implement your comparison algorithms properly (I often get them backwards) and also because it is surprisingly useful in general.

Joinable queues

When using queues in gevent, you may find that you want to run a program only until there are no messages left in a queue, perhaps because you know it recurses in a predictable manner, or you just have a finite set of messages you want to process before exiting. Enter the joinable queue, which allows you to set up a blocking environment to live until the queue is empty

joinable_queue = gevent.queue.JoinableQueue()
joinable_queue.put(('wait for me...',))

# some code to spawn workers goes here, otherwise you're going to be waiting a LONG time

joinable_queue.join()

Lifo queues

Blah blah, rhetoric about lifo queues, there’s nothing special about the usage, this is just here for completeness.

lifo_queue = gevent.queue.LifoQueue()

Subclassing queues

Extending the queue class is simple, and can be done with a pattern something like the following:

import random

class ChaosQueue(gevent.queue.Queue):
    '''A subclass of Queue that retrieves a random queue entry'''

    def _init(self, maxsize, items=None):
        if items:
            self.queue = list(items)
        else:
            self.queue = []

    def _put(self, item):
        self.queue.append(item)

    def _get(self):
        return self.queue.pop(random.randint(0, len(self.queue) - 1))

And that, as far as I know, is the extent of the gevent queuing module. Let me know if there’s anything I missed or any errors in my sample code!

One-liner while loop in bash shell

Thu, 02 May 2013 16:39:00 -0400

To keep looping on a command forever in the shell, use the following (runs COMMAND every 5 seconds)

$ while true; do COMMAND; sleep 5; done;

As it turns out, this can be surprisingly useful - embarrassed to say that I was using cron to accomplish the same thing for awhile…

Relocating RabbitMQ files

Tue, 30 Apr 2013 14:59:00 -0400

RabbitMQ, by default (on ubuntu-like systems) will load this file, if it exists:

/etc/rabbitmq/rabbitmq-env.conf

This (not /etc/environment) is where you should be putting any of the variables found on http://www.rabbitmq.com/relocate.html

How to install python lxml on ubuntu

Wed, 20 Mar 2013 11:57:00 -0400

Since I can never seem to get it right:

apt-get install libxml2-dev libxslt-dev
pip install lxml

or with python-libxml2

apt-get install python-libxml2 libxslt-dev
STATIC_DEPS=true pip install lxml

Where is my user-data script stored on EC2?

Wed, 13 Mar 2013 10:33:00 -0400

On EC2 Ubuntu instances, the user-data script supplied on launch can be found in:

/var/lib/cloud/instances/[instance-id]/user-data.txt

Note: there may be several if you, like me, tend to launch instances from copies of copies of instances!