Hi! If you enjoy this post, you might want to or .

(for a Belorussian translation of this article, )

Want to speed up your with ? Have a lot of files to process, but don't know how to NLTK across many cores?

Well, here's how you can use to do distributed of with .

execnet

is a simple library for creating a network of and that you can use for distributed computation in . With it, you can start shells over , send code and/or data, then receive results. Below are 2 scripts that will test the accuracy of against every file in the . The first script (the runner) does all the setup and receives the results, while the second script (the remote module) runs on every gateway, calculating and sending the accuracy of each file it receives for processing.

Runner

The runner does the following:

1. Defines the hosts and number of gateways. I recommend 1 gateway per core per host.
2. Loads and the default NLTK part of speech tagger.
3. Opens each gateway and creates a remote execution channel with the tag_files module (the remote module covered below).
4. Sends the pickled tagger and the name of a corpus (brown) thru the channel.
5. Once all the channels have been created and initialized, it then sends all of the in the corpus to alternating channels to distribute the work.
6. Finally, it creates a and prints the accuracy response from each channel.

run_tag_files.py

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

import execnet import nltk.tag, nltk.data import cPickle as pickle import tag_files   HOSTS = {      'localhost' : 2 }   NICE = 20   channels = []   tagger = pickle.dumps(nltk.data.load(nltk.tag._POS_TAGGER))   for host, count in HOSTS.items():      print 'opening %d gateways at %s' % (count, host)        for i in range (count):          gw = execnet.makegateway( 'ssh=%s//nice=%d' % (host, NICE))          channel = gw.remote_exec(tag_files)          channels.append(channel)          channel.send(tagger)          channel.send( 'brown' )   count = 0 chan = 0   for fileid in nltk.corpus.brown.fileids():      print 'sending %s to channel %d' % (fileid, chan)      channels[chan].send(fileid)      count + = 1      # alternate channels      chan + = 1      if chan > = len (channels): chan = 0   multi = execnet.MultiChannel(channels) queue = multi.make_receive_queue()   for i in range (count):      channel, response = queue.get()      print response

Remote Module

The remote module is much simpler.

1. Receives and unpickles the tagger.
2. Receives the corpus name and loads it.
3. For each fileid received, evaluates the accuracy of the tagger on the tagged sentences and sends an accuracy response.

tag_files.py

1 2 3 4 5 6 7 8 9 10 11

import nltk.corpus import cPickle as pickle   if __name__ = = '__channelexec__' :      tagger = pickle.loads(channel.receive())      corpus_name = channel.receive()      corpus = getattr (nltk.corpus, corpus_name)        for fileid in channel:          accuracy = tagger.evaluate(corpus.tagged_sents(fileids = [fileid]))          channel.send( '%s: %f' % (fileid, accuracy))

Putting it all together

Make sure you have and the installed on every host. You must also have access to each host from the master host (the machine you run run_tag_files.py on).

run_tag_files.py and tag_files.py only need to be on the master host; will take care of distributing the code. Assuming run_tag_files.py and tag_files.py are in the same directory, all you need to do is run python run_tag_files.py. You should get a message about opening gateways followed by a bunch of send messages. Then, just wait and watch the accuracy responses to see how accurate the built in part of speech tagger is on the brown corpus.

If you'd like test the accuracy of a different corpus, make sure every host has the corpus data, then send that corpus name instead of brown, and send the fileids from the new corpus.

If you want to test your own , pickle it to a file, then load and send it instead of NLTK's tagger. Or you can train it on the master first, then send it once training is complete.

Distributed File Processing

In practice, it's often a to make sure every host has every file you want to process, and you'll want to process files outside of NLTK's builtin corpora. My recommendation is to setup a so that every host has a common mount point with access to every file that you want to process. If every host has the same mount point, you can send any file path to any channel for processing.