Tutorial

This tutorial will explain a couple of basic Pyro concepts, a little bit about the name server, and you’ll learn to write a simple Pyro application. You’ll do this by writing a warehouse system and a stock market simulator, that demonstrate some key Pyro techniques.

Warm-up

Before proceeding, you should install Pyro if you haven’t done so. For instructions about that, see Installing Pyro.

In this tutorial, you will use Pyro’s default configuration settings, so once Pyro is installed, you’re all set! All you need is a text editor and a couple of console windows. During the tutorial, you are supposed to run everything on a single machine. This avoids initial networking complexity.

Note

For security reasons, Pyro runs stuff on localhost by default. If you want to access things from different machines, you’ll have to tell Pyro to do that explicitly. At the end is a small paragraph Running it on different machines that tells you how you can run the various components on different machines.

Note

The code of the two tutorial ‘projects’ is included in the Pyro source archive. Just installing Pyro won’t provide this. If you don’t want to type all the code, you should extract the Pyro source archive (Pyro4-X.Y.tar.gz) somewhere. You will then have an examples directory that contains a truckload of examples, including the two tutorial projects we will be creating later in this tutorial, warehouse and stockquotes. (There is more in there as well: the tests directory contains the test suite with all the unittests for Pyro’s code base.)

Pyro concepts and tools

Pyro enables code to call methods on objects even if that object is running on a remote machine:

+----------+                         +----------+
| server A |                         | server B |
|          |       < network >       |          |
| Python   |                         |   Python |
| OBJECT ----------foo.invoke()--------> OBJECT |
|          |                         |     foo  |
+----------+                         +----------+

Pyro is mainly used as a library in your code but it also has several supporting command line tools [1]. We won’t explain every one of them here as you will only need the “name server” for this tutorial.

[1]Actually there are no scripts or command files included with Pyro right now. The Command line tools are invoked by starting their package directly using the -m argument of the Python interpreter.

Key concepts

Here are a couple of key concepts you encounter when using Pyro:

Proxy
A proxy is a substitute object for “the real thing”. It intercepts the method calls you would normally do on an object as if it was the actual object. Pyro then performs some magic to transfer the call to the computer that contains the real object, where the actual method call is done, and the results are returned to the caller. This means the calling code doesn’t have to know if it’s dealing with a normal or a remote object, because the code is identical. The class implementing Pyro proxies is Pyro4.core.Proxy
URI
This is what Pyro uses to identify every object. (similar to what a web page URL is to point to the different documents on the web). Its string form is like this: “PYRO:” + object name + “@” + server name + port number. There are a few other forms it can take as well. You can write the protocol in lowercase too if you want (“pyro:”) but it will automatically be converted to uppercase internally. The class implementing Pyro uris is Pyro4.core.URI
Pyro object
This is a normal Python object but it is registered with Pyro so that you can access it remotely. Pyro objects are written just as any other object but the fact that Pyro knows something about them makes them special, in the way that you can call methods on them from other programs.
Pyro daemon (server)
This is the part of Pyro that listens for remote method calls, dispatches them to the appropriate actual objects, and returns the results to the caller. All Pyro objects are registered in one or more daemons.
Pyro name server
The name server is a utility that provides a phone book for Pyro applications: you use it to look up a “number” by a “name”. The name in Pyro’s case is the logical name of a remote object. The number is the exact location where Pyro can contact the object. Usually there is just one name server running in your network.
Serialization
This is the process of transforming objects into streams of bytes that can be transported over the network. The receiver deserializes them back into actual objects. Pyro needs to do this with all the data that is passed as arguments to remote method calls, and their response data. Not all objects can be serialized, so it is possible that passing a certain object to Pyro won’t work even though a normal method call would accept it just fine.

Starting a name server

While the use of the Pyro name server is optional, we will use it in this tutorial. It also shows a few basic Pyro concepts, so let us begin by explaining a little about it. Open a console window and execute the following command to start a name server:

python -Wignore -m Pyro4.naming

The -Wignore is added to suppress a Pyro warning that we’re not interested in right now. The name server will start and it prints something like:

Not starting broadcast server for localhost.
NS running on localhost:9090 (127.0.0.1)
URI = PYRO:Pyro.NameServer@localhost:9090

The name server has started and is listening on localhost port 9090.

It also printed an URI. Remember that this is what Pyro uses to identify every object.

The name server can be stopped with a control-c, or on Windows, with ctrl-break. But let it run in the background for the rest of this tutorial.

Interacting with the name server

There’s another command line tool that let you interact with the name server: “nsc” (name server control tool). You can use it, amongst other things, to see what all known registered objects in the naming server are. Let’s do that right now. Type:

python -Wignore -m Pyro4.nsc list

(the -Wignore again is to suppress a warning) and it will print something like this:

--------START LIST
Pyro.NameServer --> PYRO:Pyro.NameServer@localhost:9090
--------END LIST

The only object that is currently registered, is the name server itself! (Yes, the name server is a Pyro object itself. Pyro and the “nsc” tool are using Pyro to talk to it).

Note

As you can see, the name Pyro.NameServer is registered to point to the URI that we saw earlier. This is mainly for completeness sake, and is not often used, because there are different ways to get to talk to the name server (see below).

This is cool, but there’s a little detail left unexplained: How did the nsc tool know where the name server was? Pyro has a couple of tactics to locate a name server. The nsc tool uses them too: Pyro uses a network broadcast to see if there’s a name server available somewhere (the name server contains a broadcast responder that will respond “Yeah hi I’m here”). So in many cases you won’t have to configure anything to be able to discover the name server. If nobody answers though, Pyro tries the configured default or custom location. If still nobody answers it prints a sad message and exits. However if it found the name server, it is then possible to talk to it and get the location of any other registered object. . This means that you won’t have to hard code any object locations in your code, and that the code is capable of dynamically discovering everything at runtime.

But enough of that. We need to start looking at how to actually write some code ourselves that uses Pyro!

Building a Warehouse

Hint

All code of this part of the tutorial can be found in the examples/warehouse directory.

You’ll build build a simple warehouse that stores items, and that everyone can visit. Visitors can store items and retrieve other items from the warehouse (if they’ve been stored there).

In this tutorial you’ll first write a normal Python program that more or less implements the complete warehouse system, but in vanilla Python code. After that you’ll add Pyro support to it, to make it a distributed warehouse system, where you can visit the central warehouse from many different computers.

phase 1: a simple prototype

To start with, write the vanilla Python code for the warehouse and its visitors. This prototype is fully working but everything is running in a single process. It contains no Pyro code at all, but shows what the system is going to look like later on.

The Warehouse object simply stores an array of items which we can query, and allows for a person to take an item or to store an item. Here is the code (warehouse.py):

from __future__ import print_function

class Warehouse(object):
    def __init__(self):
        self.contents = ["chair", "bike", "flashlight", "laptop", "couch"]

    def list_contents(self):
        return self.contents

    def take(self, name, item):
        self.contents.remove(item)
        print("{0} took the {1}.".format(name, item))

    def store(self, name, item):
        self.contents.append(item)
        print("{0} stored the {1}.".format(name, item))

Then there is a Person that can visit the warehouse. The person has a name and deposit and retrieve actions on a particular warehouse. Here is the code (person.py):

from __future__ import print_function
import sys

if sys.version_info < (3, 0):
    input = raw_input


class Person(object):
    def __init__(self, name):
        self.name = name

    def visit(self, warehouse):
        print("This is {0}.".format(self.name))
        self.deposit(warehouse)
        self.retrieve(warehouse)
        print("Thank you, come again!")

    def deposit(self, warehouse):
        print("The warehouse contains:", warehouse.list_contents())
        item = input("Type a thing you want to store (or empty): ").strip()
        if item:
            warehouse.store(self.name, item)

    def retrieve(self, warehouse):
        print("The warehouse contains:", warehouse.list_contents())
        item = input("Type something you want to take (or empty): ").strip()
        if item:
            warehouse.take(self.name, item)

Finally you need a small script that actually runs the code. It creates the warehouse and two visitors, and makes the visitors perform their actions in the warehouse. Here is the code (visit.py):

# This is the code that runs this example.
from warehouse import Warehouse
from person import Person

warehouse = Warehouse()
janet = Person("Janet")
henry = Person("Henry")
janet.visit(warehouse)
henry.visit(warehouse)

Run this simple program. It will output something like this:

$ python visit.py
This is Janet.
The warehouse contains: ['chair', 'bike', 'flashlight', 'laptop', 'couch']
Type a thing you want to store (or empty): television   # typed in
Janet stored the television.
The warehouse contains: ['chair', 'bike', 'flashlight', 'laptop', 'couch', 'television']
Type something you want to take (or empty): couch    # <-- typed in
Janet took the couch.
Thank you, come again!
This is Henry.
The warehouse contains: ['chair', 'bike', 'flashlight', 'laptop', 'television']
Type a thing you want to store (or empty): bricks   # <-- typed in
Henry stored the bricks.
The warehouse contains: ['chair', 'bike', 'flashlight', 'laptop', 'television', 'bricks']
Type something you want to take (or empty): bike   # <-- typed in
Henry took the bike.
Thank you, come again!

phase 2: first Pyro version

That wasn’t very exciting but you now have working code for the basics of the warehouse system. Now you’ll use Pyro to turn the warehouse into a standalone component, that people from other computers can visit. You’ll need to add a couple of lines to the warehouse.py file so that it will start a Pyro server for the warehouse object. The easiest way to do this is to create the object that you want to make available as Pyro object, and register it with a ‘Pyro daemon’ (the server that listens for and processes incoming remote method calls):

warehouse = Warehouse()
Pyro4.Daemon.serveSimple(
        {
            warehouse: "example.warehouse"
        },
        ns = False)

You’ll also need to add a little main function so it will be started correctly, which should make the code now look like this (warehouse.py):

from __future__ import print_function
import Pyro4
import person


class Warehouse(object):
    def __init__(self):
        self.contents = ["chair", "bike", "flashlight", "laptop", "couch"]

    def list_contents(self):
        return self.contents

    def take(self, name, item):
        self.contents.remove(item)
        print("{0} took the {1}.".format(name, item))

    def store(self, name, item):
        self.contents.append(item)
        print("{0} stored the {1}.".format(name, item))


def main():
    warehouse = Warehouse()
    Pyro4.Daemon.serveSimple(
            {
                warehouse: "example.warehouse"
            },
            ns = False)

if __name__=="__main__":
    main()

Start the warehouse in a new console window, it will print something like this:

$ python warehouse.py
Object <__main__.Warehouse object at 0x025F4FF0>:
    uri = PYRO:example.warehouse@localhost:51279
Pyro daemon running.

It will become clear what you need to do with this output in a second. You now need to slightly change the visit.py script that runs the thing. Instead of creating a warehouse directly and letting the persons visit that, it is going to use Pyro to connect to the stand alone warehouse object that you started above. It needs to know the location of the warehouse object before it can connect to it. This is the uri that is printed by the warehouse program above (PYRO:example.warehouse@localhost:51279). You’ll need to ask the user to enter that uri string into the program, and use Pyro to create a proxy to the remote object:

uri = input("Enter the uri of the warehouse: ").strip()
warehouse = Pyro4.Proxy(uri)

That is all you need to change. Pyro will transparently forward the calls you make on the warehouse object to the remote object, and return the results to your code. So the code will now look like this (visit.py):

# This is the code that visits the warehouse.
import sys
import Pyro4
from person import Person

if sys.version_info<(3,0):
    input = raw_input

uri = input("Enter the uri of the warehouse: ").strip()
warehouse = Pyro4.Proxy(uri)
janet = Person("Janet")
henry = Person("Henry")
janet.visit(warehouse)
henry.visit(warehouse)

Notice that the code of Warehouse and Person classes didn’t change at all.

Run the program. It will output something like this:

$ python visit.py
Enter the uri of the warehouse: PYRO:example.warehouse@localhost:51279  # copied from warehouse output
This is Janet.
The warehouse contains: ['chair', 'bike', 'flashlight', 'laptop', 'couch']
Type a thing you want to store (or empty): television   # typed in
The warehouse contains: ['chair', 'bike', 'flashlight', 'laptop', 'couch', 'television']
Type something you want to take (or empty): couch   # <-- typed in
Thank you, come again!
This is Henry.
The warehouse contains: ['chair', 'bike', 'flashlight', 'laptop', 'television']
Type a thing you want to store (or empty): bricks   # <-- typed in
The warehouse contains: ['chair', 'bike', 'flashlight', 'laptop', 'television', 'bricks']
Type something you want to take (or empty): bike    # <-- typed in
Thank you, come again!

And notice that in the other console window, where the warehouse server is running, the following is printed:

Janet stored the television.
Janet took the couch.
Henry stored the bricks.
Henry took the bike.

phase 3: final Pyro version

The code from the previous phase works fine and could be considered to be the final program, but is a bit cumbersome because you need to copy-paste the warehouse URI all the time to be able to use it. You will simplify it a bit in this phase by using the Pyro name server. Also, you will use the Pyro excepthook to print a nicer exception message if anything goes wrong (by taking something from the warehouse that is not present! Try that now with the code from phase 2. You will get a ValueError: list.remove(x): x not in list but with a not so useful stack trace).

Note

Once again you can leave code of the Warehouse and Person classes unchanged. As you can see, Pyro is not getting in your way at all here. You can often use it with only adding a couple of lines to your existing code.

Okay, stop the warehouse program from phase 2 if it is still running, and check if the name server that you started in Starting a name server is still running in its own console window.

In warehouse.py locate the statement Pyro4.Daemon.serveSimple(... and change the ns = False argument to ns = True. This tells Pyro to use a name server to register the objects in. (The Pyro4.Daemon.serveSimple is a very easy way to start a Pyro server but it provides very little control. You will learn about another way of starting a server in Building a Stock market simulator).

In visit.py remove the input statement that asks for the warehouse uri, and change the way the warehouse proxy is created. Because you are now using a name server you can ask Pyro to locate the warehouse object automatically:

warehouse = Pyro4.Proxy("PYRONAME:example.warehouse")

Finally, install the Pyro4.util.excepthook as excepthook. You’ll soon see what this does to the exceptions and stack traces your program produces when something goes wrong with a Pyro object. So the code should look something like this (visit.py):

# This is the code that visits the warehouse.
import sys
import Pyro4
import Pyro4.util
from person import Person

sys.excepthook = Pyro4.util.excepthook

warehouse = Pyro4.Proxy("PYRONAME:example.warehouse")
janet = Person("Janet")
henry = Person("Henry")
janet.visit(warehouse)
henry.visit(warehouse)

Start the warehouse program again in a separate console window. It will print something like this:

$ python warehouse.py
Object <__main__.Warehouse object at 0x02496050>:
    uri = PYRO:obj_426e82eea7534fb5bc78df0b5c0b6a04@localhost:51294
    name = example.warehouse
Pyro daemon running.

As you can see the uri is different this time, it now contains some random id code instead of a name. However it also printed an object name. This is the name that is now used in the name server for your warehouse object. Check this with the ‘nsc’ tool: python -m Pyro4.nsc list, which will print something like:

--------START LIST
Pyro.NameServer --> PYRO:Pyro.NameServer@localhost:9090
example.warehouse --> PYRO:obj_426e82eea7534fb5bc78df0b5c0b6a04@localhost:51294
--------END LIST

This means you can now refer to that warehouse object using the name example.warehouse and Pyro will locate the correct object for you automatically. This is what you changed in the visit.py code so run that now to see that it indeed works!

Remote exception: You also installed Pyro’s custom excepthook so try that out. Run the visit.py script and try to take something from the warehouse that is not present (for instance, batteries):

Type something you want to take (or empty): batteries
Traceback (most recent call last):
  File "visit.py", line 12, in <module>
    janet.visit(warehouse)
  File "d:\PROJECTS\Pyro4\examples\warehouse\phase3\person.py", line 14, in visit
    self.retrieve(warehouse)
  File "d:\PROJECTS\Pyro4\examples\warehouse\phase3\person.py", line 25, in retrieve
    warehouse.take(self.name, item)
  File "d:\PROJECTS\Pyro4\src\Pyro4\core.py", line 161, in __call__
    return self.__send(self.__name, args, kwargs)
  File "d:\PROJECTS\Pyro4\src\Pyro4\core.py", line 314, in _pyroInvoke
    raise data
ValueError: list.remove(x): x not in list
 +--- This exception occured remotely (Pyro) - Remote traceback:
 | Traceback (most recent call last):
 |   File "d:\PROJECTS\Pyro4\src\Pyro4\core.py", line 824, in handleRequest
 |     data=method(*vargs, **kwargs)   # this is the actual method call to the Pyro object
 |   File "warehouse.py", line 14, in take
 |     self.contents.remove(item)
 | ValueError: list.remove(x): x not in list
 +--- End of remote traceback

What you can see now is that you not only get the usual exception traceback, but also the exception that occurred in the remote warehouse object on the server (the “remote traceback”). This can greatly help locating problems! As you can see it contains the source code lines from the warehouse code that is running in the server, as opposed to the normal local traceback that only shows the remote method call taking place inside Pyro...

Building a Stock market simulator

Hint

All of the code of this part of the tutorial can be found in the examples/stockquotes directory.

You’ll build a simple stock quote system. The idea is that we have multiple stock markets producing stock symbol quotes. There is an aggregator that combines the quotes from all stock markets. Finally there are multiple viewers that can register themselves by the aggregator and let it know what stock symbols they’re interested in. The viewers will then receive near-real-time stock quote updates for the symbols they selected. (Everything is fictional, of course):

Stockmarket 1   Viewer 1
Stockmarket 2 Aggregator Viewer 2
...       ...
Stockmarket N   Viewer N

phase 1: simple prototype

Again, like the previous application (the warehouse), you first create a working version of the system by only using normal Python code. This simple prototype will be functional but everything will be running in a single process. It contains no Pyro code at all, but shows what the system is going to look like later on.

First create a file stockmarket.py that will simulate a stock market that is producing stock quotes for registered companies. You should be able to add a ‘listener’ to it that will be receiving stock quote updates. It should be able to report the stock symbols that are being traded in this market as well. The code is as follows:

# stockmarket.py
import random

class StockMarket(object):
    def __init__(self, marketname, symbols):
        self.name = marketname
        self.symbolmeans = {}
        for symbol in symbols:
            self.symbolmeans[symbol] = random.uniform(20, 200)
        self.aggregators = []

    def generate(self):
        quotes = {}
        for symbol, mean in self.symbolmeans.items():
            if random.random() < 0.2:
                quotes[symbol] = round(random.normalvariate(mean, 20), 2)
        for aggregator in self.aggregators:
            aggregator.quotes(self.name, quotes)

    def listener(self,aggregator):
        self.aggregators.append(aggregator)

    def symbols(self):
        return self.symbolmeans.keys()

Then we need an aggregator.py that combines all stock symbol quotes from all stockmarkets into one ‘stream’ (this is the object that will be the ‘listener’ for the stock markets). It should be possible to register one or more ‘viewers’ with the stock symbols that viewer is interested in, so that every viewer is only receiving the stock symbols it wants. The code is like this:

# aggregator.py
class Aggregator(object):
    def __init__(self):
        self.viewers = {}
        self.symbols = []

    def add_symbols(self, symbols):
        self.symbols.extend(symbols)

    def available_symbols(self):
        return self.symbols

    def view(self, viewer, symbols):
        self.viewers[viewer] = symbols

    def quotes(self, market, stockquotes):
        for symbol, value in stockquotes.items():
            for viewer, symbols in self.viewers.items():
                if symbol in symbols:
                    viewer.quote(market, symbol, value)

The viewer.py itself is an extremely simple object that just prints out the stock symbol quotes it receives:

# viewer.py
from __future__ import print_function

class Viewer(object):
    def quote(self, market, symbol, value):
        print("{0}.{1}: {2}".format(market, symbol, value))

Finally you need to write a main.py that imports the above modules, creates all objects, creates a few companies that are traded on the market, connects them together, and contains a loop that drives the stock market quote generation. Because we are not yet using Pyro here, it just creates two Stockmarket objects (with a name and the companies being traded). A single Aggregator object is registered with both markets, to receive all updates. A Viewer object is created and connected to the Aggregator with a few companies that the viewer wants to receive quotes from. The code is like this:

# main.py
from __future__ import print_function
import time
from stockmarket import StockMarket
from aggregator import Aggregator
from viewer import Viewer

def main():
    nasdaq = StockMarket("NASDAQ", ["AAPL", "CSCO", "MSFT", "GOOG"])
    newyork = StockMarket("NYSE", ["IBM", "HPQ", "BP"])

    agg = Aggregator()
    agg.add_symbols(nasdaq.symbols())
    agg.add_symbols(newyork.symbols())
    print("aggregated symbols:", agg.available_symbols())

    nasdaq.listener(agg)
    newyork.listener(agg)

    view = Viewer()
    agg.view(view, ["IBM", "AAPL", "MSFT"])
    print("")
    while True:
        nasdaq.generate()
        newyork.generate()
        time.sleep(0.5)

if __name__ == "__main__":
    main()

If you now run main.py it will print a stream of stock symbol quote updates that are being generated by the two stock markets (but only the few symbols that the viewer wants to see):

$ python main.py
aggregated symbols: ['GOOG', 'AAPL', 'CSCO', 'MSFT', 'HPQ', 'BP', 'IBM']

NYSE.IBM: 74.31
NYSE.IBM: 108.68
NASDAQ.AAPL: 64.17
NYSE.IBM: 83.19
NYSE.IBM: 92.5
NASDAQ.AAPL: 63.09
NASDAQ.MSFT: 161.3
....

phase 2: separation

Note

For brevity, the code for this phase of the stockquote tutorial is not shown. If you want to see it, have a look at the stockquotes example, phase2.

This phase still contains no Pyro code, but you can already make the three components more autonomous than they were in phase 1. This step is optional, you can skip it and continue with phase 3: Pyro version below if you want.

In this phase, every component of our stock market system now has a main function that starts up the component and connects it to the other component(s). As the stock market is the source of the data, you create a daemon thread in stockmarket.py for it so that all markets produces stock quote changes in the background. main.py is a lot simpler now: it only starts the various components and then sits to wait for an exit signal.

The idea in this phase is that you tweak the existing code a little to make it suitable to be split up in different, autonomous components:

  • it helps to add a few debug print or log statements so that you can see what is going on in each component
  • each component will need some form of a ‘main’ or ‘startup’ function to create and launch it
  • the main program just needs to make sure the components are started.

phase 3: Pyro version

Finally you use Pyro to make the various components fully distributed. Pyro is used to make them talk to each other. The actual code for each component class hasn’t changed since phase 1, it is just the plumbing that you need to write to glue them together. Pyro is making this a matter of just a few lines of code that is Pyro-specific, the rest of the code is needed anyway to start up and configure the system. To be able to see the final result, the code is listed once more with comments on what changed with respect to the version in phase 1 (phase 2 is optional, it just makes for an easier transition).

Note

This time we won’t be using serveSimple to publish the objects and start the Daemon. Instead, a daemon is created manually, we register our own objects, and start the request loop ourselves. This needs more code but gives you more control.

Note

You will see the following lines appear in the code:

Pyro4.config.SERIALIZER = 'pickle' and Pyro4.config.SERIALIZERS_ACCEPTED.add('pickle')

For now, ignore the exact meaning of these particular settings. It is needed to get the stock market tutorial running in the form presented here. Basically it enables Pyro to transfer actual Python objects to remote calls, instead of only simple types such as lists and strings. In other chapters the meaning of this setting will be explained in more detail.

main

There’s no main.py anymore. This is because you now start every component by itself, in separate console windows for instance. They run autonomously without the need of a ‘main’ program to start it all up in one place.

stockmarket

The stockmarket.py gained a few print statements to see what is going on while it is running. Important: there is a single change in the code to make it work with Pyro. Because Pyro needs to transfer objects over the network, it requires those objects to be serializable. The symbols method returned the keys() of the dictionary of symbols in the stockmarket. While this is a normal list in Python 2, it is a dict_keys object in Python 3. These cannot be serialized (because it is a special iterator object). The simple solution is to force the method to build a list and return that: list(dictionary.keys()).

It also gained a run method that will be running inside the background thread to generate stock quote updates. The reason this needs to run in a thread is because the Stockmarket itself is also a Pyro object that must listen to remote method calls (in this case, of the Aggregator object(s) that want to listen to it). You can also choose to run the Pyro daemon loop in a background thread and generate stock quotes update in the main thread, that doesn’t matter, as long as they run independently.

Finally it gained a main function to create a couple of stock markets as we did before. This time however they’re registered as Pyro objects with the Pyro daemon. They’re also entered in the name server as example.stockmarket.<name> so the Aggregator can find them easily.

The complete code for stockmarket.py is now as follows:

# stockmarket.py
from __future__ import print_function
import random
import threading
import time
import Pyro4

Pyro4.config.SERIALIZER = 'pickle'
Pyro4.config.SERIALIZERS_ACCEPTED.add('pickle')


class StockMarket(object):
    def __init__(self, marketname, symbols):
        self.name = marketname
        self.symbolmeans = {}
        for symbol in symbols:
            self.symbolmeans[symbol] = random.uniform(20, 200)
        self.aggregators = []

    def generate(self):
        quotes = {}
        for symbol, mean in self.symbolmeans.items():
            if random.random() < 0.2:
                quotes[symbol] = round(random.normalvariate(mean, 20), 2)
        print("new quotes generated for", self.name)
        for aggregator in self.aggregators:
            aggregator.quotes(self.name, quotes)

    def listener(self,aggregator):
        print("market {0} adding new aggregator".format(self.name))
        self.aggregators.append(aggregator)

    def symbols(self):
        return list(self.symbolmeans.keys())

    def run(self):
        def generate_symbols():
            while True:
                time.sleep(random.random())
                self.generate()
        thread = threading.Thread(target=generate_symbols)
        thread.setDaemon(True)
        thread.start()


def main():
    nasdaq = StockMarket("NASDAQ", ["AAPL", "CSCO", "MSFT", "GOOG"])
    newyork = StockMarket("NYSE", ["IBM", "HPQ", "BP"])

    daemon = Pyro4.Daemon()
    nasdaq_uri = daemon.register(nasdaq)
    newyork_uri = daemon.register(newyork)
    ns = Pyro4.locateNS()
    ns.register("example.stockmarket.nasdaq", nasdaq_uri)
    ns.register("example.stockmarket.newyork", newyork_uri)

    nasdaq.run()
    newyork.run()
    print("Stockmarkets running.")
    daemon.requestLoop()

if __name__ == "__main__":
    main()

aggregator

The aggregator.py also gained a print function to be able to see in its console window what is going on when a new viewer connects. The main function creates it, and connects it as a Pyro object to the Pyro daemon. It also registers it with the name server as example.stockquote.aggregator so it can be easily retrieved by any viewer that is interested.

How it connects to the available stock markets: Remember that the stock market objects registered with the name server using a name of the form example.stockmarket.<name>. It is possible to query the Pyro name server in such a way that it returns a list of all objects matching a name pattern. This is exactly what the aggregator does, it asks for all names starting with example.stockmarket. and for each of those, creates a Pyro proxy to that stock market. It then registers itself as a listener with that remote stock market object. Finally it starts the daemon loop to wait for incoming calls from any interested viewers.

The complete code for aggregator.py is now as follows:

# aggregator.py
from __future__ import print_function
import Pyro4

Pyro4.config.SERIALIZER = 'pickle'
Pyro4.config.SERIALIZERS_ACCEPTED.add('pickle')


class Aggregator(object):
    def __init__(self):
        self.viewers = {}
        self.symbols = []

    def add_symbols(self, symbols):
        self.symbols.extend(symbols)

    def available_symbols(self):
        return self.symbols

    def view(self, viewer, symbols):
        print("aggregator gets a new viewer, for symbols:", symbols)
        self.viewers[viewer] = symbols

    def quotes(self, market, stockquotes):
        for symbol, value in stockquotes.items():
            for viewer, symbols in self.viewers.items():
                if symbol in symbols:
                    viewer.quote(market, symbol, value)


def main():
    aggregator = Aggregator()
    daemon = Pyro4.Daemon()
    agg_uri = daemon.register(aggregator)
    ns = Pyro4.locateNS()
    ns.register("example.stockquote.aggregator", agg_uri)
    for market, market_uri in ns.list(prefix="example.stockmarket.").items():
        print("joining market", market)
        stockmarket = Pyro4.Proxy(market_uri)
        stockmarket.listener(aggregator)
        aggregator.add_symbols(stockmarket.symbols())
    if not aggregator.available_symbols():
        raise ValueError("no symbols found! (have you started the stock market first?)")
    print("Aggregator running. Symbols:", aggregator.available_symbols())
    daemon.requestLoop()

if __name__ == "__main__":
    main()

viewer

You don’t need to change the Viewer at all, besides the main function that needs to be added to start it up by itself. It needs to create a viewer object and register it with a Pyro daemon to be able to receive stock quote update calls. You can connect it to a running aggregator simply by asking Pyro to look that up in the name server. That can be done by using the special PYRONAME:<object name> uri format. For the aggregator that would be: PYRONAME:example.stockquote.aggregator (because example.stockquote.aggregator is the name the aggregator used to register itself with the name server).

It is also nice to ask the user for a list of stock symbols he is interested in so do that and register the viewer with the aggregator, passing the list of entered stock symbols to filter on.

Finally start the daemon loop to wait for incoming calls. The code is as follows:

# viewer.py
from __future__ import print_function
import sys
import Pyro4

Pyro4.config.SERIALIZER = 'pickle'
Pyro4.config.SERIALIZERS_ACCEPTED.add('pickle')

if sys.version_info < (3,0):
    input = raw_input


class Viewer(object):
    def quote(self, market, symbol, value):
        print("{0}.{1}: {2}".format(market, symbol, value))


def main():
    viewer = Viewer()
    daemon = Pyro4.Daemon()
    daemon.register(viewer)
    aggregator = Pyro4.Proxy("PYRONAME:example.stockquote.aggregator")
    print("Available stock symbols:", aggregator.available_symbols())
    symbols = input("Enter symbols you want to view (comma separated):")
    symbols = [symbol.strip() for symbol in symbols.split(",")]
    aggregator.view(viewer, symbols)
    print("Viewer listening on symbols", symbols)
    daemon.requestLoop()

if __name__ == "__main__":
    main()

running the final program

To run the final stock quote system you need to do the following:

  • Open a new console window to start the name server in.

  • Set the following environment variable:

    set PYRO_SERIALIZERS_ACCEPTED=pickle      (windows)
    export PYRO_SERIALIZERS_ACCEPTED=pickle   (unix/mac/linux)
    

    Once more, ignore the exact meaning of this particular setting. It is needed to get the stock market tutorial running in the form presented here. In other chapters the meaning of this setting will be explained.

  • start the Pyro name server (python -m Pyro4.naming) in this window.

After that, start the following, each in a separate console window (so they run in parallel):

  • start the stock market
  • start the aggregator
  • start one or more of the viewers.

The output of the stock market looks like this:

$ python stockmarket.py
Stockmarkets running.
new quotes generated for NASDAQ
new quotes generated for NASDAQ
new quotes generated for NYSE
new quotes generated for NASDAQ
...

The output of the aggregator looks like this:

$ python aggregator.py
joining market example.stockmarket.newyork
joining market example.stockmarket.nasdaq
Aggregator running. Symbols: ['HPQ', 'BP', 'IBM', 'GOOG', 'AAPL', 'CSCO', 'MSFT']
aggregator gets a new viewer, for symbols: ['GOOG', 'CSCO', 'BP']

The output of the viewer looks like this:

$ python viewer.py
Available stock symbols: ['HPQ', 'BP', 'IBM', 'GOOG', 'AAPL', 'CSCO', 'MSFT']
Enter symbols you want to view (comma separated):GOOG,CSCO,BP    # <---- typed in
Viewer listening on symbols ['GOOG', 'CSCO', 'BP']
NYSE.BP: 88.96
NASDAQ.GOOG: -9.61
NYSE.BP: 113.8
NASDAQ.CSCO: 125.11
NYSE.BP: 77.43
NASDAQ.GOOG: 17.64
NASDAQ.CSCO: 157.21
NASDAQ.GOOG: 7.59
...

If you’re interested to see what the name server now contains, type python -m Pyro4.nsc list:

$ python -m Pyro4.nsc list
--------START LIST
Pyro.NameServer --> PYRO:Pyro.NameServer@localhost:9090
example.stockmarket.nasdaq --> PYRO:obj_fc742f1656bd4c7e80bee17c33787147@localhost:50510
example.stockmarket.newyork --> PYRO:obj_6bd09853979f4d13a73263e51a9c266b@localhost:50510
example.stockquote.aggregator --> PYRO:obj_2c7a4f5341b1464c8cc6091f3997230f@localhost:50512
--------END LIST

Running it on different machines

For security reasons, Pyro runs stuff on localhost by default. If you want to access things from different machines, you’ll have to tell Pyro to do that explicitly. This paragraph shows you how very briefly you can do this. For more details, refer to the chapters in this manual about the relevant Pyro components.

Name server
to start the nameserver in such a way that it is accessible from other machines, start it with an appropriate -n argument, like this: python -m Pyro4.naming -n your_hostname_here
Warehouse server

You’ll have to modify warehouse.py. Right before the serveSimple call you have to tell it to bind the daemon on your hostname instead of localhost. One way to do this is by setting the HOST config item:

Pyro4.config.HOST = "your_hostname_here"
Pyro4.Daemon.serveSimple(...)

You can also choose to leave the code alone, and instead set the PYRO_HOST environment variable before starting the warehouse server. Another option is pass the required host (and perhaps even port) arguments to serveSimple:

Pyro4.Daemon.serveSimple(
        {
            warehouse: "example.warehouse"
        },
        host = 'your_hostname_here',
        ns = True)
Stock market servers
This example already creates a daemon object instead of using the serveSimple call. You’ll have to modify the three source files because they all create a daemon. But you’ll only have to add the proper host argument to the construction of the Daemon, to set it to your machine name instead of the default of localhost. Ofcourse you could also change the HOST config item (either in the code itself, or by setting the PYRO_HOST environment variable before launching.

Other means of creating connections

In both tutorials above we used the Name Server for easy object lookup. The use of the name server is optional, see Name Server for details. There are various other options for connecting your client code to your Pyro objects, have a look at the client code details: Object discovery and the server code details: Pyro Daemon: publishing Pyro objects.

Ok, what’s next?

Congratulations! You completed the Pyro tutorials in which you built a simple warehouse storage system, and a stock market simulation system consisting of various independent components that talk to each other using Pyro. The Pyro distribution archive contains a truckload of example programs with short descriptions that you could study to see how to use the various features that Pyro has to offer. Or just browse the manual for more detailed information. Please consider joining the Pyro mailing list (see front page). Happy remote object programming!