Remote Access
The service agent and message exchange framework operate based on the concept of processes which are a part of a distributed application being permanently connected together. This model works fine on corporate networks, but is not always practical when run across the Internet. One drawback of this approach is that it is often necessary to open up special ports on a corporate firewall to permit access.
For many instances where communication across the Internet is required, a connected model of operation isn't actually required. Instead, many types of operations can be carried out using a request/reply model whereby a connection is only maintained for the lifetime of the request. This is precisely the type of mechanism which is used by HTTP.
Because of the wide acceptance for HTTP a number of remote procedure call protocols have been developed which operate within the bounds of a HTTP request. The most well known of these are XML-RPC and SOAP. Unfortunately, both of these protocols are actually lacking in certain respects and have not been found to be a totally satisfactory medium.
In place of these protocols, an alternative RPC over HTTP protocol is provided called NET-RPC. At present, the only client available is implemented using Python. If you are writing a closed system this shouldn't present a problem. In those cases where public access may be required, gateways for XML-RPC and JSON-RPC are are available, but using them will place a limitation on the type of data which you can pass around.
Note that whichever RPC over HTTP protocol you do decide to use, the code for your services is the same. In fact, your application may include gateways for all three protocols and a user can use whichever type of client they find easiest. In this respect, it doesn't matter too much which protocol wins out. Even if a new protocol comes along, it is a relatively simple matter to incorporate yet another gateway, again without you having to make modifications to the core of your system.
Note that with OSE 8.2, the SOAP gateway has been dropped. This is because the SOAP gateway was always a poor solution due to the limitations of the SOAP encoding for use in implementing a basic RPC mechanism. Interoperability problems were also always an issue when using SOAP.
The RPC Gateway
The gateway which accepts an RPC request is actually an instance of a HTTP server object. The gateway will accept a request and based on the URL determine which service the request applies to. The request will then be translated into a call over the service agent framework, with the corresponding result being packaged up and returned to the remote client.
Because the service agent framework can operate in a distributed manner using the message exchange framework, the service which a request applies to need not even be in the same process as the RPC gateway. So that a remote client can't access any arbitrary service however, a mechanism is provided to limit which services are actually visible. The mechanisms for client and user authorisation implemented by the HTTP servlet framework can also be used to block access as appropriate.
The RPC gateway is implemented by the RpcGateway class. When created, the gateway needs to be supplied the name of a service group. Only those services which are a member of that service group will be accessible through that particular instance of the RPC gateway. Having created an instance of the RPC gateway, it needs to be mapped into the URL namespace of a HTTP daemon object.
1 import netsvc
2 import signal
3
4 class Validator(netsvc.Service):
5 def __init__(self, name="validator"):
6 netsvc.Service.__init__(self,name)
7 self.joinGroup("web-services")
8 self.exportMethod(self.echo)
9 def echo(self, *args)
10 return args
11
12 dispatcher = netsvc.Dispatcher()
13 dispatcher.monitor(signal.SIGINT)
14
15 validator = Validator()
16
17 port = 8000
18 group = "web-services"
19 httpd = netsvc.HttpDaemon(port)
20 rpcgw = netsvc.RpcGateway(group)
21 httpd.attach("/service",rpcgw)
22 httpd.start()
23
24 dispatcher.run()
In this example, any HTTP request made using a URL whose path falls under the base URL of http://localhost:8000/service/, will be regarded as being a valid request. The name of the service which a request applies to is determined by removing the base URL component from the full URL. The full URL used to access the service in this example would therefore be http://localhost:8000/service/validator. Note that the service is only visible however, because it had added itself to the group "web-services", the same group as the RPC gateway had been initialised with.
The methods of the service which are available are the same as those which would be accessible over the service agent framework internal to your application. That is, a service must export a method for it to be accessible. The only such method available in this example would be echo().
Note that prior to OSE 8.2, the netsvc.RpcGateway class only supported the NET-RPC protocol and to provide access for the XML-RPC protocol the netsvc.xmlrpc.RpcGateway class had to instead be used. This might have been used by itself, or two distinct gateways, one for each protocol may have been mapped under different URLs. From OSE 8.2, the netsvc.RpcGateway will accept requests for the NET-RPC, XML-RPC or JSON-RPC protocols, automatically determining which was used and mapping the request as necessary.
The Client Application
Client side access to the NET-RPC protocol is available through the Python netrpc module. This module is not dependent on the netsvc module and is pure Python. The name of the class used to make a request to a remote service is RemoteService. This class behaves in a similar fashion to the LocalService class from the netsvc module except that the service name is replaced with the URL identifying the remote service.
1 import netrpc
2
3 url = "http://localhost:8000/service/validator"
4 service = netrpc.RemoteService(url)
5 print service.echo(1, 1L, 1.1, "1")
Only the "http" protocol is supported. If the URL specifies an unsupported protocol, the exception AddressInvalid will be raised. If the URL didn't identify a valid service on the remote host, a ServiceUnavailable exception is raised. Other possible exceptions which may be raised are AuthenticationFailure and TransportFailure. All the more specific exceptions actually derive from ServiceFailure and the ServiceFailure exception is also used for errors generated by the service itself, so it is often sufficient to watch out for just that type of exception.
Restricting Client Access
In addition to being able to dictate precisely which services are visible, it is also possible to restrict access to specific clients. This can be done by allowing only certain hosts access, or by limiting access to specific individuals by using user authentication. Both schemes rely on features within the existing HTTP servlet framework.
1 class HttpDaemon(netsvc.HttpDaemon):
2 def __init__(self, port, hosts=["127.0.0.1"]):
3 netsvc.HttpDaemon.__init__(self, port)
4 self._allow = hosts
5 def authorise(self, host):
6 return host in self._allow:
7
8 class RpcGateway(netsvc.RpcGateway):
9 def __init__(self, group, users=None):
10 netsvc.RpcGateway.__init__(self, group):
11 self._allow = users
12 def authorise(self, login, password):
13 return self._allow == None or \
(self._allow.has_key(login) and \
self._allow[login] == password)
14
15 users = { "admin": "secret" }
16
17 port = 8000
18 group = "web-services"
19 httpd = HttpDaemon(port)
20 rpcgw = RpcGateway(group, users)
21 httpd.attach("/service", rpcgw)
22 httpd.start()
When user authentication is being used, the login and password of the user can be supplied as additional arguments to the RemoteService class when it is created.
1 url = "http://localhost:8000/service/validator"
2 service = netrpc.RemoteService(url, "admin", "secret")
3 print service.echo(1, 1L, 1.1, "1")
If a login and password aren't supplied when required, or the details are wrong, the AuthenticationFailure exception will be raised.
Duplicate Services
Because a URL identifies a unique resource, a conflict arises due to the fact that within the service agent framework it is possible to create multiple services with the same name. What happens in this circumstance is that the RPC gateway will remember which service agent was the first it saw in the required service group, having a particular service name. While that particular service agent exists, it will always use that service agent as the target of requests.
When there are multiple service agents with the same service name and the first one seen by the RPC gateway is destroyed, the RPC gateway will then fall back to using the second one it saw. That is, the RPC gateway will always use the service agent which it has known about the longest. In general, if you intend to make services accessible using the RPC gateway, it is recommended that you always use unique service names within the service group dictating which services are actually visible.
User Defined Types
The NET-RPC protocol supports all the types supported by the service agent framework, as well as the concept of user defined scalar types. That is, if a service responds with data incorporating additional scalar types, they will by default be passed back as instances of the Opaque type, where the "type" attribute gives the name of the type and the "data" attribute the encoded value. Similarly, new types may be sent by initialising an instance of the Opaque type with the name of the type and the value in its encoded form.
1 url = "http://localhost:8000/service/validator"
2 service = netrpc.RemoteService(url, "admin", "secret")
3
4 # following are equivalent
5 value = complex(1, 1)
6 print service.echo(value)
7 print service.echo(netrpc.Opaque("python:complex", repr(value)))
Encoders and decoders for additional user defined scalar types can be provided by registering the appropriate functions using the encoder() and decoder() functions available in the netrpc module. The functions for registering the encoder and decoder functions are used in exactly the same was as those in the netsvc module. In fact, they are the same functions as the netsvc module imports them from the netrpc module, as it does for the implementations of all of the extended types.
As is the case in the service agent framework, you need to be mindful about the effect of registering arbitrary encoders and decoders at global scope, especially if your client application makes calls against different services implementing their own scalar types. This becomes even more of an issue if the netrpc module is used to make client side calls from inside a server side application using the netsvc module. This is because they will share the same global encoders and decoders.
If you need to support types which are specific to a service being called, rather than registering the encoder and decoder function at global scope, the safer way is to supply your own functions just for that service. This is done by supplying the function using a keyword argument when initialising the instance of the RemoteService class. The keyword argument for the encoder function is "encode" and that for the decoder function is "decode". The functions you supply should call the corresponding global function if it doesn't know what to do with a specific type.
1 def encodeObject(object):
2 if type(object) == MySQLdb.DateTimeType:
3 return ("xsd:string", object.strftime())
4 elif type(object) == MySQLdb.DateTimeDeltaType:
5 return ("xsd:string", str(object))
6 return netsvc.encodeObject(object)
7
8 url = "http://localhost:8000/service/validator"
9 service = netrpc.RemoteService(url, encode=encodeObject)
Managing User Sessions
A common practice with web based services is to have a request initiate a unique session for a user. Having opened the session, any requests will then be identified with that session, with information regarding the session potentially being cached on the server side until the session is closed. Such a session might also be used as a way of allocating a server side resource to that user, or creating a database cursor dedicated to a particular user so more complex queries can be made.
A scheme suitable for use over the service agent framework was previously described, however that implementation was based on the ability to subscribe to the existence of the owner of the session, with the session being automatically closed when the owner was destroyed. When the RPC gateway is used, this approach can't be used, as the sender of the request will be a transient service created by the RPC gateway to service just that request. An alternative when the RPC gateway is being used is to automatically close the session after a set period of inactivity.
1 class Database(netsvc.Service):
2 def __init__(self, name="database", **kw):
3 netsvc.Service.__init__(self, name)
4 self._name = name
5 self.joinGroup("database-services")
6 self._database = MySQLdb.connect(**kw)
7 self._cursors = 0
8 self.exportMethod(self.cursor)
9 def executeMethod(self, name, method, params):
10 try:
11 return netsvc.Service.executeMethod(self, name, method, params)
12 except MySQLdb.ProgrammingError,exception:
13 self.abortResponse(1, "Programming Error", "db", str(exception))
14 except MySQLdb.Error(error, description):
15 self.abortResponse(error, description, "mysql")
16 def cursor(self, timeout=60):
17 self._cursors = self._cursors + 1
18 name = "%s/%d" % (self._name, self._cursors)
19 cursor = self._database.cursor()
20 Cursor(name, cursor, timeout)
21 child = "%d" % self._cursors
22 return child
The idea is that when a request is made, a unique instance of a service is created specific to the session, with a name which is then passed back to the remote client. In the example shown, if the service was originally accessible using the URL http://localhost/database, the instance of a service created for that specific session would be the same URL but with the session id appended, separated by "/". Eg., http://localhost/database/1. Obviously, a session id which could not be easily guessed should however be used.
The client would now direct all future requests to the new URL. When the client has finished with the service it would call the close() method on the service. If for some reason the client did not explicitly close off the session, it would be automatically closed after a period of 60 seconds of inactivity, or whatever period was defined when the session was initiated. An implementation of the database cursor service for this example might be as follows.
1 class Cursor(netsvc.Service):
2 def __init__(self, name, cursor, timeout):
3 netsvc.Service.__init__(self, name)
4 self.joinGroup("database-services")
5 self._cursor = cursor
6 self._timeout = timeout
7 self._restart()
8 self.exportMethod(self.execute)
9 self.exportMethod(self.executemany)
10 self.exportMethod(self.description)
11 self.exportMethod(self.rowcount)
12 self.exportMethod(self.fetchone)
13 self.exportMethod(self.fetchmany)
14 self.exportMethod(self.fetchall)
15 self.exportMethod(self.arraysize)
16 self.exportMethod(self.close)
17 def encodeObject(self, object):
18 if hasattr(MySQLdb, "DateTime"):
19 if type(object) == MySQLdb.DateTimeType:
20 return ("xsd:string", object.strftime())
21 elif type(object) == MySQLdb.DateTimeDeltaType:
22 return ("xsd:string", str(object))
23 return netsvc.Service.encodeObject(self, object)
24 def executeMethod(self, name, method, params):
25 try:
26 return netsvc.Service.executeMethod(self, name, method, params)
27 except MySQLdb.ProgrammingError,exception:
28 self.abortResponse(1, "Programming Error", "db", str(exception))
29 except MySQLdb.Error(error, description):
30 self.abortResponse(error, description, "mysql")
31 def _restart(self):
32 self.cancelTimer("idle")
33 self.startTimer(self._expire, self._timeout, "idle")
34 def _expire(self, name):
35 if name == "idle":
36 self.close()
37 def execute(self, query, args=None):
38 result = self._cursor.execute(query, args)
39 self._restart()
40 return result
41
42 # additional methods
43
44 def close(self):
45 self._cursor.close()
46 self.cancelTimer("idle")
47 self.destroyAgent()
48 return 0
Using the netrpc module to access the service, a client might be coded as follows. In this case a separate cursor is created in relation to the queries made about each table in the database.
1 import netrpc
2
3 url = "http://localhost:8000/database"
4 service = netrpc.RemoteService(url)
5
6 tables = service.execute("show tables")
7
8 timout = 30
9 for entry in tables:
10 table = entry[0]
11 print "table: " + table
12 name = service.cursor(30)
13 print "cursor: " + url + "/" + name
14 cursor = netrpc.RemoteService(url + "/" + name)
15 cursor.execute("select * from " + table)
16 desc = cursor.description()
17 print "desc: " + str(desc)
18 data = cursor.fetchall()
19 print "data: " + str(data)
20 cursor.close()
In general, giving open access to a database in this way may not be advisable, especially over the Internet. Such a mechanism might be restricted to a corporate intranet. Alternatively, custom interfaces should be layered on top of the database providing interfaces based on functional requirements.
The XML-RPC Gateway
If the Python NET-RPC client implementation can't be used because of the need to use a different language for the client, you might instead consider using the XML-RPC protocol. Clients for the XML-RPC protocol are available in many different languages, many of which are listed at http://www.xmlrpc.com. If wishing to support only the XML-RPC protocol, your will need to change your server application to instantiate an instance of the XML-RPC gateway instead of the NET-RPC gateway.
1 import netsvc
2 import netsvc.xmlrpc
3
4 dispatcher = netsvc.Dispatcher()
5 dispatcher.monitor(signal.SIGINT)
6
7 validator = Validator()
8
9 port = 8000
10 group = "web-services"
11 httpd = netsvc.HttpDaemon(port)
12 rpcgw = netsvc.xmlrpc.RpcGateway(group)
13 httpd.attach("/service", rpcgw)
14 httpd.start()
15
16 dispatcher.run()
Note that from OSE 8.2 onwards, the netsvc.RpcGateway class supports the XML-RPC protocol as well as the NET-RPC protocol. Thus you only need to use netsvc.xmlrpc.RpcGateway if you wish to ensure that only the XML-RPC protocol is supported.
If you do decide to rely upon the XML-RPC protocol instead of the NET-RPC protocol, you will be constrained as to what types you can use. This is because the XML-RPC protocol has a more limited set of core types and is not type extendable as is the NET-RPC protocol. One major deficiency of the XML-RPC protocol, is that it has no way of passing a null value, such as that implemented by the Python None type. Some XML-RPC clients have been extended to support a null value, but this gateway does not implement such an extension and adheres to the XML-RPC specification as originally created.
Because of the limitations of the XML-RPC protocol in respect of passing a more diverse set of types, when these implementations are used, types which don't have a direct equivalent in XML-RPC will have their encoded value passed as a string, with a subsequent loss of type information. For example, the Python None type will be sent as an empty string. Note this only applies to the result of a request when it is being returned via an XML-RPC request. Since XML-RPC doesn't support the extra types, a client strictly conforming to the XML-RPC protocol would not have been able to generate them in the first place.
A further complication which can arise in using XML-RPC is that the specification isn't precise in certain areas. Although an XML-RPC message is notionally XML, the specification indicates use of ASCII values in strings only. This is in conflict with XML which requires at least UTF-8. Another issue is that the XML-RPC specification mentions nothing about needing to support XML comments, CDATA or various other XML constructs. This has lead to some implementations of the protocol not supporting such features of XML and others relying on them. This gateway in particular does not support the use of XML comments or CDATA within the body of an XML-RPC request.
Although there are numerous third party XML-RPC clients available, including a number for Python, an XML-RPC client is also provided with OSE. This client is interface compatible with that provided by the netrpc module and is available in the netrpc.xmlrpc module. This provides exactly the same interface as the netrpc module, even to the extent of being able to reconstruct the more informative failure responses provided by the service agent framework.
1 import netrpc.xmlrpc
2
3 url = "http://localhost:8000/service/validator"
4 service = netrpc.xmlrpc.RemoteService(url)
5 print service.echo(1, 1L, 1.1, "1")
What happens when a failure occurs is that the additional information provided by the service agent framework is encoded into the description field of an XML-RPC fault. When this is received by the xmlrpc module it extracts out the information into separate fields once more. If you are using a third party XML-RPC client this will not occur. What you will find instead is that the fault code will equate to the error code of a failure, with the description included with the fault looking something like the following.
origin -- the description additional fault details
That is, the description is prefixed by the origin of the failure, separated by "--". The additional details of the failure will then appear separated from the description by a blank line. You could either use this as is, or separate out the information yourself.
Note that when using the xmlrpc module the encoders and decoders become largely irrelevant given that the XML-RPC protocol is not type extendable. Although the xmlrpc module provides an interface compatible with the netrpc module, it still may be used to make requests against third party XML-RPC servers.
Using Multiple Gateways
Because it is possible to attach multiple HTTP server objects to a particular instance of a HTTP daemon object, you aren't restricted to having only one instance of an RPC gateway. Such RPC gateways can and would generally be associated with different groups of services. It is possible that some of the RPC gateways might be protected using user authentication. At the same time, a HTTP file server object or custom HTTP server object might also be attached to the same HTTP daemon.
1 files = netsvc.FileServer(os.getcwd())
2 httpd.attach("/download", files)
3
4 user = netsvc.RpcGateway("web-services")
5 httpd.attach("/service", user)
6
7 admin = netsvc.RpcGateway("admin-services")
8 httpd.attach("/admin", admin)