Message Encoding

As the service agent framework is designed as a distributed system covering multiple programming languages, it is necessary that any data being passed around within a report, request or response be serialised into a form suitable for transmission as part of a message. At present the encoded form of the data uses a subset of XML. That is, it would qualify as being XML, however to make the implementation easier, the code for decoding such messages will not accept arbitrary XML.

At present the exact form of the XML being used is not revealed as this is being reviewed and will most likely change. Further, the protocol used between message exchange endpoints is unique to this software. It too is being reviewed and will most likely be changed to use some more commonly accept form of handling message boundaries. Any new mechanism will likely also be designed to be able to proxy through HTTP servers, thus avoiding issues with closed firewalls.

That the precise details are not being revealed actually makes no difference as it has no bearing in relation to using the software. This is because everything is hidden under a high level API which hides such details, thus allowing for change in the formats used without requiring changes to applications using the software. The only instance where changes might have a visible affect is in respect to the NET-RPC protocol for RPC over HTTP. This would only be an issue if you tried to write your own client for this protocol.

The one aspect of how data is encoded which will not change is in relation to the means of identifying different types. Here the XML Schema Datatypes 2001 specification is used as a guide, with Python types being assigned corresponding types with respect to this specification. Through introduction of customised encoders and decoders, support for user defined scalar data types may however also be added.

Supported Data Types

Communication between services is mediated through a layer of code which is written in C++. The only exception to this is when the LocalService class is used as a proxy to send a request to a service in the same process which is also implemented in Python. This means that except for when the LocalService class is being used, any data which is being transferred between services must go through a process of being encoded into a serialised form at the point of sending and then deserialised at the point of reception.

Data which is being sent between services is not limited to that of just a string. The data to be sent can consist of any of the basic Python scalar types, a list, a tuple or a dictionary. In addition to this, the Python "None" value may be used, as well as a number of extended types. The only limitation in respect of the Python compound types is that when using a dictionary, the keys must be of type string. Further, when a tuple appears within any data, the recipient will see it as a list and not a tuple. It is not possible to send data which is cyclically self referential.

   1 self.publishReport("string", "value")
   2 self.publishReport("list", [1.1L, 1.1, None])
   3 self.publishReport("dictionary", {"key" : "value"})

The extended types which are supported are Boolean, Binary, Date, DateTime, Time and Duration. For the Boolean type, there are also predefined values for True and False. The Boolean type is an alias for the Python builtin bool type where Python supports it. If the default arguments for the constructor of Date and DateTime types are used, they will be initialised to the current local date and current local date and time respectively.

   1 self.publishReport("true", netsvc.True)
   2 self.publishReport("false", netsvc.False)
   3 self.publishReport("boolean", netsvc.Boolean(1))
   4 
   5 self.publishReport("binary", netsvc.Binary("value"))
   6 
   7 # current local date
   8 self.publishReport("date", netsvc.Date())
   9 
  10 # current local date/time
  11 self.publishReport("dateTime", netsvc.DateTime())

When using the various date and time types, they should be initialised with string values corresponding to what type they represent. The format and range of these values should be the subset of values possible under the ISO 8601 date/time standard as described by the XML Schema Datatypes 2001 specification, examples of which are illustrated below.

For the date and time types, the current Python implementation does not do any checking to determine if the supplied values are valid, but will pass them as is. Note that the XML Schema Datatypes specification does allow for a timezone in a date and time, but it is recommended that all date and time values be sent as UTC. In the C++ library, only classes corresponding to Date and DateTime exist. These are OTC_Date and OTC_Time. The OTC_Time class is not able to handle timezones.

The only difference between the Binary type and using a string is that the value supplied via the Binary type, will be encoded internally using "base64" encoding when being passed around. This has relevance because in XML most control characters are not permitted in string values. An XML implementation can also collapse a "\r\n" combination to just "\n". If such characters may appear in a string, you should use the "Binary" type to ensure that they are preserved as is. Note that you do not however have to encode the string using base64 encoding first as the internal implementation will do this for you automatically.

Mapping of Scalar Types

When data is being serialised, the names attributed to scalar types derive from the XML Schema Datatypes 2001 specification. The only exception to this is the None type, which notionally is passed around internally with an empty type value. The mapping from Python types to those described in the XML Schema Datatypes specification is as follows.

If a service is implemented using the OSE C++ class library directly, different size versions of the integer and floating point types are available and can be generated in the serialised form of any data. A consequence of this is that when converting any data from its serialised form into instances of Python types, a broader range of possible values types need to be accommodated.

Note that at the present time, not all of the XML data types in respect of non positive and non negative integers are accommodated. These will most likely be added at some time in the future, however in the short term they don't add anything extra in relation to the Python interface. Support for the type "xsd:hexBinary" will also be added at some point in the future as well. If you wish to send a "Unicode" string, you should convert it into a string using UTF-8 encoding.

User Defined Types

The intent with the XML Schema Datatypes specification is that additional scalar data types can be introduced by assigning a new name scoped within a distinct namespace. In respect of the types defined by this specification, the namespace "xsd" is used. Note that within this implementation, the namespace is not linked to a URI containing any form of definition for that type. If sending a data value of your own type, it is up to your code to ensure that both ends know what the type means.

The simplest way of adding your own types is by using the Opaque class. When initialised this takes two values, a string identifying the type of value and a string representing the value in its encoded form. It is not necessary to escape any characters in the encoded value which may be special to XML as such values will be automatically escaped as necessary.

   1 data = complex(1,1)
   2 type = "python:complex"
   3 self.publishReport("complex", netsvc.Opaque(type, data))

In reality, it isn't actually necessary to encode a Python complex value in the way shown as a special mapping is by default installed for this type. For this Python type the namespace "python" is used. If defining your own type it is recommended you use some other namespace value which is in some way specifically associated with your application or some third party standard relating to additional XML types.

As a special mapping is provided for the Python complex type, it will be decoded into an instance of the Python complex type on reception. If however a mapping is not available for a specified type, the value will be converted back into an instance of the Opaque type. The type associated with the value can then be queried using the "type" attribute and the actual encoded data using the "data" attribute.

   1 def dump(self, object):
   2     if isinstance(object, netsvc.Opaque):
   3         print object.type, object.data

The Opaque class provides a means of sending a value without a defined mapping, or of you being able to receive values for which no mapping is defined. If necessary the interface of the Opaque class can be used to dynamically handle such unknown values and perhaps still make some sense of them.

Adding New Mappings

Mappings for new types can be added at two levels. These are at global scope or such that they only apply within the scope of a single service. If a type mapping is added at global scope, you should realise that such a mapping will be applied to any service. Adding new mappings with global scope should therefore be carefully considered as it may inadvertently affect the operation of another service.

To add a new mapping at global scope the functions encoder() and decoder() should be used to register functions to do the appropriate conversions. When registering the encoder, the first argument should be either the type object or class object as appropriate. When registering the decoder, the first argument should be the qualified name you have given the type.

The encoder function which you register should accept a single argument, that being an instance of your type. The function should return a tuple containing the qualified name you have given the type and the value encoded as a string. The decoder function should accept two arguments, they being the qualified name you have given the type and the value encoded as a string. The function should return the corresponding instance of the type as described by the encoded value. If the encoded value is invalid, the function should raise an appropriate exception.

   1 def _encode_Complex(object):
   2     return ("python:complex", repr(object))
   3 
   4 def _decode_Complex(name, string):
   5     return complex(string)
   6 
   7 netsvc.encoder(types.ComplexType, _encode_Complex)
   8 netsvc.decoder("python:complex", _decode_Complex)

To define a mapping which applies only within the context of a single service, you need to override the member functions encodeObject() and decodeValue() as appropriate. Note that the default implementations of these methods will apply any global mappings which are present. If your version of these functions, don't identify the type you are interested in, your function should call the base class version of the function. The arguments to these functions are similar to the global encoders and decoders.

   1 class Database(netsvc.Service):
   2     def __init__(self, name, **kw):
   3         netsvc.Service.__init__(self, name)
   4         # ...
   5     def encodeObject(self, object):
   6         if hasattr(MySQLdb,"DateTime"):
   7             if type(object) == MySQLdb.DateTimeType:
   8                 return ("xsd:string", object.strftime())
   9             elif type(object) == MySQLdb.DateTimeDeltaType:
  10                 return ("xsd:string", str(object))
  11         return netsvc.Service.encodeObject(self, object)

Providing a mapping which is specific to a service is most often used when the service interacts with a Python module which defines its own types for such values as date and time. In this circumstance, the mapping function can automatically translate an instance of the type into a type appropriate for the encoded data. This avoids your own code having to manually translate values into corresponding values of the correct type before hand. A service may also override the default decoders for extended types such as the date and time types if desired.

Handling Structured Types

The encoding mechanism for data does not provide a way of adding support for your own structured types, whereby the type of that object can also be transmitted. All objects need to be able to be converted into instances of scalar types, dictionaries, tuples or lists. To avoid having to do this conversion manually, it is however possible to define an encoder for a structured type which will do this for you.

At the global level, such a function is again registered using the encoder() function. The difference between this function and that for scalar types however, is that instead of returning a string giving the name of the scalar type, the value "None" should be returned in its place. The second value in the tuple should then be the instance of the structured type translated into either a scalar type, dictionary, tuple or list.

   1 def _encode_UserList(object):
   2     return (None, list(object))
   3 
   4 netsvc.encoder(UserList.UserList, _encode_UserList)

Having returned the translated value, it will be represented to the encoder. Thus it is only necessary to translate the top level of the data structure as enclosed values will in turn be translated automatically if required and if an encoder is registered. This mechanism may also be used in an encoder specific to a service.

CategoryOSE