4.4 step5

Step 5 - Function fields

We now have a model, data and the possibility to use them. Once again, you should update your environment on the step5 branch to make sure that you are up to date:

git checkout 4.4/step5

Introduction

This step fill be fully dedicated to Function fields. What makes them so important ?

What you need to understand is that tryton uses fields for everything:

• You cannot display anything but fields to the end user
• You need fields to add constraints (more on this later)
• You also need fields to fine-tune the interface

So let's focus on the first point, which is easier to understand. We told earlier that you should as much as possible avoid to duplicate information. We even said that, for instance, storing the number of books of an author should not be done with a field, because it could be calculated.

However, you cannot display an information that is not a field. So showing the user how many books an author wrote will require a field anyway, even though we know we can calculate it from existing data.

Enter Function fields. As you may have guessed by now, Function fields are calculated fields. They allow to:

• Avoid data duplication
• Display any kind of information. It can be as simple as a "shortcut" (for instance if we want to display the author's birth date directly on a book), to complex calculated informations (how many books of the same genre the author had already written when the book was released), or even external informations fetched from another system or the internet (we could imagine getting a global popularity ranking for our book)
• Be tryton fields, which allows for a lot of things as we will detail later

Creating a Function field

A simple field

Open the library.py file and add the following after the gender field of the library.author model:

age = fields.Function(
    fields.Integer('Age'),
    'getter_age')

This is the most basic Function field that you will write, and encounter. Function fields are declared the same way as other fields, the difference is in the parameters.

The first parameters is another field instance. This instance will contains all the standard information of your field. Its type obviously (here, it will be an Integer), all the parameters we already saw (help, required, etc.) and those we did not (yet). The easy way (at first) to write your function field is to write it the same way you would if it was a real field, and then add it as the fields.Function first parameter.

The second parameter describes how the field's value will be computed. The expected value of this parameter is the name of a method on the model. This method will be called to compute the field's value for the required records.

Note: The logic of using a string as the name of a method that must be called under certain circumstances is rather frequent in tryton, this is just the first time we are seeing it

There are multiple ways to write the getter_age method, we are going to go with the easy one this time. First, at the top of your file, import the python datetime module:

import datetime

Now, in the body of the Author class, under the field declarations, add:

def getter_age(self, name):
    if not self.birth_date:
        return None
    end_date = self.death_date or datetime.date.today()
    age = end_date.year - self.birth_date.year
    if (end_date.month, end_date.day) < (
            self.birth_date.month, self.birth_date.day):
        age -= 1
    return age

We will not talk about the algorithm, which is not the purpose of this training module. However, we will explain how this will calculate the age field value.

The getter_age method's name matches the second parameter of the age field definition. So when tryton will need to read the age of a library.author record, he will look for this method. It is an instance method, so the first parameter (self) will contain the record that we want to calculate the age for.

Warning: it is important to understand that a "record" of the library.author model is not the same as an instance of the Author class. The "record" is a python object created by the python server from a class which inherit from the Author class, but which may contain many other things as well

The second parameter contains the name of the Function field for which the method was called. It is used when the same getter method is used for different but similar Function fields which are calculated the same way. In that case the name parameter allows to slightly tweak the algorithm accordingly.

The expected return value of the method is the value we want to read in the age field. The returned type depends on the type of the first parameter of the Function field:

• For basic data fields (Char, Text, Integer, etc...), the associated python type
• For Numeric, instances of the Decimal class
• For Many2One fields, the id of the record we want to reference
• For Many2Many, a list of ids
• For One2Many, a list of dictionaries, with the key / values of the target records we need, or a list of ids

Warning: The id of a record is not the same thing as the id attribute we already saw in XML files. A record's id is a technical integer field that is managed by tryton (set once and for all when creating the record), and which uses as a foreign key in the database to manage relations between models

Additionally, all Function fields getters may return None as a value for the field.

Regarding the actual code of the getter, you can see that we can access the values of the birth_date or death_date fields for the current record by using self.birth_date or self.death_date. This is possible in all the server code, and tryton will do what it takes to get the information. In the birth_date case, for instance, it will:

• Check that the current used is allowed to read the field
• If the field value is already in memory, return it
• If not, it will load it from the database (if birth_date was a Function field, this is where the getter would be called)
• Then store it in a cache, and return it

Now that we got our field, we can display it like any other field. Add it in the form view of the library.author model, after the death_date field. Reload the server, the view, and the age field should appear.

If you do not already have, create a new author, and set it a birth_date and death_date. You will notice that the age stays empty until you hit the "Save" button. Also, if you modify any of the dates, the age is not recomputed until you save the record. This is because Function fields are only calculated when records are read.

Reading a record happens when:

• You try to access it "server-side" by writing my_record.age
• The record is displayed client-side
• The record is saved client-side (saving always triggers a read right after)
• You manually call the read method on your record

There are means to make the age be refreshed in real time, which we will talk about later.

Performances

Add another field on the library.author model:

number_of_books = fields.Function(
    fields.Integer('Number of books'),
    'getter_number_of_books')

Finally, the information we wanted to have but could not because it would be a "duplicate"... but why under a Performances title? The reason is that the number of books information requires to either iterate over the list of books of the author, or directly query the database.

Enters tryton performance optimisations. When the client displays 100 records to the user (for instance, 100 authors) in a list, it will make one read call to the server for the 100 records at once! This optimize the bandwidth usage between the server and the client. On the server side, if the server is asked to read 100 records at once, it will try to make only one database call for the same reasons (and also, the overhead of a database query is very big compared with "simple" read queries).

All this leads to: with the getter definition we saw above, reading 100 records would trigger 100 calls to it. In the case of the number_of_books field, the basic implementation would be very inefficient, because we would have to do another 100 database calls to get the value of the field. Fortunately, tryton provides a way to do this in an elegant fashion. Add those imports at the top of your file:

from sql.aggregate import Count

from trytond.pool import Pool
from trytond.transaction import Transaction

and then the getter implementation (under the existing getter_age method):

@classmethod
def getter_number_of_books(cls, authors, name):
    result = {x.id: 0 for x in authors}
    Book = Pool().get('library.book')
    book = Book.__table__()

    cursor = Transaction().connection.cursor()
    cursor.execute(*book.select(book.author, Count(book.id),
            where=book.author.in_([x.id for x in authors]),
            group_by=[book.author]))
    for author_id, count in cursor.fetchall():
        result[author_id] = count
    return result

Let's detail all of this.

The obvious difference between getter_age and getter_number_of_books is that they do not even have the same prototype. The latter is a classmethod, and it has cls and authors as parameters in place of the self of the former. How do we go from one to the other ?

Well, remember the scenario with 100 records being read at once ? In this case, the getter_number_of_books would be called once, with the 100 records passed as the second argument, authors. The expected result in this case is a dictionary, with the record's id as keys, and the result (number of books for the author) as values.

Note: Since python parameters are not typed, it is important that parameter names are properly named. Here, authors is author from library.author and s, for a list of authors

Now the implementation. We could, as we talked about earlier, iterate on all authors in the list, then query the database for its number of book. However, in that case we could as well use the instance method kind of getter we used for the age field, there will still be 100 requests sent to the database.

What we are doing here is doing only one query to get the number of books for all 100 records at once. To do so, we use the sql module, which is internally used by tryton for all database related operations. We will now go through the code, line by line, and explain what is going on.

result = {x.id: 0 for x in authors}

We must initialize the return value for all authors. We are going to query the database on the library_book table, grouping by authors. However, if an author does not have any book, we must still return a value for the number_of_books field, so we take care of this by initializing the return value to 0 for all authors.

As explained earlier, we are going to query the library_book table. To be more accurate, we are going to query the table which tryton uses to store the data about records of the library.book model. Usually, the name of the table created by tryton for a model will be derived from the model's __name__ (again) by replacing the dots (.) with underscores (_). However, this may not always be the case, depending on various factors. So we will use the proper way to get a database table for a query: by asking tryton to give us one:

Book = Pool().get('library.book')
book = Book.__table__()

You will see (and write) the first line or variants of it very often when developing tryton modules. The Book variable (note that it is capitalized) is going to hold the model class for the library.book model. This model IS NOT the Book class later defined in the library.py file. In a few words, it is a class that is build by tryton to include the Book class, and all modifications that are made by other installed modules to the library.book model. If we imagine another module that adds a new field on the library.book model, the Book variable that we got here will contain the information about this field, when the Book class of the library.py file does not.

Long story short, when you need to get the class that corresponds to a model, you should use Pool().get('<model_name>').

Note: The Pool we got here is the same one than that of the __init__.py file, in which we registerd our python classes

So our Book variable now holds the python class that tryton uses to represent the library.book model. We then call the __table__ method on it to get a sql table object for the library.book model.

Note: You could have directly written:

from sql import Table
book = Table('library_book')

however there would still be the risk that you be in one of the few case where the database table name is not directly derived from the __name__ of the model We got a table object, let's go on:

cursor = Transaction().connection.cursor()

The Transaction object will be covered in depth later on. Here we use it to get an open connection to the database, which is the only way to directly execute queries. That is how tryton does internally when it needs to manipulate the database, and something that you will have to do, more or less often depending on your needs.

The rest of the code should be straightforward. We execute our query with the cursor that the Transaction gave us, parse the return values, update the result, then return it.

We successfully managed to group the computation of our number_of_books field, so that reading 100 authors at once will only make one query (for this field).

Additional informations

Naming

The rules to name a Function field are the same than those of "standard" fields. They are usually located after the "normal" fields, but there may be exceptions for particularly important Function fields.

The getter of a Function fields should start with getter_ or less ideally get_. The latter is prefered by some, though it does not directly identify the method as a getter because there are many non-getter methods which start with get_.

Finally, the second parameter of the classmethod Function fields should be named so as to reflect their contents. This is not a rule specific to getters, but this is the first time we got to talk about it. We do not know the parameters type from just reading the method declaration, so that got to be included in the variables names when possible.

Choosing the right getter mode

There are no absolutes, but guidelines:

• If the instance method contains database queries / searches, or iterations on non trivial lists, going classmethod may significatively improve performances
• A field displayed in a tree / list view will always be "mass-read" by tryton. So if its getter has bad performances, it will be directly noticeable by the end-user. It is a strong case for classmethod getters

However:

• An expensive Function field that will be called once in a while, on few records, may not be worth rewriting in a classmethod
• All instancemethod getters cannot be optimized by being converted to classmethods
• Classmethod getters code is usually more complicated to understand. A small performance increase at the cost of unreadable code may not be worth
• "premature optimization is the root of all evil"

The hidden getter prototype

There is a rarely used version of the classmethod getter which uses the names parameter rather than name. Tryton is able to detect this and behaves differently in this case. However, the use cases are very limited, so we will not cover this case here.

Setters

Function fields have an optional third argument, the setter. When set, it will make the field modifiable. The setter value must be, as the getter, the name of a function that will be called to set the field value. Its prototype is the following:

@classmethod
def setter_my_field_name(cls, instances, name, value):
    pass

Here, the instances parameter will contain the list of records to update, name the name of the field (the same than for the getter function), and value the value that should be set for the field.

We will not use setters in this training module, because it has some inconvenients that can be avoided in most cases by other means (that we will talk about later):

• When a user sets multiple Function fields with setters, then saves, the setters are called in an unknown order, which may be inconvenient to manage
• Setters are called after the other fields are set, and will trigger additional validation calls after they made their modifications

All in all, they exist but are not often used. You way consider using them in some cases, but it should be limited to specific needs.

Searchers

The second, more often used, optional parameter of Function fields is the searcher. When a user tries to apply filters on a model (for instance, he wants to find all authors who were born before a given date), tryton converts the search parameters to a SQL query which is then executed on the database. This works well for "hard" data fields, which are in fact columns in the database (birth_date < XXXX => WHERE author.birth_date < 'XXXX'), but not so well for Function fields. Indeed, tryton has no way to guess how a age > XXXX can be converted to a database query, since the age column does not exist.

Searchers provides a way to explain to tryton how to convert the age > XXXX query to a database query. The code is usually rather similar to that of the classmethod getters, so writing a searcher may be a good occasion to upgrade an instancemethod getter to a more efficient classmethod.

Python: searcher is an optional argument, so when setting one, you should write it like so:

my_field = fields.Function(
    fields.Char('My Field'),
    'getter_my_field', searcher='searcher_my_field')

Generally speaking, when a parameter is optional, you should always specify it when calling the method / class, even when it is not stricly necessary

Tryton will not let you search on Function fields for which no searcher is defined (meaning it will raise an error), because it really does not know how to perform the search. Searchers are usually not defined for all Function fields, because:

• There are many small fields that the user has no reason to search on
• Writing a searcher requires better understanding of both the model, and SQL, so it may take some development and maintenance time

There are some use cases right now that we could implement in our library module, but that will come after we had a talk on how searches are technically done in tryton, in the next step.

Order

As explained in step 4, all columns that are displayed to the users in a tree view can be clicked on to order the list of values on that column. The same way as for searches, tryton converts the request to order in a SQL query which includes the order criterion. The same as searches, Function fields cannot be natively ordered (the tryton client usually raises an error when trying to do so).

It is possible to make it possible for tryton to order a Function fields. Contrary to getter, setter or searcher methods, it is not done through an argument in the field declaration, but simply by declaring a classmethod named def order_<my_field_name>.

The implementation of this method is outside the scope of this module, consult the tryton documentation for more details.

Function fields usage

Function fields are essential to writing good tryton modules. They allow to avoid data duplication across modules, display calculated informations to the user, and is unavoidable when trying to fine tune the UI (more on this on the next step).

Properly deciding between instancemethods and classmethods is important, since it may cause performances problems. Sometimes the answer is obvious, sometimes it requires some consideration since the optimization may be too costly, and the field not so often used.

rec_name

We saw in the previous step that we could use the _rec_name variable to tell to tryton how it should display a record. For now, we set it to existing Char fields, and it looked good.

There is a hidden Function field in tryton, named rec_name. This mean that if there is no existing field that does what you need, you can use it to write the string you want.

Add the following code in library.book.exemplary:

def get_rec_name(self, name):
    return '%s: %s' % (self.book.rec_name, self.identifier)

And voilà. Our exemplaries will now be identified by both the name of the book they are linked to, as well as their unique identifier, which will be far more useful.

You can also replace the identifier field in the view/exemplary_list.xml file so that it uses rec_name.

Homework

You will have to add the following fields in your module. For each of them, ask yourself the following questions:

• Should it be a Function field ?
• If so, should the getter be an instancemethod or classmethod ?
• Should there be a searcher method ?

For every field you should follow the rules that were detailed in step 3 regarding naming, positioning, etc. You should as well add the fields in the views, and check that they are properly calculated. If there are Function fields for which you think there should be a searcher, declare it but leave it empty for now (we did not learn how to write it yet).

Here are the fields that you should add:

• The date at which a book was published
• The genres an author has written in
• The most recent exemplary of a book
• The ISBN of a book
• The most recent book an author wrote
• The number of exemplaries of a book
• The number of books an editor published
• The list of books an editor published in the past year

Once you are done, you can compare your code with that of the step5_homework branch for differences. You can then read here about what you should have done, and why.

What's next

We are fleshing up our model with calculated informations, which are the last building blocks of tryton.

Next we are going to add constraints and rules on our models / fields to have consistency across our data.