Gathering repeated information: working with Docassemble Groups

Although I understood the underlying datastructures, Docassemble groups are a concept that took me some time to wrap my head around. I hope you can use this as a guide that will help you understand the why and how of groups and get started using them in practice. I hope to add more similar posts as our Docassemble Working Group at Greater Boston Legal Services continues.

Building a template with Docassemble is straightforward if you’re converting a simple letter or most legal forms. It can be a one-to-one mapping from the blank fields in the template into a Docassemble variable.

What happens when the template is asking you to provide very similar information more than once, such as a row of household members? Should you write person1, person2, person3, followed by person1_birthdate…, person1_income, etc? Well, that would work just fine, but Docassemble offers a very powerful alternative with the concept of Groups. Groups make gathering multiples of data more accurate, with less effort to make use of the gathered data, too. It’s just one more way to keep related data together in a logical and consistent way.

Groups are a Docassemble-specific term for a subset of what might be called datastructures in other languages. You can think of a group as a collection of related variables, with special features that make it easier to work with the full collection or a part of the collection at a time.

What is a Group?

Docassemble uses the term groups to describe a feature very similar to the basic computer science concept of an array, which is also the basic structure behind what HotDocs calls a REPEAT instruction. The basic concept of all of these datastructures is to let you store multiples of the same kind of item.

There are three different kinds of basic groups in Docassemble: Lists, Dictionaries, and Sets. Tuples are a fourth, special group that you may use in advanced circumstances.

What is a list?

Most of the time, when you want a group, you want to use a list. It’s a basic group that lets you keep adding as many items as you want. Each item in a list can be referred to using a numeric index. The Class name for a Docassemble list is DAList. Docassemble has a nice utility method, .item(number) that lets you refer to an item in a DAList that hasn’t been defined yet without generating an error. You can also refer to list items like this: household[0].name.first. The brackets after the list’s name mean you’re trying to retrieve item number from the list.

Lists can be combined, sorted, and have items removed and added at any place in the list. You can also filter a list to show items that match specific criteria.

You can create a list in a code block by using square brackets, with each item separated by a comma, or a special Docassemble list using something called a class constructor, which is just the name of the Class, followed by parentheses surrounding 0 or more arguments:

my_standard_list = ['item1','item2']
empty_list = []
empty_docassemble_list = DAList()

In Docassemble and Python, a list grows to the size you need and you don’t need to say how big it will be when you create it. Just as in most other computer programming languages, the first item in a list has an index of 0, not 1. Lists are ordered, meaning that if you add the item A, followed by B, E, and D, it will print in the same order, A, B, E, D.. However, you can sort a list.

Use a list anytime you aren’t sure exactly how many of an item you are going to gather.

What is a dictionary?

A dictionary is a group, like a list, but with the special feature that each item in the group can be retrieved or set using a keyword (key) rather than a numeric index. You use the keys both to set each entry in a dictionary and to retrieve it. Typically you know the keys in advance. Docassemble uses a dictionary internally to store the response provided in a checkbox field. The class name for Docassemble dictionary is DADict. To handle a checkbox field, Docassemble offers special methods .true_values() and .any_true() as well as .false_values() and .any_false() that return a list of just the items that are true/false or a single True/False that lets you know if any item on the checkbox list was marked true. We still use the bracket syntax to retrieve or set a dictionary item, but instead of a number, you a text keyword.

For example, suppose you have a list of possible clauses that you want to include in a template, in a field named clauses. One option, or key, in the checklist is named no_jurisdiction. The user can select the items that they want using a checkbox field. You can make use of the responses in your template like this:

{%p if clauses['no_jurisdiction'] %} 
This court has no jurisdiction over the claim. 
{%p endif %}

You can create a dictionary in a code block using curly braces, separating each item by a comma, and each keyword from the value using a colon, or a Docassemble dictionary using the class constructor DADict():

clauses = {'key1': 'value1','key2':'value2'}
my_empty_dict = {}
my_empty_docassemble_dict = DADict()

You likely won’t create a dictionary by hand very often in Docassemble, unless you are creating your own custom modules. More likely, when you’re using a DADict, it is going to be one created by using the checkbox datatype in a standard Docassemble question. A dictionary is unordered, meaning it won’t keep track of the order you add items into it. You can sort a dictionary if you want, using the keys or the values.

What is a Set?

A set is a list with a special logical property: each item in a set is unique. Using a set makes sense if you want to let the user keep adding items, but you never want two of the same item to end up in the resulting list that you make use of. You might use a set to keep track of one attribute of the items you’re gathering in an ordinary DAList. For example, suppose you are gathering cars, and you want to know what colors of cars have been gathered so far. Suppose you have the DAList with elements below:

my_list.elements = [{type: "Subaru Forester",
  color: "red"},
 {type: "Honda Accord",
  color: "blue"},
 {type: "Honda Accord",
  color: "green"},
 {type: "Toyota Tercel",
  color: red"}
]

my_set = set([item.color for item in my_list])

Then ${my_set} will display red,blue,green instead of red,blue,green,red.

The fact that each item in a set is unique means that you can do the sort of operations you may remember from learning about Venn diagrams in your high school math class, such as union, intersection, and difference.

If you want unique items in your list, or if you want to do mathematical operations to compare two lists, you may want to use a set. The class name for a Docassemble set is DASet, and you can create a new, empty DASet using its class constructor, DASet(). Like a dictionary, a Set is unordered.

What is a tuple?

More set theory and advanced math terms here, but a tuple is actually very simple. You create a tuple just by surrounding two or more values with parentheses, with each item separated with a comma:  my_tuple = (‘value1′,’value2’).

A tuple is ordered and immutable (the contents can’t be changed). That makes it not very useful for storing a Docassemble interview’s field.

Docassemble doesn’t have any special classes to implement a tuple in an interview. However, some advanced Docassemble built-in functions return a tuple, and there is special syntax to work with a tuple return type. If you want to assign the return value of a function to multiple variables instead of just one, you can make use of the parentheses syntax to break the tuple apart: (var1,var2) = function_that_returns_tuple(my_arguments).

Working with Lists in Docassemble

Let’s skip an advanced discussion of tuples, lists and dictionaries here. Although all three offer special additional features, you can generally treat a Dictionary or a Set in Python just like a list and a list is the datatype you’ll use most often.

DALists are objects

DALists are objects. If you want to use one in your interview, you first need to tell Docassemble using the initial block objects in your interview file. You will also need to include two modules, docassemble.base.core and docassemble.base.util.

---
modules:
  - docassemble.base.core
  - docassemble.base.util
---
objects:
  - household_members: DAList

If you want to store a particular object type in your DAList, such as Individual, you should tell Docassemble with the .using() method.

---
objects:
- household_members: DAList.using(object_type=Individual)

Using an interview to gather a DAList

Once you understand how lists work in Python, you still need to learn how to gather the items into your list in Docassemble. This should ideally map onto the most natural way you would ask your user about the information in a one-on-one interview. Docassemble offers a linear interview experience, which constrains the basic structure of gathering list items. However, the available options still give you a lot of flexibility.

There are at least four different ways to build a list in Docassemble:

1. Gather one a time, asking a follow-up question each time to find out if the user wants to keep gathering more.
2. Ask for the total number of items up front, and then gather the items one at a time.
3. Add some items with code, either using information the user already provided you, external information, or information you know when you create the interview
4. Use a table to display progress in gathering the list, and offer the user the opportunity to click a button to add more items.

Options 2 and 4 are the most likely to be a natural way to interact with your end user. I suggest that you ask for the total number of items up front whenever it’s likely that the user normally thinks about the total. For example, you could use this approach if you’re asking for information about the user’s children.

If the total isn’t easily mentally recalled and the user would have to count the total of something, don’t make them! Let them see the items they’ve already added as a mental cue that will help them decide if the list is complete or if there are still more items to add.

You can mix and match all 4 methods, wherever it provides the most usable experience for your end user.

Explaining the special attributes .there_are_any, .there_is_another and auto_gather

When you use a DAList, you have two options: allow Docassemble to gather the list items automatically (i.e., as part of the questions you give to the end user) or to build it with code. If you know you will build your DAList with code, you can turn off automatic gathering by setting my_list.auto_gather = False or in the initial objects block with .using:

---
objects:
- household_members.using(object_type=Individual, auto_gather = False)

When using auto_gather = False, you should have a code block that sets the value of gathered to True when the list is complete.

If you want to use Docassemble questions to add items to your list, you need to understand two Boolean (True/False) attributes that each DAList has: there_are_any and there_is_another.  If your DAList object is named household_members, you can set or access the value of both attributes with the familiar dot notation: in a code block or in a regular Docassemble question or template. E.g., household_members.there_are_any. Alternatively, you can set or access the attributes in your objects block with the .using syntax.

---
objects:
  - household_members.using(object_type=Individual, there_are_any=True)

If there_are_any is False, then the DAList will be empty. If it is True, Docassemble will look for a question that defines household_members[0]. After you gather the first item, Docassemble will look for the definition of household_members.there_is_another to see if it needs to keep gathering household_members[1], household_members[2], and so on. If there_is_another is false, Docassemble will stop trying to add items for you.

Special variable i

So far we have talked about gathering list items with a specific index number. In order to write a question that can fill in any item in your list, you make use of a special variable, i. To go along with computer science conventions, i stands for an index in your list. In Docassemble, you can also use the variables j, k, l, m, and n.

The following question will gather information about any item in your list household_members:

---
question: |
  Information about Household Member ${i}
fields:
  - First name: household_members[i].name.first
  - Last name: household_members[i].name.last
  - Date of birth: household_members[i].birthdate
    datatype: date

In the question above, i will be replaced with 0, 1, 2, and so on until we’re done gathering items for our list. You don’t have to use i: you could use 0, 1, 2, etc. and write a separate question for each household member, but it wouldn’t make much sense to do so. Adding in j, k, l, m, n allows you to add up to 6 levels of nested lists in a single question. You can reuse the variable i in multiple unrelated questions. It will also stand for the index of the list that your question refers to.

By the way, there’s a special function to make it nicer to work with the index variable. Remember, 0 represents the first item in your list, 1 the second, and so on. Instead of forcing you to keep track of this, the Docassemble function ordinal() returns the word first for 0, second for 1, and so on. ordinal() is internationalized and can be automatically translated to match the language of your interview.

---
question: |
  Information about the ${ordinal(i)} Household Member. 
fields:
  - First name: household_members[i].name.first
  - Last name: household_members[i].name.last
  - Date of birth: household_members[i].birthdate
    datatype: date

This will display Information about the first Household Member in your Docassemble interview.

A simple interview to gather list items

Let’s put what we’ve learned so far together. Going all the way back to our list of 4 different ways to gather a list, this example will be for method 1.

--- 
modules:
  - docassemble.base.core
  - docassemble.base.util
---
objects:
  - household_members: DAList.using(object_type=Individual)
---
question: |
  Does anybody live with you?
yesno: household_members.there_are_any
---
question: |
  Information about the ${ordinal(i)} Household Member. 
fields:
  - First name: household_members[i].name.first
  - Last name: household_members[i].name.last
  - Date of birth: household_members[i].birthdate
    datatype: date
---
question: |
  So far you have told us about ${len(household_members)} household members.
  Does anyone else live with you?
yesno: household_members.there_is_another
---
mandatory: True
question: |
  The household members are ${household_member}

The example above uses one more function we haven’t gone over yet, len(). Len is short for length, and returns the number of items in a list.

You can gather all of your lists this way, but it might be a little clunky for your end user. There are a lot of clicks, and there isn’t a lot of feedback along the way. After each item in the list, the interview will ask “is there another?” until you answer no.

Gathering a specific number of items

If you believe that your end user will know the number of items in the list, it makes sense to ask the user up front. We do this by setting the .ask_number attribute of our list to true and writing a question to define the .target_number  attribute.

The benefit of doing this is that the end user will now just be asked about each item in turn, saving a click on the “Is there another?” question. Below is a sample interview that uses this technique:

--- 
modules:
  - docassemble.base.core
  - docassemble.base.util
---
objects:
  - household_members: DAList.using(object_type=Individual,ask_number=True)
--- 
question: | 
  Does anybody live with you? 
yesno: household_members.there_are_any
---
question: |
  About your household
fields:
  - How many people live with you? household_members.target_number
    datatype: integer
---
question: |
  Information about the ${ordinal(i)} Household Member. 
fields:
  - First name: household_members[i].name.first
  - Last name: household_members[i].name.last
  - Date of birth: household_members[i].birthdate
    datatype: date
---
mandatory: True
question: |
  The household members are ${household_member}

Gathering the list with a table

An alternative way to gather the items in your list is to use a special table that gives you the option of editing, deleting, or adding new items to a list. Docassemble still takes you to a separate screen to add or edit your item, but the experience is much closer to that of editing a row in a spreadsheet. The user can both see where they are in the list, and see the details of each item that has already been gathered. When you use this method, you may still want to make use of the .there_are_any attribute, but probably not the .there_is_another attribute. Instead, you’ll rely on the user to click a button to add another item.

Docassemble automatically asks the there_is_another question, but it won’t display the table on its own. You need to write a bit more code to use this method of gathering items. You’ll need to add in a mandatory code block that refers to the table, a table block, and a question that displays your table. You also make use of the special method of a DAList, add_action() which displays a button in an interview that adds a new item to your list.

One more note: the code block that displays the table to review your list will run each time you review the list. So, make sure that you’re only prompting for questions in this code block, not running any actions that can’t run more than once. This is called idempotence.

--- 
modules:
  - docassemble.base.core
  - docassemble.base.util
---
objects:
  - household_members: DAList.using(object_type=Individual,there_is_another=False)
---
mandatory: True
code: |
  if household_members.there_are_any:
    review_household_members
---
question: | 
  Does anybody live with you?
yesno: household_members.there_are_any
---
table: household_members.table
rows: household_members
columns: 
  - Name: row_item
  - Birthdate: row_item.birthdate
edit:
  - name.first
---
question: | 
  Household members
subquestion: |
  So far you have added ${household_members.number_as_word()} household members.
  
  ${household_members.table}

  ${household_members.add_action() }
field: review_household_members
---
question: |
  Information about the ${ordinal(i)} Household Member. 
fields:
  - First name: household_members[i].name.first
  - Last name: household_members[i].name.last
  - Date of birth: household_members[i].birthdate
    datatype: date
---
mandatory: True
question: |
  The household members are ${household_member}

Here is what that looks like.

Using tables to display information

By the way, a Docassemble table can also be used to display information. Just leave off the edit field. As of this writing, you can’t use the same table block for both editing and display.

One thing I discovered is that a table can be used to organize your data in other ways. The rows do not need to come from a DAList that you’ve gathered. For example, I have a dictionary of income types that is generated by a function non_wage_income_list(), and then I can use a table to group my list of gathered income into a table, where the rows are the type of income and each cell contains a total for that income type.

---
table: income_summary_table
rows: non_wage_income_list().keys()
columns:
  - Type: |
      non_wage_income_list()[row_item]
  - Client: |
      currency(client.incomes.total(type=row_item,period_to_use=12))
  - Spouse: |
      currency(spouse.incomes.total(type=row_item,period_to_use=12))
  - Household members: |
      currency(household.incomes.total(type=row_item,period_to_use=12))

Gathering items with more than one question in a logical order

Usually, you can use one question to add each item to your list. If your interview uses two or more questions to fill out all of the attribute of an object, you may want to make sure that each object is completely gathered before you move on to collecting the next object.

To help you do this, each DAList has an attribute complete_attribute  which should be set to the name of the variable that means the object has been completely gathered. Complete attribute is a string that refers to the name of an attribute of your object. That means it should be placed in quotes. It’s a good practice to always use complete_attribute if you are gathering an object to prevent problems if you change your question structure later.

If you want complete_attribute to refer to more than one attribute of your object, which is probably most common, you can do so with a code block that sets the attribute you named to True at its end. Docassemble will go through the code block in order for each item in the list, asking a question to define each variable you mention in the code block. It won’t stop defining the variables until the end of the code block when you assign a value to complete_attribute. Notice that if you know you gather two variables together (such as name.first and name.last), you only need to mention one. To make the code block work for every element in the list, we’ll use our special variable i  again.

---
objects:
  - household_members: DAList.using(object_type = Individual, complete_attribute='complete')
---
code: |
  household_members[i].name.first
  household_members[i].birthdate
  household_members[i].complete = True
---
question: |
  Name of ${ordinal(i)} household member
fields:
  - First: household_members[i].name.first
  - Last: household_members[i].name.last
---
question: |
  Birthdate of ${household_members[i]}
fields:
  - Birthdate: household_members[i].birthdate

Advanced list operations

Combining lists

Anywhere that you can use Python code (such as in certain question fields, in a Mako or Jinja2 statement, or in a code block), you can perform list operations.

The + operator lets you combine two lists and returns a new list. Wherever you want the combined list, you can instead write list1 + list2. The extend method modifies the list in place to add the items in the first list to it. E.g., after you run list1.extend(list2), list1 will have all of the items in both list1 and list2. However, the return value of the extend operation is None. So, you should use the extend method in a code block before you want to use the extended list.

To add items one at a time in a code block, you can use the .append() or .appendObject() methods. Use the appendObject method when the item you’re adding to the list is a new Object. You’ll also need to specify the object type when using appendObject().

Sorting a list

If your list contains elements that it makes sense to sort (such as a list of sentences or numbers), use the sorted() function to create a new list with the items in that logical order. Docassemble will try to sort your list no matter what kind of elements it contains, but it may not make sense. It is possible to tell Python (and thus Docassemble) how to sort custom objects but that is an advanced topic discussed in the link above.

Filtering lists

Suppose you’ve gathered a list of cars, and you only want to display the red ones. You could either use a for loop or a shorthand one-lined syntax called a list comprehension.

For loops

For loops are the most common type of loop in almost every programming language. They are an example of what is called a control structure in computer science. The basic idea is that you do something a certain number of times. Most commonly, you will do something to each item in a list.

In Docassemble, you can use a for loop in a code block, in a question block or from-scratch document (using Mako tags) or in a Docx template file (using Jinja2 tags).

Here’s a basic for loop that will assign every ‘red’ car in the list cars to a new list named red_cars:

---
code: |
  red_cars = DAList()

  for car in cars:
    if car.color == 'red':
      red_cars.append(car)

Let’s go through the example block above in more detail.

In line 1, we create a new, empty list. We are creating a DAList so we can make use of the special features it has, but we could also have created our new list with the [] syntax.

On the second line, we create a new variable car that will refer to each temporary item in the existing list cars. In line 4, we have an if statement which tests each item to see if it is ‘red’. Line 5 will run only if the test in line 4 evaluates to True. If so, it uses the appendObject method of a DAList to add the item to the new list.

Here is a similar for loop in a Docx template that will display only the red cars:

{%p for car in cars %}
{%p if car.color == 'red' %}
{{ car }}
{%p endif %}
{%p endfor %}

List Comprehensions

A list comprehension may take a little time to wrap your head around as a fairly advanced Python concept, but once you understand it it turns out to be very simple to use in practice. A list comprehension is a shorthand way to go through each item in the list, run an optional action on the list, and apply an optional test on each item in the list before running the action and adding the item to the new list.

---
code: |
  red_cars = [car for car in cars if car.color == 'red']

Let’s take a closer look at the code example above.

In section 1, we have an expression that represents the item that will be added to the new list. In the example, we’re leaving the car object alone, but we could have transformed it with any operation of our choice. E.g., if we wanted a list of all of the colors in our list of cars, we could have replaced the expression with car.color. The expression could also include some operation on the items in the list, the results of a method or function, or any simple mathematical expression that made sense, such as car+1, car*car, etc.

In section 2, we have what should be a familiar for … in loop. Here we iterate through the list cars (on the right hand side of the expression) and assign each list item to a new variable named car.

In section 3, we have an expression that begins with if that should return a Boolean value. Our new list will only contain the items where the expression returns True.

There is a special shorthand method of a DAList as of Docassemble 0.3.14 which simplifies this and makes it a little more readable for this common operation on a list of objects into the following:

---
code: |
  red_cars = cars.filter(color='red')

Where color could be replaced with any attribute of the objects stored in the list cars, and ‘red’ is a value that we want the attribute to match. This simplified syntax will only replace this specific type of list comprehension. It would not work, say, if you wanted to go a second level deep in your list of objects. And it will only filter for one attribute of your object. Notice that we use color=red and not color==red here. This is because we’re not writing an if expression; we are using a Python keyword argument.

Conclusion

Working with groups can be a little overwhelming. I hope this short discussion helps.