MADE is a wizard-like online interview that helps tenants who are being evicted. It can be completed on a smartphone or desktop computer completely without the help of a lawyer. It covers 300 questions and it educates tenants about their rights, helps tenants file paperwork, and sends tenants automated text-message reminders about follow-up paperwork and their court date.
As of this writing (From November 2018 until the beginning of May, 2019), without any publicity, about 500 people have used MADE to fight their eviction case.
I wrote this post to summarize the development experience of MADE, in hopes that it will be useful to people working on similar projects. I am not even close to an expert on user-centered or community-centered design, but I did my best to follow a similar process.
Why create MADE?
I started working on MADE to speed up our attorneys’ work; make eviction help available to more tenants; and to cut down on what I call the “referral loop.”
My employer, Greater Boston Legal Services, has run a weekly Monday eviction clinic for decades. We ask tenants to come in at about 9 AM, and tenants can usually leave by 1:00. Some of our attorneys would keep tenants until 3 or even 4:00 in the afternoon!
Why did it take so long? In the clinic, we would go through paperwork that covered those same 300 questions, line by line in a group. We had to go through each question for every tenant, even though some tenants had much simpler cases and each case involved a different subset of the issues.
After tenants complete the paperwork, they get a personal review and our administrative assistant photocopies and collates all of their signed paperwork. This photocopying step alone takes a significant amount of time.
It wasn’t realistic to have tenants fill out the paperwork on their own. Above is a small screenshot of just a few of the questions we guided tenants through in the clinic. For example, one question asks if “The landlord did not terminate my tenancy properly.” Not an easy questions to answer, even for attorneys! Not the fault of the editors: over the years there were many attempts to simplify the forms, but there was never a consensus to take out information that related to the tenant’s important legal rights and defenses. The alternative, a 100 page form just wasn’t possible.
Beyond the complexity of the forms, our clinic wasn’t truly accessible to everyone. Many tenants are not able to attend a 4 hour clinic on a Monday. That includes tenants with children and no childcare, tenants who would have to take unpaid time off from work, and tenants with disabilities that make it hard to leave their home or to sit in a clinic setting for an extended amount of time.
Finally, many people experiencing legal or social problems experience what I call the referral loop. I’ve been dismayed to see people seeking legal help through various venues for the same problem without getting assistance, and instead getting a mix of new and old referrals. Because Greater Boston Legal Services runs clinics around the region and partners with tenant organizers, shelters, and social service agencies, often those clients end up speaking with a GBLS attorney at each place they try, which makes their many stops painfully clear in our case management system. This can be exhausting for people experiencing poverty who already have many demands on their time and limited money for transportation.
How MADE solves the problems
Because MADE is completely self-guided and available on a smartphone, tenants are able to access and learn about MADE all at the same place: when they attend an intake clinic at a shelter like Rosie’s Place, seek rental help at a social service agency, or even when they call our intake triage partner. There’s no more referral loop.
We still run our weekly clinics, but instead of taking 4 hours or more, tenants take about an hour to an hour and half to go through the system on their own. Each set is signed electronically, meaning no more photocopying and collating. When they are helped by an experienced advocate in a one-on-one setting, it typically takes about 30 minutes (instead of an hour as it usually would on paper). I often help people one-on-one at a local shelter. Cutting down the time to 30 minutes means more time for advice and less filling out paperwork.
MADE has been used in an unforeseen way; I did not know when I created it how frequent this problem is, but in just the first few weeks of our expanded testing, we had two separate tenants call who were completely homebound. One tenant could only leave their home in an ambulance. People in this situation often have social workers, and for these two tenants that was also true. With MADE their social worker was able to assist them with completing the interview and we were able to help them print and file the paperwork in court. They had time to request a reasonable accommodation from the housing court.
With the paper forms, social workers rightly felt uncomfortable helping a tenant with completing the Answer and Discovery forms, let alone even basic motions. Yet, because of the short time window for a response, tenants often have had no other option. MADE adds a level of comfort for social workers, as they are not required to understand or advise tenants on the law to help them complete the paperwork on time.
User Centered Design Process
In building MADE, I did my best to follow principles of user-centered design. My goal was to replicate a friendly one-on-one session with an attorney in a computerized form. The system would be linear with a logical and accessible order of questions.
User centered design is a framework which places users at the center product development. There are four phases, which include
- Discovery (understanding the context)
- Specifying the user’s requirements
- Designing a prototype
- Evaluation and testing
Creating MADE involved hundreds of revisions, with user testing in between each prototype.
One of the most important aspects of user-centered design is that it is iterative. The image above is a visualization of almost a year of code additions/deletions to MADE. After building a prototype and placing it in the hands of users to get feedback, you repeat any of the 4 phases to continue to refine the product. It’s key to get a prototype up and running early to test assumptions about users made in the discovery phase.
The first code was written in December 2017. Overall, the initial discovery to prototype phase took about 3 months. The next months involved several iterations with user testing, with usable but not yet perfect forms, finally being ready to release to the public in a “beta” version in November 2018.
When developing MADE, the initial discovery and specification phases were informal. I spent time talking to my colleagues about what they thought would be useful, but I also was able to rely on my 10 years of experience running eviction clinics and assisting tenants one-on-one at Rosie’s Place. I had experience both with using the existing paper forms, and with an earlier computer-based system that had poor adoption among our staff.
I focused a lot of time on building an early prototype. I knew that the first draft wouldn’t meet all of my goals. My first draft was a literal translation of the paper form, which is complex and hard for pro se users.
I had to swallow my pride with this first draft. It felt risky to show people something that didn’t meet my final standards. I was especially concerned about showing people an early draft that wasn’t very good and them deciding not to pay attention to later drafts, but it really paid off. I learned a lot from this rough prototype that guided where I spent my time in working on more polished versions of the tool.
User Testing
I was lucky enough to be able to collaborate with Rina Padua, a recent graduate of Harvard College who was invaluable at gathering user feedback and testing. Here are some of the ways that we did user testing (many of it driven by and facilitated by Rina):
- One-on-one sessions at City Life/Vida Urbana, Harvard Legal Services Center, and Harvard Legal Aid Bureau
- Sessions included both student attorneys, and observations of tenants using the system on their own
- We used both live data (i.e., asked a user who had a court case coming up to try using the system) and had users who had completed interviews on the paper form try to match the output using our computerized system
- Regular weekly sessions at Rosie’s Place as part of live clinic services, driven by myself (experienced housing attorney)
- Sessions at Greater Boston Legal Services, in real weekly clinic settings
- These were driven completely by tenants, with a floating attorney to answer questions and troubleshoot problems
Stakeholder engagement and feedback
Meetings with stakeholders were a key part of the development process for MADE. Overall, Rina and I together conducted about a dozen demos of MADE that focused both on outreach and gathering feedback from different kinds of users of the tool.
These sessions were sometimes with just 1 or two people, and up to about 15-20 users. We went through the system in detail, demonstrating features and soliciting input.
Some of the stakeholders we met with included:
- Court Service Center at the Edward Brooke Courthouse
- Summary Process clerks at the Eastern Housing Court
- Attorneys at Greater Boston Legal Services, in groups and with individual attorneys
- Tenant organizers and tenant groups at City Life/Vida Urbana
- Student and clinical instructors at Harvard Legal Aid Bureau
- State-wide legal aid Housing Coalition meetings
- HomeStart
- South Shore Elder Services
- Mass Coalition for the Homeless
- The Legal Service Corporation’s annual Innovations in Technology Conference
- Classes at Suffolk Law School and the Suffolk Legal Invocation and Technology Laboratory
- Mass Law Reform Institute, with Caroline Robinson providing extensive, detailed evaluation of plain language at an early stage. Maura Kelly, access to justice coordinator for the State Trial Courts, also provided helpful plain language feedback on key questions.
Individual user testing consisted of allowing the user to drive the interview on their own, observing how they interacted with it, and prompting them when they ran into roadblocks to allow them to continue to use the system. We tried to get both general feedback from the user’s reports and from observation of what steps in using the system caused difficulty.
Advocate-driven individual testing was done in a similar observational style. In addition, as both author and user of the system at Rosie’s Place, I gauged changes needed for the tool by trying out the order and phrasing of questions in real-time on real cases. It often led to re-wording or reordering of questions.
Gathering Effective Feedback
We found that the perfect size to get good feedback from live demos was about 10-15 users. This was enough to prompt lively discussions and everyone had a chance to express their opinions. I went to meet these audiences at their offices. This encouraged wide attendance representing a spectrum of members of the respective organizations.
I approached each session as a learning opportunity, with the result that I learned quite a bit that influenced the final version of MADE. We gathered notes from each session and paid particular attention to feedback that was repeated among different audiences; that made the language clearer or more accurate; and that protected substantive rights of the tenants.
What kinds of feedback we gathered
By doing this testing we learned several things that I wouldn’t have learned on my own, even with 10 years of experience with the paper forms. In particular, we learned a lot about how users experienced the forms, and about the needs of judges and court personnel that would make the system accessible in the Court Service Centers. We gained very helpful perspective on the needs of the court from clerks, internal access to justice staff, and court service center staff.
We had two very different kinds of pro se users: some were intimidated by too much information. These were often slower readers and focused on information on the screen that wasn’t relevant. We wanted to limit the information shown to those user to the smallest relevant amount, and design questions to require little explanatory text. Other users were hyper-learners. They wanted to really understand everything about the questions we were asking.
Once we identified the two groups of users, we were able to experiment with adding different amounts of information on the screen or behind popups, links, and the help button. We also made changes to the phrasing of questions. For example, we removed compound statements (“Did your landlord enter your home without your permission or notice”) and replaced them with two statements, with logic that combined them in the final forms when appropriate.
All of the users wanted feedback early on about the results of what they were telling us. We ask a lot of questions that relate to claims and defenses. Giving them information about what the answers led to was important. To address this need, we added interstitial messages at key points that reflected back the results of the user’s input. This also made the system more transparent.
We had a risk of exhaustion, which made sense given 300 possible questions. To reduce this, we made sure to update the user about the amount of progress they had made. We also added additional steps to reduce time, such as address autocompletion, automatic selection of courts, and more efforts to leverage information the computer could easily discover.
We learned that an important use case was tenants with low literacy. As a result we enabled text-to-speech support.
We learned more about how to accommodate both users of the system who could return to the forms online later for follow-up, and how to provide information in an accessible way that the user could print and read later.
We went through dozens of revisions for some key screens, particularly to use language at an appropriate grade level (6-8th grade was our target ceiling), take away information that wasn’t relevant, and to customize screens to show only appropriate options (such as removing the option to provide service by mail on the Answer date). Grouping of options on different screens, providing summary screeners, and more were all features of the system that changed due to user feedback and observation.
MADE was written on top of a platform for building interactive legal applications called Docassemble (similar to Odyssey Guide and File, A2J Author, and HotDocs). As a result of our user testing and requests, the author of Docassemble added numerous customizations and features that have benefited other users. For example, changing the location of interface buttons, the visual appearance of the text-to-speech feature, and substantive features such as the address auto completion were all based on our collaboration with the author.
Conclusion
I am certain that MADE would not have been as good a product without the focus on user testing and engaging with so many stakeholders throughout the process. Each time I did a demo or user test, I learned something new that I was able to incorporate into the system and make it better for more users. This was time consuming but worthwhile, and it didn’t require any external resources.
You can test MADE at https://interviews-dev.gbls.org/start/eviction, or run a real session at https://www.gbls.org/MADE.
0 Comments