Chapter 11: Design a System to Track Your Program

Be a yardstick for quality. Some people aren't used to an environment where excellence is expected.

Steve Jobs

Once your planning committee has decided to create an ADR program, determined what your goals for the program are and has used those goals to decide the program design, you will need to figure out how you’ll know if the program, as designed, is doing what it should. In other words, you need a system to track, or monitor, how your program is working. This is an ongoing process that allows you to compare program functioning from year to year and use that information to make any necessary changes. An analogy is a coach tracking a hurdler’s time off the block, running speed and form when jumping the hurdles to determine if any aspect of her racing needs to be worked on and to firm up what she is doing well. In this case, you collect data about the program and check it regularly to see what about the program is working well and to make sure there are no problems that need to be addressed.

Many, if not most, courts track their programs, if minimally. If you are tracking the number of cases that go to mediation, you are tracking your program. You’ll definitely want to do more if you really want to know about the health of your program.

What are the Benefits of Tracking Your Program?

Generally speaking, there are three reasons to track your program. First, you will have a handle on what is happening with your program and, therefore, can make better decisions about it. Second, you will be able to make the case to stakeholders that your program is effective. You will be able to credibly show your program’s worth to stakeholders and funders and be accountable to them and to the public by letting them know how effectively you are using public resources. Third, you will be able to improve your program’s performance, if needed. In sum, you will be better able to ensure that those who come to you to resolve their disputes are being provided the quality of service you and they expect of the court, and be better positioned to tout your successes.

Tracking will tell you what’s going on with your program by answering questions such as those presented below.

Number of cases: Is the number of cases going through ADR properly matched to the amount of staff time allocated to it? That is, are there too many cases for a part-time staff person to be able to properly administer the program? Are there too few to warrant keeping a staff person devoted to it full-time?
Achieving goals: Is the number of cases going through ADR sufficient for achieving program goals? If, for example, you are counting on ADR to reduce case burden on judges, but only a few cases are going through ADR, you can get a sense early on that this goal probably isn’t being met.
Outcomes: Is your program having the desired outcomes? For example, are parties satisfied and reaching agreements?
Neutral performance: How well are particular neutrals performing? Are parties noting issues with particular neutrals on their post-session surveys?

Another way to think about tracking is to imagine what would happen to your program if you didn’t track it. For example, what would happen when the one powerful lawyer whose case dragged through ADR (due to the lawyer’s own behavior) complained to the presiding judge that ADR slows everything down? You would not have data to show whether his experience was the norm or was an outlier. Or, what would happen when there are court budget cuts? You would be unable to make the case for the positive effect of your program.

Planning and Designing Your Tracking System

If your planning committee is in the early stages of developing your ADR program, you should include a tracking system as a component of the program design and put it in place as the program is launched. If your program is already in place, you can create the tracking system at any time during the life of the program. In either case, you need to make it an integral part of the program, with an expectation that program tracking will be done consistently. If you are already tracking your program, but don't know what to do with the data, you may want to use this chapter to help you focus on which data you already have that will be most useful. Then, in the long run you might rethink your tracking system so that you can more easily and automatically interpret the results.

An effective tracking system involves:

Deciding what you want to track and what information is needed
Figuring out how you will collect, store and analyze the information
Determining how to report on the program and to whom you will provide reports
Using your reports to understand your program

Note that program tracking is different from evaluation. Evaluation will give you an in-depth look at your program at a point in time and often focuses on goal achievement. Using the hurdler example provided above, evaluation would be similar to examining the hurdler’s diet, workout regimen and fitness markers, as well as direct performance indicators such as speed and form over the hurdles. These factors aren’t continuously tracked, but would provide more data in order to make overall improvements.

Step 1: Decide What You Will Need to Track and Where That Information Will Come From

When you can measure what you are speaking about . . . you know something about it; but when you cannot measure it . . . your knowledge is of a meager and unsatisfactory kind.

Lord Kelvin

Because program tracking is an ongoing process, you will need to find the line between need for information and the effort to get it. It’s tempting to want to track everything about how your program is functioning. It’s even more tempting to only collect information on the easiest data to capture: how many cases are going through ADR and what the results are, such as how many arbitration awards are accepted or how many mediations end with an agreement. You definitely want this information, but if you track more than this, you will get a much better picture of how effective and efficient your program is, and will be able to make better decisions about it.

The information you collect may depend on what your program goals are and what its indicators of success are, but also should be governed by the feasibility of collecting that data. Fortunately, it is becoming easier to automate data collection, making tracking a number of factors much more feasible. The information that should be collected for all programs is listed below.

Number of cases referred: This data can be tracked through information on the order of referral to ADR or through a report submitted by the neutral.
Number of cases going through the ADR process: This data can be tracked through reports filed by the neutrals after the ADR process has been completed – or after the process has been terminated in some other way (such as a party not appearing).
Outcome of cases going through ADR: This data can be tracked though reports filed by the neutrals, by the motions filed by the parties, or by information entered into the Clerk’s database. (In this context, “outcome” means what happened to the case, such as whether the parties accepted the recommendation of the early neutral evaluator or reached agreement in mediation.)
Reasons cases referred do not end up going through the ADR process: For example, are cases settling before reaching ADR? Are parties not appearing for ADR? This provides an ongoing picture of the effectiveness of the referral system and can be collected from the neutral reports or other forms. Note that for confidential processes, you will need to be careful to maintain anonymity and not report on issues encountered in particular cases.
Participant perception of the ADR process: Participant surveys will help you understand how parties and their attorneys view the services they receive. The answers can be surprising. For example, a common belief is that lenders and their attorneys don’t like participating in foreclosure mediation. By collecting participant surveys, RSI was able to learn that both bank representatives and the banks’ attorneys were highly satisfied with their experience in foreclosure mediation, and generally as satisfied as the homeowners.
How well specific neutrals are doing: Monitoring the neutrals is essential to the quality of the program. One method for doing so is through participant surveys. At least one study has shown that attorney assessment of mediator performance has accurately reflected the mediators' skill levels. Another method is to match outcomes to neutrals. For example, are the awards of particular arbitrators more prone to rejection than others? RSI and the American Bar Association Section of Dispute Resolution have created Model Surveys and an accompanying guide to provide you with the tools to do this for mediators.

You can also track the specific factors that you and your stakeholders have decided are important for the proper functioning of your program. The first step in doing this is to understand what the different stakeholder groups want out of the program. Their perspectives can be very different; judges may want to know that issues they normally decide are being resolved in ADR, while lawyers may want to know that their clients are not being burdened with a time-consuming process and policy-makers may be interested in whether the court backlog is being reduced. By incorporating stakeholder perspectives into your tracking system, you will increase the likelihood that reporting on those measures will lead them to use and/or fund the program in the future.

Below are some examples of program indicators that are specific to particular stakeholder interests.

Amount of funds spent on the program: If funds are being expended, judicial leaders, legislators, policymakers or grant makers will want this information.
Time to disposition: This is particularly important if stakeholders have an interest in ADR not adding time to the overall time from filing to closure, such as in foreclosure cases. This information can be determined from case files or program-specific case management systems.
Who is doing the referring and what is the result of those referrals: This provides data to show which judges are referring cases and how effectively they are doing so. This can help the chief judge and/or program administrator to determine whether any judges seem to have reservations about the program or would benefit from additional education on the ADR process. The necessary information for this could be found in court orders, neutral reports or the Clerk’s case management system.
Amount of time being spent in ADR sessions: If the parties are paying for the ADR process, this can give some information about the amount of resources they are spending on ADR. If staff neutrals are conducting the ADR processes, this information can provide the courts with staff oversight and budgeting information.

Apply This Advice to Program Indicators

When RSI designed a system for tracking a variety of foreclosure mediation programs, one of the more important targets for each program was the percentage of eligible homeowners who participated in the program. This may seem very straightforward, but it turned out to be somewhat complex. We had to determine what it meant to participate in the program. We also wanted to be able to address problems with participation rates by correctly ascertaining where the problem was -- was it in recruitment of homeowners or keeping homeowners in the program once they started it?

What we ended up doing was separating the “participation” target into four variables: 1) how many homeowners contacted the program, 2) how many were provided information helpful to them either through an educational session or an intake phone call, 3) how many “entered” the program by beginning the process of completing their loan modification packet, and 4) how many mediated their case. By doing this, we were able to show how homeowners were helped, even if they dropped out of the program, and determine what the cause of low participation might be.

Step 2: Decide How Program Information Will Be Collected

For program tracking to work, information has to be collected regularly. The two keys to ensure the information you want is collected are to (1) set up a system that makes sense and is easy to use and (2) build in an expectation among court staff and neutrals from the beginning that tracking activities are part of the way the program operates, not an optional activity.

Who will be in charge of the tracking the program?

Someone in your program should be assigned to the job of making sure that all the tracking procedures are being followed and that reports are being generated and distributed regularly. Without someone who is on top of this and who is able to enforce or get enforcement of the program tracking procedures, it is likely that program tracking will fall by the wayside, the necessary forms will not be returned, the data will not be collected, the reports on the program will not be generated and you will not know how your program is doing.

How will the information be obtained?

Some information needed may already be collected in the court’s case management system, or a code might be added for court staff to enter, such as whether a case was referred to ADR. But much of the information you need will be obtained through neutral reports and participant surveys. This includes ADR outcomes and participant experience.

How will you ensure that forms are returned by neutrals and/or attorneys?

The most common data collection problem faced by programs is getting forms from neutrals and participants. Some courts place all responsibility for forms in the hands of the neutrals, with no follow up. This is not usually a good solution. With no follow up, neutrals tend not to complete the reports or distribute surveys to the participants. Others use staff to follow up with the mediators to increase the probability that they will complete their reports, but this requires additional staff time. Some include sanctions in their rules for noncompliance, such as removing mediators from rosters or withholding payment until the report is completed. The latter is often effective. The best way to address this issue, though, is to not let it become a problem. When a program starts with the expectation that all forms will be completed in a timely manner from the very first case, the participants are more likely to comply with expectations.

Apply This Advice For Encouraging Mediators

When RSI trains neutrals for court ADR programs, we model the collection of participant surveys as part of the training. If we are demonstrating a mediation, for example, we have the mediator distribute and collect completed surveys at the end of the demonstration. We also distribute the surveys to neutrals new to the ADR program and explain why this information is important to the continuation of the program.

How will participant surveys be collected?

Some programs ask parties and attorneys to complete surveys after they have left the ADR session. This is done either by handing them the survey, along with a self-addressed envelope, or sending them the survey link via email. Having the participants complete the survey after they have left is considered to be best practice, as it reduces bias and allows the participants time to reflect on their experience in mediation. However, it also has significant drawbacks. Thus, most programs have the participants complete the survey while they are still in the ADR room. This is easier, less expensive, and gives a higher response rate. The drawback to this method is that the parties’ responses may be biased because they are completing the survey at the neutral’s request and while the neutral is nearby. You can promote more candid responses by having the neutral ask the parties to put their surveys into an envelope and seal it. Some courts go further by having the parties put the survey in a secured box or other receptacle.

In what form will data be collected?

In order to properly track your program, you will need a way to aggregate and report on the data. You have a number of options for doing this.

Court staff can create a simple spreadsheet for entering case data: This is best for small programs with few cases and no need to track and manage cases as they move through the program, such as small claims mediations that take place on the day of trial. Information entered would be minimal, such as date of filing, date of mediation and its outcome, as well as survey responses.
- Pros: It’s free and most people are familiar with it.
- Cons: A staff member will need to enter data. More complex operations, like calculating the number of days between dates, and analyzing the data may require a deeper understanding of Excel (or other spreadsheet software) than court staff possess.
Court staff, court IT personnel or a consultant can develop an Access or other type of database: This is best for larger programs with no need to track and manage cases moving through the program.
- Pros: You can create data entry screens that make it easier to enter data. You can associate tables, which makes it easier to maintain data and analyze it. You can create custom reports that are already formatted. You can incorporate streamlined functions (see below).
- Cons: A staff member needs to enter some of the data. If you don’t have someone on staff who knows how to develop the database, you will have to pay a consultant.
You can use commercially available cloud-based case management software. This is best for programs that manage cases as they go through the program.
- Pros: It has already been developed. It has a reporting mechanism, and you can customize more reports (for a fee). You can download data into a spreadsheet or statistical software for more manipulation.
- Cons: These are focused on case management, not on reporting. If your program doesn’t involve the need to manage cases (e.g., cases are referred by a judge to private neutrals, who manage their own cases), this type of software is inappropriate. Even if appropriate, you may need to pay to customize the software to fit your needs.

How will data be entered?

If your program is small and you aren’t collecting a lot of data, it’s possible you can get away with court staff entering all the data. However, you can also streamline data entry by making it more automated. For example:

You can have the neutrals enter their reports online. Depending on the complexity and number of reports you will have, you can use a free or inexpensive online survey program. Or you can customize your cloud-based case management and data tracking system to allow for online reporting.
You can use optical mark recognition (OMR) software to scan and upload participant surveys. Relatively low-cost, good quality options exist.
You can have participants respond to surveys online after an ADR process. This would require either tablets to be used in the ADR office or an email be sent to the participants. Because response rates for emailed surveys tend to be low, your staff would also have to send follow up emails to remind them to complete the survey.
You can have certain case information be uploaded from the Clerk’s software to the program’s software. This would require the court’s IT personnel to work with the developer of your program and, depending on the complexity of the data transfer, can be expensive.

Apply This Advice For Automating Data Entry

When RSI designed the data collection system for our foreclosure mediation programs, we made use of many of these options. We started with a commercially available cloud-based case management system, which we used grant funding to customize for our needs. For one program, we had case information uploaded from the Clerk’s case management system. For another, we uploaded case information from a commercial database, and for a third, we uploaded information from the court’s custom intake system, which homeowners used to enter the program. We also had mediators complete their reports online, while we had participants complete surveys on paper and then used OMR software to read them so no data entry was required. Doing this minimized the amount of time spent collecting data that was beyond what was being entered into the case management system.

Step 3: Decide on Your Reports

Now that you’ve worked to put together a system that will provide you information on your program, the fun part starts: Deciding what your reports will cover. Reports are where all the hard work to collect data will come to fruition, where you will see the information that matters to your program. You can have a single report cover every aspect you want to learn about, or you can create a standard report that is regularly generated, and a more tailored report to answer specific questions or to provide tailored information for particular program stakeholders. This would be put together at more extended intervals.

What will be in the standard reports?

You will want to regularly report on basic information, such as the number of cases referred to ADR (and as a percentage of eligible cases), the number and percentage of cases for which ADR was conducted, and the outcomes of those cases, e.g., agreement, partial agreement, no agreement. You will also want to include information on participant satisfaction. Within this standard report, you will probably report on the results for two or three questions that are most important to you, such as how satisfied participants are with the mediation and whether the mediator treated them fairly.

What could be in tailored reports?

If you want to dig a little deeper into your data, you can compare outcomes by case type, look at procedural justice questions in depth (these are ones that ask whether the parties had a chance to talk about what was important to them, felt others understood them, and felt respected), or explore program timeframes. You may also want to provide stakeholders with information that they think is important. You can read about those in the next section.

Who will receive the reports?

Knowing this will help to determine the data that will be collected and how the report will be written. The basic reports should be distributed to all stakeholder groups. You can tailor your reports to individual groups if you want some information to remain confidential. This is particularly true of reports that focus on individual performance, such as referral rates by judge or settlement rates by neutral.

The following stakeholder groups that should receive reports include:

Referring judges and others in the judiciary, such as supervisory and chief judges, the local court administrator, state court office of ADR, etc: If you want, you can tailor the reports even among these groups. For example, referring judges might want to know what the outcomes are for the cases they refer and supervising judges might want to know which judges are referring cases and how many they are referring, along with the outcomes and party perceptions of individual neutrals.
Neutrals: They will want to know how parties perceive their ADR services, although it is not typical to provide survey results to mediators on an individual basis.
Funders: For them, you can adapt your regular program tracking reports to meet their specific timeframes and interests. You can also identify particularly positive data to highlight in your reports and proposals.
Other program partners: Partners in your program should receive reports as well. For example, in a child protection mediation program you may want to report regularly to the public defender, prosecutor, guardian ad litem, state department of children and families, agencies providing social work services to families, and any other entities who regularly participate in the program. These reports should go to the leaders of these groups – so that they are aware of the profile of the program – and be provided to the staff who participate regularly in the program – so that they can picture the program beyond the particular cases in which they have participated.

Accountability also calls for making reports public. Probably the easiest way to do this is to post them on the program’s or court’s website. As with any public communication, it is probably best to check with the judiciary before publicizing program information.

When will reports be generated?

How often you generate reports depends on the size of your program and to whom you are giving the report(s). In the early stages of a program, you may want to run standard reports on a monthly basis, to later be reduced to quarterly. However, if your quarterly report would only include a few cases, you may want to report every six months or even annually. Tailored reports are generally distributed less often -- annually or biannually.

Who will create reports?

If streamlining options are not viable, you will need to set aside staff time to enter data. Someone will also need to put together program tracking reports on a regular basis.

Step 4: Use the Program Tracking Information

Having put in the effort of designing a program tracking system and collecting the data, you will be well-positioned to use it to strengthen your program and to communicate the state of its health to all program stakeholders.

To strengthen your program, be sure to review the data regularly and act on what it tells you. Use program tracking to help make informed decisions to improve, expand or otherwise change the program so that those who use it get the highest quality services possible. Here are some ideas about how to get started analyzing your data:

Spend some time just scanning the data to see what jumps out at you
Consider the data from the perspective of various leaders and stakeholders; ask yourself what would be important to communicate to them
Share the report informally with someone you know will dig into it and will ask you probing questions
Compare time periods, for example, the current quarter to the previous quarter or the current time period to the same time period from the year before
Get used to the data so you can easily spot anomalies and report on them

Apply This Advice When Looking At Data

RSI noted that one of our three foreclosure mediation programs had a very low participation rate, as compared to our other two programs. We used the information we collected to determine what was keeping homeowners from using the program. After reporting our findings to the court, they authorized changes to the program based on our findings. When we analyzed the data after the changes were made, it was clear that the participation rate had risen significantly.

Once you’ve dug into the data and understand what it means, you will need to create a report that will be read and used. Here are some tips for creating accessible reports:

Always start with the most important information, which would include what the stakeholders most want to know about how the program is doing, any big changes in program performance, any major successes and any major issues
Think of ways to report that are more visual and easier to digest, such as using PowerPoint or a one- or two-page summary with graphs
Attend a meeting and present the report orally

Pitfall to Avoid When Considering Program Tracking

Don't treat program tracking as an afterthought.

If your planning committee treats it this way, no one will give it the attention it needs. As your program rolls out, you will not have the information necessary to support it. Make sure everyone knows that tracking tasks are important, explain why they are important, and make them an intrinsic part of the administrative process.

Conclusion

No matter what the reason for starting a court ADR program or the goals for that program, if a court refers cases to ADR through a structured procedure, it has an inherent responsibility to ensure the quality of service to litigants and to manage that program well. While a case is in the ADR program, all the responsibilities and duties of the justice system to litigants continue. Effective program tracking will provide you with the information to fulfill those responsibilities successfully.