Performance Metrics in Tax Controversy and Litigation

If you’re facing a tax dispute, you want to hire the best possible lawyer. But how are you supposed to evaluate the skills and experience of a particular law firm? How can you gauge the likelihood of a law firm winning your case – and what that effort is going to cost you?

The only way you can objectively compare lawyers and evaluate results is by looking at hard data. But no law firm publishes independently audited performance data.

Until now

We feel so confident about our track record in fighting tax disputes that we hired an independent accounting firm to examine our client success data for an entire year and prepare an assurance report. We are not aware of any other law firm in the world that has ever invited independent accountants to audit its performance data so prospective clients can look at concrete metrics of success. And we’re very proud of the results.

You can’t just claim to be good

For litigation lawyers, evidence is the key. Any assertion you make in court has to be backed up with evidence. But most lawyers simply declare their prowess, without feeling obliged to prove it. For example, it appears that BigLaw is content to rely on vanity reports and brochures containing generic We’re the Best slogans in bold type, alongside pictures of impressive-looking office towers.

Vanity reports do not include much, if any, information to enable clients to evaluate and compare law firms.  Generally speaking, the information is useless to clients.  If anything these glossy brochures and unsubstantiated claims just provide further evidence that the law firms behind them are not, in fact, the transparent client-centric organizations that their marketing departments try to assert.

So why don’t other law firms publish client-centric data and metrics?

That’s a question that preoccupied Guy Kurlandski and Toby Unwin, co-founders of Premonition Analytics, a Florida-based firm that uses data mining and analysis to help clients evaluate and choose the right lawyer. Premonition reinforces a view we’ve long held – that too many clients choose law firms based on advertising, brand perception and untested recommendations, as opposed to the realities of performance. Unwin explains in the video below how flawed this decision-making process tends to be – and how the key to finding the best lawyer is measurable data.

The search for tangible performance data

Premonition is data mining public court records to uncover the lawyers that win the most in a specific type of case and court.  Premonition’s research indicates that there is a strong correlation between wins and the relationship between lawyer and judge.  Premonition argues, on that basis, that clients can increase the probability of success by learning the judge assigned to the case and choosing the lawyer with the best winning percentage in front of that judge. 

Interestingly, Premonition’s research also indicates that there is no correlation between win rate and billing rate.  In other words, Premonition’s data supports what most of us have long contended: hiring BigLaw does not necessarily increase your chances of success.

The search for tangible performance data also preoccupies Reese Arrowsmith, general counsel at Lincoln Financial Group – a Fortune 500 holding company that operates a number of insurance and investment management companies. Arrowsmith doesn’t accept the claim of most law firms that measuring litigation wins was too complicated, and that “every matter is too special”. So he’s spearheading Lincoln’s efforts to compile its own data on law firm results.

In particular, Arrowsmith and his colleagues are testing measurement methods to analyze the total cost of legal spend with the outside firms retained by various subsidiary companies in litigation matters. The goal, he says, is “to turn legal spend metrics into simple math equations.” He adds that the metrics he’s uncovered “won’t apply to every matter, but this is a great start.”

The next step: analyzing litigation outcomes

Premonition’s data and analysis is superior to the current methodology and may provide clients with a great start.   It is clear that Premonition Analytics and Lincoln Financial Group are on the right path leveraging the available data. But to take the next step in understanding lawyer efficacy – to get to a deeper understanding and complete answer – it is important to evaluate more than win percentages. 

Why?  Here are a few reasons.

1. In many cases, tax litigation cases are resolved before the hearing of an appeal and settlements are not published. There is no public data.
   
   
2. Even when litigation proceeds, the decision process – and therefore outcomes – are rarely simple, binary win/lose scenarios. Litigation is nuanced.
   
   
3. Performance analysis must take into account business priorities and the evaluation of diverse probabilities and risks that underlie any decision to pursue litigation, especially in the face of long odds.
   
   
4. Using the win percentage alone does not take legal fees and other litigation costs into account, nor does it measure the most important client metric – clients’ return on investment.
   
   
5. Last, evaluating lawyers based on winning percentages alone ignores that many litigators avoid court, difficult cases, and long odds to cultivate and protect an illusion of success.

We are developing ideas to create a system to set, evaluate, and verify outcomes in tax litigation.  Arrowsmith’s work at Lincoln is on a similar path, and it appears that he is encountering some of the same obstacles.  Arrowsmith has commented that “outcome[s] can be difficult to measure” and “it’s too complex in corporate litigation to say somebody won or lost for us; there are too many nuances. That’s probably why nobody tackles it”. 

We believe that CounterMeasure™ holds part of the answer.  But our work in developing a model to measure the quality of litigation outcomes is still a little way down the road.

Law firm transparency & audited data: the numbers say it all

If law firms want to prove that they are the best choice, they must be willing to pull back the curtain and share measurable results with clients and their advisors.  To complete the dataset – to help clients and third-parties evaluate law firms and outcomes – law firms should disclose audited client-centric data and metrics.  To provide transparency, a complete dataset would include:

1. the firm’s success in hitting the target outcome that the parties identified when litigation began;
   
   
2. quantitative measures of the ultimate settlement results; and
   
   
3. clients’ return on investment.

We encourage prospective clients to compare our firm against the alternatives. We will leave the qualitative statements and vanity brochures to other law firms.  We prefer to give our clients tangible evidence and data.  To that end, we’ve developed unique systems to boost efficiency and maximize results. We’ve created CounterMeasure™, our litigation risk and decision analysis software. We showcase our verifiable expertise and record of success within the competitive landscape. And now we’ve become the first law firm to release independently audited data and performance metrics.

 Here’s how we did it:

To test the quality of our work and results, we tracked, compiled and cleaned our case data, then uploaded it into IBM Watson Analytics to uncover meaningful performance metrics. Applying criteria based on what we’d want in a tax lawyer, we identified four key elements:

1. Proportion of favourable outcomes

A good lawyer may win cases.  But a great lawyer evaluates the client’s goals, patience, risk tolerance, and definition of success, then surveys all available options and pursuing the best course of action for that particular client and case.  We know that working this way achieves the most favourable outcome for a client as often as possible – so that’s what we wanted to measure in our data.

2. Total tax, penalties, and interest saved

The total amount saved quantifies our portfolio size combined with our effectiveness at keeping dollars in our clients’ pockets that would otherwise have gone to the Canada Revenue Agency.

3. Average percentage saved of the total amount in dispute

The primary objective of a tax dispute is to save the maximum possible percentage of the amount in dispute.  We track our clients’ total savings and compare this to the amount in dispute in each matter.  We calculated the savings percentage for each matter and then took the average of all savings’ percentages.

4. Clients’ average return on legal fees

We believe that return on investment (“ROI”) is the most important client metric.  Calculating ROI involves a number of factors, including legal fees, duration of the case, and the reduction of the amounts in dispute. The higher the ROI figure, the more adept we’ve been at weighing options, comparing trade-offs, evaluating costs/benefits and, if necessary, determining the most cost-effective stage at which to settle.

In reviewing Counter’s numbers for our last fiscal year, we calculated ROI by taking the amount of our clients’ total savings and dividing it by the legal fees we invoiced to yield those savings – regardless of whether all of the individual matters across our client base were successful.

We were very happy with the results – but again, we weren’t satisfied to have such an important proof of value come down to our perspective. So we retained Fuller Landau LLP, an independent accounting firm, to audit all of our data, and produce an assurance report.

The accountants confirmed that Counter’s record of success is as solid as we’d determined. And their conclusions reinforce our larger point: when it comes to choosing the right firm to handle your tax dispute, the numbers say it all.