Many legal tax strategies that the wealthy have long used to minimize their taxes are likely to come under new scrutiny by the Internal Revenue Service.
And as a first step in dealing with the new attention, tax experts agreed, the wealthy and slightly less wealthy should keep better records.
President Joe Biden has proposed adding $80 billion to the IRS budget and giving the agency more authority to crack down on tax evasion by high-earners and large corporations. And that was before the reports of how little in taxes the richest Americans paid from 2014 to 2018, which have intensified interest in Congress in the tax code.
“Families in the highest net-worth categories are really at a higher risk of audit now,” said Elizabeth Glasgow, a partner in the private client group at the law firm McDermott, Will & Emery.
This is not to say that the wealthy — and the industry of advisers who work with them — have done anything more than use the tax code to their advantage. But over the past decade, the IRS budget has been falling, and the “amount of funding and staff allocated to enforcement activities has declined by about 30% since 2010,” according to a Congressional Budget Office report last year. So while the IRS uses a computer-matching program to check returns, the number of audits done by humans has fallen precipitously.
But that would change if Biden’s plan is adopted. Some of the additional money in his budget would go toward reviving an underfunded organization within the IRS called the global high-wealth industry group, which focuses on the complicated tax returns filed by the affluent.
The plans to restart the group, better known as the “wealth squad,” have already led the lawyers, accountants and financial advisers who advise on tax planning to talk to their clients about getting their taxes together now and to curtail aggressive tax plans.
Or as Glasgow put it: Wealthy families “are going to need to do an internal family audit to make sure their tax plans are in order.”
Bernie Kent, chairperson of Schechter Investment Advisors in Birmingham, Michigan, who has been a tax adviser for four decades, said he had already begun to talk to clients in certain income brackets and those who have claimed certain deductions.
Some red flags, he said, include having a high income — generally defined as more than $1 million a year — making large charitable contributions, taking a home-office deduction or owning a business that files taxes as a Schedule C, particularly if it shows losses. (Typical Schedule C businesses are structured as sole proprietorships or single-member limited-liability companies.) In fact, significant losses anywhere on a return could bring further scrutiny.
A big risk, even for honest filers, is that they have poor records and can’t back up their claims if they’re audited.
“It may seem like the simplest of answers, but immaculate record-keeping and the implementation of the plan are really going to matter,” Glasgow said. “As beautiful as the paperwork can be, if the implementation of moving money around doesn’t happen with the same degree of formality that it should, then there could be problems.”
For example, a family could put part of their business in a trust for their children, with the stipulation that a certain amount of interest be paid back to the family at a set time. Those steps need to be taken. If only part of the plan was put into effect but then lapsed because it’s a transaction within a family, that could create auditing risks.
The main question to ask, she said, is this: “Is it possible we haven’t implemented this plan in the best way, like we would have if it was an outside party?”
Trying to correct a mistake could attract an auditor’s scrutiny. Christopher Karachale, a partner at the law firm Hanson Bridgett in San Francisco, pointed to a provision in the tax code that allows early employees at startups to exclude from their taxes a portion of their gain from early stock grants. Called a qualified small business stock election, the savings can be substantial in some cases.
But Karachale cautions clients who may have forgotten to claim the stock election and decide to amend their tax return.
“If you’re asking for $50,000 back and you have these other high wages, is it worth it?” he said. “Maybe the IRS approves that $50,000, but they’re going to look at something else. Have you excluded the right amount of other stuff? You’re introducing risk.”
Right now, the key for filers is to make sure their tax return tells a consistent story, he said. If that story changes — say, when a lawyer becomes an equity partner in a law firm and gets different tax documents — then the filer needs to be ready to provide supporting documents to back up the new tax story.
“Changes are going to trigger scrutiny,” he said. “And when there’s a change in the filing profile, that creates risk.”
Regardless of what was done in the past, some practices are going to need to change. As a start, one set of advisers to a wealthy family is going to need to communicate better with the other accountants, lawyers and financial advisers working with the family.
In the area of valuation discounts, this is particularly true. It is common and legal for people who own privately held family businesses to reduce the business’s value when they transfer some part of it to a trust or an heir. This is allowed because it would be hard to sell that stake at its full value to someone outside the family. An accepted discount is 30%.
But that provision has been abused. While an argument can be made for a 30% discount on a manufacturing company, should a portfolio of marketable securities held in a family partnership get the same discount?
“People have used the same discount for years, but now you’re going to see variations on the discounts depending on the assets,” said J. Christopher Cooke, a certified public accountant and lawyer at the Cooke Financial Group. “If I put together a family limited partnership with a 30% or 40% discount in the past, if I did that same partnership tomorrow, it might be a 20% discount.”
Ali Hutchinson, managing director at Brown Brothers Harriman, a private bank, predicted a wave of audits of those discounts. “The IRS has nothing to lose,” she said. “‘You took a 35% discount, we think you should have taken a 21% discount.’ Then it’s just a negotiation of numbers, and not something untoward.”
Another possible area of scrutiny could be transactions that allow investors to defer taxes. These include 1031(b) elections, which allow real estate investors to sell a building and use the assets to buy other buildings without paying taxes on the gains, as well as qualified opportunity zones, which allow the tax gains from any asset to be deferred if the money is invested in designated areas in need of economic growth.
“The IRS is going to look at any activity that involves deferral,” Karachale said. “All of these are where people are playing fast and loose. They’re anxious to get into the box to exclude the tax.”
He said that record-keeping for these transactions, which are legal, is going to be a greater priority. “You should assume you’re going to get audited now,” he said. “What documents do you have to prove your position? It’s really easy to memorialize good documentation if, for example, you sell your company. If you have to go back three years later, it’s really hard to go back and start creating those documents at that time.”
Hutchinson said high net-worth people should assume they will be audited and start getting second opinions on their tax returns. “Have someone who has prepared audited returns be a second set of eyes on your return,” she said. “Just because a small accounting firm is doing the returns doesn’t mean anything is wrong. But practice makes perfect, and there are firms defending 30 audited returns a year versus someone who does one defense every two years.”
It’s better, in other words, to be safe.