Judge Call Sources by Policies That Last, Not Day-One Sales

Two sources each sent you 20 calls last week. Source A's calls produced six sales. Source B's produced two. The obvious move is to raise A's bid and cut B's.

It may be the wrong move. The week's sales are a small, early reading of a slow outcome. This piece covers three ways that reading misleads, and what it takes to judge sources by the result that counts. Figures in the examples are illustrative.

A same-day sale is an early reading, not the result

For a Medicare agency, a sale logged on the day of the call starts a chain. The application has to go through, the policy has to take effect, and the member has to keep it. The sale tells you a source can produce callers who say yes. It doesn't tell you the policy took effect, or that it was still there three months later.

Andrew Grove met the same kind of lag reviewing managers at Intel. He once rated a manager highly on strong output figures, then found the figures reflected work from years before:

“

"The output indicators merely represented work done years ago—the light from distant stars, as it were—which was still holding up." — Andrew S. Grove, High Output Management (p. 192)

Call sources have their own version. The sale is visible today; the outcome it stands for arrives months later. A source can look strong for a whole quarter on sales whose policies are already falling away.

Small sources swing the most, so their best weeks mislead

Twenty calls is a small sample. With so few, chance alone moves the rate a long way. Daniel Kahneman shows this with kidney cancer rates by county: the counties with the lowest rates and those with the highest were both mostly small and rural. Size, not cause, put them at the extremes.

“

"Small samples yield extreme results more often than large samples do." — Daniel Kahneman, Thinking, Fast and Slow (ch. 10, "The Law of Small Numbers")

Take a source whose callers buy at a steady 15%, sending 20 calls a week. Nothing about the source changes, yet its weekly rate could easily look like this:

Give a second source the same 15% and 400 calls a week, and its weekly rate would rarely stray more than a few points from 15. The only difference between the two is the sample size. Source A's six sales in 20 calls, a 30% week, sits inside the range chance alone produces. It says less about the source than it seems to.

One bad week does not prove a source got worse

Kahneman tells of a flight instructor who found that praise after a smooth landing was usually followed by a worse one, and harsh words after a rough landing by a better one. He concluded that criticism works and praise doesn't. Kahneman's verdict:

“

"The instructor had attached a causal interpretation to the inevitable fluctuations of a random process." — Daniel Kahneman, Thinking, Fast and Slow (ch. 17, "Regression to the Mean")

Call buyers can do the same with bids. Source B has a bad week, you cut its bid, and the next week it does better. The cut seems to have worked. If the source's traffic hasn't changed, a likelier story is that it returned to its normal rate, with or without the cut. The reverse holds too: raise a bid after a source's best week, and its next week will often look worse.

Sources do get worse for real. Traffic changes, a partner adds a new sub-source, the season turns. A drop that lasts for several weeks, or that shows up in a large sample, deserves action. A single week that triggered a change is a poor yardstick for judging that change.

Retention separates sources that sell from sources whose sales last

Even with enough calls, same-day sales can rank sources in the wrong order. Some callers agree on the call and the policy never takes effect. Some policies take effect and are gone within weeks. If those cases cluster in one source, its sales count overstates it.

The two measures do different jobs. Same-day numbers catch a source that breaks: days with no sales at all, or a run of short calls. Ninety-day retention tells you what a source is worth. The loop with a gate covers fast feedback on CPA; retention is the slower check that sits behind it. Grove's point about managers applies to sources too:

“

"As managers, we are really called upon to judge performance, not just to see and record it when it's in plain sight." — Andrew S. Grove, High Output Management (p. 192)

A retention figure is only as good as its matched outcomes

To count a policy in force at 90 days against the source that sent it, you need the call, the sale, the policy and its effective date on one record. Each lives in a different system and arrives at a different time. One record per caller walks through how TapQuality keeps the click, the call and the sale together.

Two things can still go wrong, and both bias a source's numbers.

The second problem is time. A cohort of calls from last month can't have 90-day retention yet, because most of its policies haven't been in force for 90 days. Mixing young and old cohorts makes the young ones look better or worse at random, depending on what has been counted so far.

TapQuality's Buyer Insights joins call data to your dialer, CRM, appointments and enrollment records. It reports policies still in force 90 days after they take effect, and sales that never took effect, by source. Your outcomes return to TapQuality every hour and attach to the caller. Buyer Insights isn't self-serve: we connect your records with you, then open access by role. It is in production with a Medicare agency.

The Buyer Insights page shows the funnel from landing page through number lease, call, transfer and sale to 90-day retention, with illustrative data. The film at the top of How it works follows one click through the form, tracking number, voice agent and transfer to the sale.

Source reviews need equally mature cohorts

A weekly source review can stay weekly. It just has to know which numbers are ready.

• Put the call count next to every rate. A rate without a count invites a story.
• Set a minimum number of calls before a source's rate moves its bid.
• Compare cohorts of the same age. Judge sources on calls from the same weeks, measured the same number of days after the policy took effect.
• Judge a bid change over several weeks, not against the week that set it off.
• Check each figure's data date and the unmatched count before acting on it.
• Keep same-day numbers for alarms and 90-day retention for value.

Why sources know things about each call that buyers can't see at the bid is the subject of Pay-per-call's lemons problem. To see your own sources judged by what lasts, book a walkthrough.