SiSu RevOps

M4  /  L4.3  /  about 16 minutes

Evidence cheaper than the work

Four hundred rows came back. You are not going to read four hundred rows, and if you were, you would have done the job by hand.

So you scroll, spot two rows that look right, and decide the whole thing is fine. That is not checking. It is reassurance with a keyboard, and it fails exactly where it matters: the plausible-wrong output is the one that survives scrolling.

Verification is a design problem with a budget.

What you will be able to do

  • Design a verification with a named source of truth, a stated sample size and at least one reconciliation.
  • Decide when verification cost alone should stop you handing a task over.
  • Read an exceptions file as a signal about your brief, not about the data.

The governing rule

If checking costs more than doing it by hand, do not hand it over.

That rule excludes real workflows. Some tasks are faster to do than to verify, and the correct decision is to keep them. Saying so keeps every time-saved claim in this course honest.

It also means verification cost sits inside any time-saved number: a workflow that saves 40 minutes and adds 35 of checking saves 5, and you write 5.

And the procedure is written before the run. A check designed after you have seen the output is justification, not verification: you know what the answer says, and you will design a check it passes.

Name one workflow you were about to hand over that fails on verification cost.

Three cheap kinds of evidence

Three instruments. Use at least two. They catch different things.

Reconciliation. Totals in the output must agree with totals in the source: row counts, sums of amounts, distinct counts of accounts or opportunities. This catches silent drops, the most common failure and the hardest to see, because a list missing 40 of 400 rows looks exactly like a list. Cost: seconds. If you do only one thing, do this one.

Sampling. Pull n rows and check them by hand against the source. State n before the run. A rule of thumb, not a standard:

  • at least 10 rows, always;
  • at least 5 percent of rows for anything draft-only;
  • every row in the top decile by amount for anything that touches a number a VP will read, because the distribution of consequences is not the distribution of rows.

Sampling catches what reconciliation cannot: rows that are present and wrong. A total can be right while individual rows are misclassified.

Tripwires: known-answer rows. Rows whose correct treatment you already know, planted in the input, then checked in the output. On your own data you plant them yourself; that is the habit.

On the shipped Talvera data you do not need to plant any: documented rows with a known correct treatment already exist. The four opportunity ids that appear twice with different amounts are OPP-006790, OPP-006886, OPP-007099 and OPP-007110 (answer-keys/PLANTED-ISSUES.md in your pack, which you open only after attempting the lab, class duplicate-opportunity-rows). Each must be counted once, and any roll-up that counts one twice has failed a check you could have designed in advance. That is all a tripwire is: a row where you know the answer before you look.

The exceptions file

One instruction. It is the highest-yield habit in this course.

Anything you cannot resolve goes to exceptions.csv with the row identifier and
the reason. Do not guess.

Without it, unresolvable rows do not disappear. They get resolved anyway, by something that had to pick, and the picking is invisible: a guessed value looks exactly like a known one in a CSV. Then read the file, because it is diagnostic:

  • An empty exceptions file on messy data is a warning sign, not a success. The shipped dataset carries 320 documented defect records across 23 classes (PLANTED-ISSUES.md). A pass that resolves everything either did not look or did not tell you.
  • A long exceptions file usually means the brief was underspecified, not that the data was bad. Read five rows. If they share a shape, that shape is a rule you did not write. "Cannot determine currency" forty times is one missing sentence in your Constraints, not forty problems.
  • Exceptions are never failures, not in this course and not in your own head. A run that flags 30 rows it could not resolve did better work than one that silently resolved all 30, and the second looks better on the page. Rewarding the second is how organisations end up with confident wrong numbers.

The checklist, written first, and what it caught

The checklist for the account snapshot, written before the run:

Source of truth: 10-data/accounts.csv and 10-data/opportunities.csv, as shipped.
Reconciliation 1: row count of opportunities for this account_id, against the
                  distinct count of opportunity_id. State both.
Reconciliation 2: sum of Closed Won New Business and Expansion amounts, in the
                  account's currency, against accounts.arr_usd converted at the
                  documented fixed rate. Renewal amounts excluded: they restate
                  ARR (DATA-DICTIONARY.md).
Sample:           all rows. This account has 4, so the sample is the population.
                  On a 400-row output the sample would be 10 rows plus the top
                  decile by amount.
Tripwire:         OPP-007099 is a known duplicate id. It must appear once in any
                  count and its conflict must appear in exceptions.csv.
Cost:             about 3 minutes.

Why reconciliation 2 has a clause in it: without the exclusion, the check itself is wrong. Three of the four rows on ACC-02848 are type = Renewal, and renewal amounts restate current ARR. A check that sums everything against ARR fails on a correct output, which is worse than no check: you will then "fix" a correct output.

Why a sample size appears on a four-row account: the checklist is a template for accounts with forty rows, and the sample rule must exist before it is needed.

What it caught, on the run from Lesson 1. Reconciliation 1: 4 rows, 3 distinct ids, disagreement flagged. Tripwire: OPP-007099 appeared once and its conflict appeared in exceptions.csv with both amounts. Reconciliation 2: the single New Business amount, 55,000 USD, matched accounts.arr_usd of 55,000 exactly.

What it did not catch: which OPP-007099 amount is right. Nothing in the data settles it. ARR agrees with the 55,000 row, and that row has the blank close date. The checklist's job was to make sure I found out there was a question. Answering it means asking Ursula Ulrich, who owns the opportunity. A good verification produces questions for humans. A bad one produces confidence.

Consistency check or verification: tell them apart

The most expensive confusion in the course, because a consistency check feels rigorous. Numbers were compared. Something agreed with something. The something was itself.

A consistency check compares the output against itself. A verification compares the output against the source. An output that is internally consistent can be entirely wrong, and it will look better than a correct output that raises a question.

Example A: a consistency check

The snapshot for ACC-02848 prints:

Opportunities: 4
Total value:   234,000 USD
Average:       58,500 USD

Check it: 55,000 plus 55,000 plus 69,000 plus 55,000 is 234,000, and 234,000 divided by 4 is 58,500. Every number agrees with every other number on the page.

Two things are wrong, and neither is visible from the page.

  • It counts one opportunity twice. OPP-007099 appears on two rows (PLANTED-ISSUES.md, class duplicate-opportunity-rows). There are three opportunities, not four.
  • It adds renewals to new business. Three of the four rows are type = Renewal, and renewal amounts restate current ARR rather than adding to it (DATA-DICTIONARY.md in your pack's data folder). So 234,000 is not the total of anything: not lifetime value, not ARR, not bookings.

And the account's recorded ARR is 55,000 USD. The page is out by a factor of more than four, and internally perfect.

Example B: a verification

Same output, three comparisons against source:

  1. Row count against distinct identifier count, from opportunities.csv: 4 against 3. They disagree. A source comparison, because the distinct count is a property of the file.
  2. Sum of Closed Won New Business and Expansion amounts against accounts.arr_usd: 55,000 against 55,000. Agrees, and only because the check names which types go into the sum.
  3. Tripwire: OPP-007099 is a known duplicate. Does it appear once, and is the conflict in exceptions.csv?

Check 1 fails, and that is the check doing its job. The corrected output says: three opportunities; current ARR 55,000 USD as recorded on the account; one unresolved conflict on OPP-007099, two amounts, one blank close date, routed to exceptions. That output is uglier than the wrong one, and it is the only one you can put your name on.

What good feedback on this looks like

A learner submits the pass-1 roll-up with the note: "verified: the amounts sum to the stated total and the count matches the number of rows returned." The response:

You ran a check and it passed. The problem is what it was a check of. You compared three numbers in your output against each other. That is a consistency check. It proves the arithmetic on the page is sound, and it cannot detect any error that entered before the arithmetic started.

You never compared anything against opportunities.csv or accounts.csv. Two comparisons would have taken ninety seconds. Row count against distinct opportunity_id count: four rows, three distinct ids, because OPP-007099 appears twice with different amounts and one blank close_date, both documented, in classes duplicate-opportunity-rows and missing-values. Your "4 opportunities" is a row count wearing an opportunity count's label. And your total against accounts.arr_usd: the account records 55,000 USD, you printed 234,000. Not a rounding disagreement, a category error: three of your four rows are renewals, and renewal amounts restate ARR.

Why this matters more than the specific error: a consistency check almost always passes, because one process produced the output, so its parts agree by construction. "Verified" after a check that cannot fail transfers your confidence without any evidence, and the next person reads it and stops looking.

The repair is one line in your checklist, written before the run: name the file and column each output number will be compared against. A number with no named source is either derived from numbers that have one, and you say from which, or it does not go in the output.

One thing you got right: you stated what you checked. Most people say "verified" and stop. Because you wrote it down, this conversation is possible. Keep the habit, change what goes in the sentence.

What you keep

revenue-workspace/40-reference/verification-checklist.md, a file you write yourself (template: templates/M4/verification-checklist.md in your practice pack), written before the next run, containing:

  • the named source of truth, by file and column;
  • at least one reconciliation, with the clause that makes it meaningful on your data;
  • a stated sample size and how it was chosen;
  • at least one tripwire row and its known correct treatment;
  • the estimated cost in minutes, which goes inside your time-saved number.

Plus exceptions.csv from your runs, kept rather than deleted: the evidence of what your brief does not yet handle.

Check yourself

  1. Three verification procedures are proposed for a 400-row hygiene pass: (a) re-run the pass with a second brief and compare the outputs; (b) reconcile row counts and sums against source, then sample 20 rows including the top decile by amount; (c) read every row. Pick the one that is sufficient and cheaper than the work, then say why (c) is wrong and why (a) is worse than it looks.
  2. Your exceptions file has zero rows after a pass over the shipped accounts data. Name the two most likely explanations and the first thing you would check.
  3. A run reports "totals reconciled". Write the one question that tells you whether that was a verification or a consistency check.
  4. Your workflow saves 40 minutes and its verification takes 35. What number goes in your delegation policy, and what would have to change for the workflow to be worth keeping?
Answers

1. (b). One line per option:

  • (a) Re-run and compare: wrong, and worse than it looks. Two outputs agreeing is a consistency check with extra steps. Both runs read the same source the same way, so they can agree and both be wrong, at the cost of a second full run.
  • (b) Reconcile against source, then sample 20 rows including the top decile: correct. Two instruments, both comparing against source, minutes against a 400-row job. The top decile is there because the distribution of consequences is not the distribution of rows.
  • (c) Read every row: sufficient and wrong. More expensive than doing the job by hand, which is the rule that overrides thoroughness. A check nobody performs twice is not a check.

2. Either the exceptions rule was not in force, or the scope was narrow enough that nothing ambiguous fell inside it. Check whether an in-scope ambiguous case was resolved silently.

3. "Which file and column was each number compared against?" No answer means nothing was compared against source.

4. You write 5. To make it worth keeping, either cut the verification cost (a reconciliation plus a sample rather than a full read) or raise the baseline by running it over more rows per pass.

Where this goes next

Next lesson: what to do while a run is going wrong in front of you, and the record that lets you answer "where did this number come from" six weeks later.

Track labs: the exercise sheet in labs/LAB-04 of your practice pack (start with its README). Part A pass 3 requires you to write the checklist, run the reconciliation and a 10-row sample against source, find the discrepancy, and record it. Every findings-list lab from Module 5 onward is scored on detection.