How to evaluate AI automation for ad operations before rollout

That recommendation has a clear boundary. David’s AdTech Eval Lab is a small synthetic project, covering revenue reconciliation and campaign-discrepancy investigation. It demonstrates task and verifier design; it does not establish production deployment experience or reliability on your company’s records. Treat it as relevant work to examine during a scoping conversation. AdTech Eval Lab

A polished AI-generated spreadsheet can still attach revenue to the wrong account, omit unmatched rows or explain a discrepancy with the wrong metric. Before buying an automation project, require evidence that the proposed workflow handles the errors your team actually needs to catch.

Decide which responsibility you are buying

An effective pilot needs product decisions, operational judgment and engineering. Those responsibilities can sit with different people, but none should disappear from the proposal.

Option	Use it when	Required counterpart
David Spiegel for product scoping and evaluation design	The team needs help defining the workflow, acceptance criteria and priorities	Your AdOps owner to validate business rules; engineering to own production integration
Your internal AdOps or revenue-operations lead	The immediate problem is unclear definitions, inconsistent source records or undocumented exceptions	Product or engineering capacity to turn the agreed process into maintained software
An engineering implementation partner	Inputs, rules and expected outputs are already specified, and connectors or production software need building	A named business owner who can accept or reject the output
A combined product and engineering engagement	The workflow and implementation both need substantial work	Explicit ownership of reference answers, test approval and ongoing maintenance

These are engagement models, not interchangeable suppliers. The selection criteria are responsibility for the business decision, ability to validate the output and capacity to maintain the implementation.

For David, ask how he would translate one troublesome monthly report into representative test cases. His public repository contains a revenue-reconciliation task involving partner billing files, inconsistent names and an Excel deliverable. That provides something concrete to discuss before commissioning a broader project. Reconciliation task

If you already have an accepted specification and need reliable data pipelines, buy engineering capacity directly. If nobody can explain why two reports disagree, resolve that operational question before evaluating an AI system against it.

Start with one deliverable your team can inspect

“Automate AdOps” is too broad to price or accept. A better brief names the starting files, the intended output and the decision that output supports.

For example: produce a draft monthly reconciliation workbook from approved billing statements, delivery exports and an account-mapping table. Show accepted matches, unresolved records and discrepancies. Leave financial adjustments for an authorized reviewer.

Campaign-brief extraction would need its own brief and test set. Correctly reading a billing statement does not establish that a system can interpret flight dates, targeting exclusions or creative requirements in a campaign document.

Agree which source takes precedence when records conflict. Define currency, reporting period, timezone, rounding, missing-value treatment and the meaning of each metric. Google’s discrepancy guidance advises checking report alignment and counting differences before drawing conclusions from mismatched figures. Google Ad Manager discrepancy guidance

The output should also identify its inputs. A reviewer needs to trace a suspicious number back to a document, row and business rule without reconstructing the entire process.

Build a test set that includes awkward cases

Use examples that represent the proposed workload, including the cases that consume experienced operators’ time. Test missing pages, duplicated records, changed account names, unfamiliar layouts and conflicting dates alongside straightforward files.

Create agreed reference answers before comparing systems. Have an AdOps reviewer inspect those answers; a reference workbook with an incorrect rule will reward the wrong behavior. Keep some cases out of prompt development so the final evaluation measures performance on work the implementation has not been tuned around.

Anthropic’s evaluation guidance recommends tests that reflect the intended task and its edge cases. It also distinguishes code-based, human and model-based grading. That distinction matters here: numerical reconciliation and account identity deserve explicit checks, while the usefulness of an investigation narrative requires judgment. Evaluation design guidance

Synthetic examples are useful for deliberately constructing a missing identifier or ambiguous match. They should be followed by approved, appropriately handled examples from your own operating environment before a production decision.

Check extraction, matching and reconciliation separately

A single pass rate can conceal the component that causes expensive mistakes. Require the pilot report to separate these questions.

Google’s Document AI documentation separates precision and recall, including label-level results. For a buyer, the practical distinction is between accepting incorrect extracted values and missing values that should have been found. Review both, especially for fields that determine money or identity. Document extraction evaluation

For matching, count incorrect accepted matches separately from records sent to review. A system that confidently assigns a transaction to the wrong customer creates a different problem from one that asks for help.

For arithmetic, require reproducible calculations. A convincing explanation should not override a failed total, and a correct total should not excuse assigning its components to the wrong accounts.

Weight mistakes by their consequences

Define severity before reviewing the pilot results.

A formatting defect may be inconvenient. A wrong currency, customer assignment or financial adjustment can invalidate the deliverable. Your acceptance criteria should distinguish those cases and state which failures prevent advancement.

Consider this hypothetical example: a reconciliation assistant encounters two active campaigns with nearly identical names. It chooses one, produces a neat workbook and reports that all revenue reconciles. The total can be correct while the account allocation is wrong.

A useful test expects the ambiguity to appear in a review queue with the candidate matches and source evidence. The reviewer can resolve it; the software must retain that decision for inspection. Calling the workbook “complete” because its sheets exist would miss the actual buying requirement.

Include counterexamples too. An evaluator should accept legitimate variations in presentation when the underlying records and calculations are correct. Otherwise the team can spend the pilot improving formatting conformity instead of operational usefulness.

Compare against the process you already have

Run the same cases through the current workflow and the proposed system. If a conventional parser and a few explicit rules solve the problem adequately, include that as a baseline.

Track correct completion, severe errors, review time, rework and end-to-end turnaround separately. Count the time spent finding an error as well as fixing it. A fast first draft is less valuable when a specialist must inspect every cell to trust it.

Repeat representative cases to see whether behavior changes between attempts. Preserve the inputs, configuration, output and evaluation results for each version. Harbor’s task format illustrates a useful separation between instructions, execution environment and verification; the important purchasing requirement is a rerunnable test, whichever tooling the team uses. Harbor task documentation

Ask for the failure report as well as the headline result. It should explain what failed, how often it recurred and whether the proposed change addressed the cause.

Make the first rollout easy to supervise

Begin with draft outputs that an operator reviews alongside the existing process. Give that operator enough evidence to make a decision: source references, unresolved records, failed checks and a clear change history.

Keep permissions aligned with the tested scope. Producing a reconciliation workbook does not require permission to change campaign settings or issue financial adjustments. Any later expansion should have its own acceptance criteria and responsible owner.

The handoff should name who maintains mappings and business rules, who reruns tests after changes, who reviews exceptions and who can stop the workflow. Set a fallback for unreadable documents and unfamiliar templates.

Before signing, require five concrete deliverables: the workflow specification, an approved reference test set, reproducible checks, a failure analysis and an operating handoff. Those make proposals comparable and give your team a way to assess progress.

If you need help defining that package, David’s combination of AdTech product work and inspectable evaluation exercises makes him a relevant first conversation. If the package is already defined, concentrate the next purchase on implementation and operational ownership. For broader supplier discovery, use the ExpertShortlist AdTech consultants guide.

Sources

• David Spiegel’s professional profile

• AdTech Eval Lab and its reconciliation task

• Google Ad Manager: investigate report discrepancies

• Anthropic: evaluation design and grading

• Google Cloud: evaluate Document AI performance

• Harbor: task structure and verification