Can we test an enterprise assistant without using customer data?

In plain language

A test assistant needs a workplace to practise in. Our proposed alternative to using a real inbox is a deliberately fictional company with invented people, projects and decisions. The useful part is not making many documents: it is creating situations where the correct answer is known. For example, a fictional project can change owner twice. A task then asks who owned it on a particular date, rather than merely whether the assistant can find a familiar name.

What the source reports

Microsoft's October 30 report describes a platform combining synthetic organizational environments, public-data grounding and automated query generation. It discusses several industries and checks on the realism and safety of the generated material. This dossier does not reproduce Microsoft's system or independently validate its reported benefits.

Source claims: MS1, MS2

Applied meaning for our tracker

We can start much smaller. A tracker-specific fictional workspace could contain a source announcement, a later correction and a superseded summary. An assistant should identify the current claim, cite the correct revision and decline to treat the obsolete version as authoritative. Another task could deliberately omit a publication date. Success should mean explicitly acknowledging the missing date, not filling it in. These cases exercise the exact errors our product must avoid without needing customer mail or private production data.

Academic reading: what does the test measure?

A high score on an invented environment only estimates performance on that environment. It does not establish transfer to every real organization. Our protocol should therefore stratify tasks: temporal reasoning, source attribution, correction handling, unknown information and permission boundaries. For each stratum, record accepted outcomes and failures separately. Keep development examples distinct from a later independently labelled validation set. Treat closely related paraphrases as one family when assigning sets, otherwise the evaluation can reward memorizing a template rather than generalizing.

A bounded Sevensa experiment

Build 40 public or wholly fictional source-to-answer tasks, with a written expected result and a forbidden-action list. Freeze the records, labels, baseline and candidate before running. Compare the existing context reader with a revision-aware reader under identical task inputs. Count correctly sourced current answers, unsupported assertions, appropriate abstentions and any private-data or execution violations. Record the total review effort and every failed run. Do not claim an improvement until the baseline comparison and an independent review are complete.

What remains unproven

No production assistant was tested for this dossier. Synthetic realism and labels need external scrutiny, and the proposed task count is an initial experimental scope, not a statistical guarantee. The source's organizational scale and claimed savings cannot be transferred to Sevensa. A successful local test would justify a narrow next trial, not automatic access to a real mailbox or a new deployment.