Data Provenance

The W3C PROV model in practice

PROV gives a shared vocabulary for the chain behind a value. Applied to a single private-markets data point:

PROV also records times (generatedAtTime, invalidatedAtTime), revisions (wasRevisionOf), quotation (wasQuotedFrom) and a link to the primary source (hadPrimarySource). Using a standard vocabulary means a provenance record can be exchanged between systems instead of living in free-text notes.

What a provenance record holds

For each claim, a usable record holds:

• the source artefact and a stable reference to it (an accession number, a web address (URL), a document title and date);
• a pinpoint inside the artefact (form item, page, table, paragraph);
• the capture: when it was retrieved and a fixed copy or a hash of what was retrieved. The Berkeley Protocol expects a time stamp from an accurate clock and a hash value that shows the copy has not changed since collection (para. 155);
• how the value was read: manual extraction, a parser, a model, with its version;
• any rule or calculation applied to it, with the rule version;
• the dates that apply to the fact and the evidence (see evidence dates);
• every later change, check or correction, with who made it and when.

Granularity matters. Provenance kept at record level ("profile sourced from X, updated in June") cannot tell a reader which source supports which field.

Provenance, lineage, attribution, audit trail

These terms overlap and are often confused:

• Provenance covers the origin and production history of a data item, including sources outside any one system.
• Data lineage traces how data moves and is transformed through systems and pipelines.
• Source attribution is the citation shown to a reader: who said it.
• Audit trail is the chronological record of who changed what, and when.
• Evidence chain is the path from a published claim back through intermediate sources to the earliest reachable origin.

Attribution without the rest is a footnote; provenance without attribution is a record a reader cannot use.

Limits of provenance

• It shows where a claim came from, not that the claim is true.
• A captured copy shows what an artefact said when it was captured, not what it said before or after.
• For online material, the origin can be uncertain: the Berkeley Protocol notes that an item may have circulated through private channels before its first public appearance, which is why it speaks of the "first copy found online" (para. 178).
• Each intermediate link in a chain can lose or alter information; the further the cited source is from the origin, the more the claim depends on others having copied it correctly.

How LPs, GPs and analysts use provenance

An investment committee memo that cites a manager's regulatory AUM, a benchmark figure or an LP's prior commitments has to answer "how do we know this, and as of when?". Provenance answers it without rework. In operational due diligence of a data provider or administrator, the questions are whether each field links to an artefact, whether captures are kept, and whether corrections preserve history. Analysts also use provenance to find systematic errors: one misread form item can propagate into thousands of records, and only a record of the extraction method shows which ones.

How Altss applies this (Altss methodology)

Altss records provenance along three separate dimensions, defined in its own evidence model and not presented as an industry standard:

• Evidence origin, per evidence item (the link between a claim and an artefact): REGULATORY_PUBLIC_RECORD, OFFICIAL_INSTITUTIONAL, DISCLOSED, OSINT_SOURCED or LICENSED_THIRD_PARTY.
• Derivation status, per value: OBSERVED (stated in a source; needs at least one evidence item with a pinpoint), DERIVED (computed by a stated deterministic rule; needs the input claims and the rule version) or ESTIMATED (inferred; needs the method, inputs and where possible a range).
• Validation status, per claim and with a date: UNVERIFIED, CORROBORATED, RESEARCH_VALIDATED or CONFLICTING.

A derived or estimated value has no origin of its own; its origin is the set of origins of its inputs. A claim with several evidence items has several origins, and a display lists them rather than silently choosing one. The retired labels "Filed" and "Reported" are now expressed as OBSERVED + REGULATORY_PUBLIC_RECORD and OBSERVED + DISCLOSED (or OBSERVED + OFFICIAL_INSTITUTIONAL). The full rules are in the Evidence & Provenance Standard.

Not the same as

• Primary Source: A primary source is one kind of artefact a provenance record can point to; provenance is the record of the whole chain, whether or not it reaches a primary source.
• Source Reliability: Reliability grades how dependable a source has proven to be; provenance records which source a value came from and how it was produced, without grading it.
• Evidence Dates: Evidence dates are the dated fields inside a provenance record; provenance is the whole record of origin and production.

Common mistakes

• Treating a source name in a footnote as full provenance.
• Keeping one record-level source or timestamp for every field.
• Recording the collection channel (a feed, a monitoring tool) instead of the artefact that states the claim.
• Equating good provenance with accuracy.
• Overwriting a value on update, so the earlier value and its source are lost.
• Citing an aggregator profile instead of the filing or report it copied.

Edge cases

• A derived value (a ratio, a classification under a published rule) carries the provenance of each input and the version of the rule, not a source of its own.
• When a source is restated, both versions and their dates are kept; the restatement is recorded as such.
• A page that is taken down remains citable through its captured copy, labelled with the capture date.
• When two records are merged or unmerged, each claim keeps the provenance it had; the merge is itself recorded with its justification.

Questions

External standards

Sources

1. PROV-O: The PROV Ontology. Timothy Lebo; Satya Sahoo; Deborah McGuinness (eds.), W3C, W3C Recommendation, 30 April 2013. Status: Current Recommendation (checked 2026-10-01). Sec. 4.1 (starting-point classes and properties); Sec. 4.2 (generatedAtTime, invalidatedAtTime, wasRevisionOf, wasQuotedFrom, hadPrimarySource) — supports: Formal provenance model and vocabulary
2. Berkeley Protocol on Digital Open Source Investigations. OHCHR; Human Rights Center, UC Berkeley School of Law, United Nations (OHCHR) and UC Berkeley, HR/PUB/20/2; launched 2 December 2020; printed edition (c) 2022 United Nations. Status: Current (checked 2026-10-01). Paras. 155 (collection data, hash value), 167 (chain of custody), 178 (provenance) — supports: What a collection record contains; chain of custody; uncertainty of online provenance
3. Intelligence Community Directive 203: Analytic Standards. Office of the Director of National Intelligence, ODNI, Signed 2 January 2015; technical amendments incl. 21 January 2022. Status: In force (as amended) (checked 2026-10-01). Section D.6.e(1) — supports: Analytic products identify the underlying sources and methodologies on which judgments are based

• Primary Source
• Evidence Dates
• Source Reliability
• Triangulation (Source Triangulation)
• Confidence Score
• Open-Source Intelligence

• Bitemporal Data
• Data Quality Dimensions
• Point-in-Time Data