What this does and does not claim
Claimed on every commit, by tests/test_record_refset_manifests.py in a tier with no torch and no network:
- Three manifests whose every
content_sha256andpixel_sha256recomputes from the committed image bytes. - Three
set_manifest_sha256values that recompute from the manifests and are found in this file. - The three
space_idvalues, againstverd.space_cache.SPACE_IDS, so a space the pipeline can load with no set published for it fails the suite.
Claimed by --check, which regenerates: ten committed images whose bytes are exactly what the closed-form generator draws.
Claimed by measurement, and re-checkable on demand: thirty vector_sha256 values, each the digest of a vector that was actually produced by the named checkpoint. Not one is illustrative — errata E26 is the item that catalogues shipping placeholder digests as though they were real, and a reference set of invented vectors would repeat that defect at the one place §8.2 makes load-bearing. CI cannot afford ~1.5 GB of weights, so it re-checks none of them; --check --vectors re-checks all thirty, and the test suite re-checks three (one per space) wherever torch is installed.
Not claimed. These manifests make set_manifest_sha256 satisfiable. They do not by themselves make a conformance claim, because a conformance claim is made by a space descriptor, not by a manifest:
- §8.2's tolerances —
tolerance_cosine,tolerance_cosine_p5,tolerance_cosine_worst,tolerance_hamming_max— are **declared by aspaceDescriptor.referenceblock**. Nothing in this directory declares one, and nothing in this directory can: a tolerance is a producer's assertion about its own reproducibility, and the manifest is the evidence that assertion is measured against. - This project's pipeline does not publish reference-bearing descriptors.
verd.models.SpaceDescriptor.to_json()has noreferencekey at all, so every Recordverdcurrently emits warnsVER404(space-reference-missing), and errata E13 rates such a descriptor as making no conformance claim. Wiringreferenceinto the descriptor —set_uri, theset_manifest_sha256values above, and tolerances measured by re-embedding this set across the runs a producer wants to claim reproducibility over — is follow-up work that has not been done, not a property this directory confers. - No tolerance number appears anywhere in this directory, deliberately. The cosine figures above are *sensitivity measurements of one implementation against itself* over thirty vectors. A
tolerance_cosineis a floor a producer is prepared to be held to across checkpoint reloads, library upgrades and hardware — an empirical corpusRELEASE.mdlists as its own outstanding ratification prerequisite. Publishing 0.9999999 as a tolerance on the strength of this run would be exactly the "three round figures with a coherence argument and no data" the draft criticises.
