1# ODF validation harness 2 3This harness has OpenOffice save a corpus of documents as ODF, checks every 4saved package against the OASIS ODF 1.2, ODF 1.3 and ODF 1.4 RELAX NG schemas with 5jing, and reports the errors grouped by kind. Comparing those reports against 6a baseline shows whether an export change fixed errors or introduced new ones. 7 8It runs on macOS only (zsh scripts, `hdiutil`, `.app` layout). 9 10## Requirements 11 12- An AOO macOS build (`.dmg`). 13- The Python that the office's pyuno links against. Check with 14 `otool -L <app>/Contents/program/libpyuno.dylib`, then set `ODFV_PYTHON`. 15- `jing.jar` (set `ODFV_JING`) and `java`; `ODFV_ODFVALIDATOR` and `ODFV_JAVA` for `crosscheck.py`. 16- `gtimeout` (GNU coreutils), used by `drive.sh`. 17 18Defaults for `ODFV_APP`, `ODFV_PYTHON`, `ODFV_JING` and `ODFV_SOLVER` are in 19`env.sh`. 20 21## Setup 22 23 ./fetch-schemas.sh # OASIS schemas, checksum-verified, into schema/ 24 ./setup-app.sh <path-to-AOO.dmg> # copies OpenOffice.app into app/ 25 ./fetch-odfvalidator.sh # optional: odfvalidator for crosscheck.py 26 python3 -I selftest.py # validator negative controls 27 28The schemas are not committed because they carry OASIS copyright terms. 29 30## Running a corpus 31 32 ./make-jobs.sh runs/base_strict/out > runs/base_strict/jobs.tsv 33 ./drive.sh strict 2013 runs/base_strict/jobs.tsv runs/base_strict/done.tsv 34 python3 -I validate.py runs/base_strict/out runs/base_strict/val.json 35 python3 -I report.py runs/base_strict/val.json baseline/strict.kinds.json 12 36 37The steps: 38- `make-jobs.sh` collects the test documents under `main/` (doc, docx, rtf, 39 xls, ppt, sx*, od*). 40- `drive.sh <ext|strict> <port> <jobs> <done>` saves them in ODF 41 1.2 Extended (`DefaultVersion=3`) or strict ODF 1.2 (`DefaultVersion=4`). 42 - It runs a headless office on the given port, using a profile under `runs/`. 43 - It resumes where it left off, kills a hung office and continues, and 44 stops at the first crash. 45- `validate.py` checks content, styles, meta, settings and manifest streams 46 against 1.2 as written, against 1.3 (`13`) and 1.4 (`14`) with 47 `office:version` rewritten, and against 1.2 and 1.3 as extended documents 48 (`12x`, `13x`): `foreign.py` first removes 49 foreign elements and attributes as Part 1 §2.2.2 B.1 and §3.17 describe (the 50 manifest is left as written). Use `report.py <val.json> <kinds.json> 12x` to 51 measure what is wrong apart from foreign content. 52 Embedded objects whose `content.xml` is MathML are skipped (ODF 1.2 Part 1 53 §2.2.1 B.4). 54- `report.py` groups the errors by normalized message. 55- `pkgcheck.py <outdir> <pkg.json>` checks the ODF 1.2 Part 3 package rules 56 (§2.2.1, 3.2, 3.3, 3.4.1): zip methods, `mimetype` first / stored / no extra 57 field / equal to the `/` media type, one manifest entry per file, META-INF 58 contents, `manifest:size` and STORED on encrypted entries, and signature 59 files (well-formed, root, `OpenDocument-v1.2-os-dsig-schema.rng`). Findings 60 are `shall:` or `should:`; only `shall` findings are non-conformances. 61- `doctypecheck.py <outdir> <dt.json>` checks Part 1 §2.2.1 A.2/E and 62 §2.2.3-2.2.10: the `mimetype` file (or `office:mimetype` of a flat file), the 63 root element and the `office:body` child must agree on one document type. 64 Embedded sub documents are compared with their manifest media type 65 (`should:` findings). Two deviations from the literal text of §2.2.5: it 66 spells the drawing values `application.vnd.oasis.opendocument.graphics` and 67 `...graphic-template`; the registered `...graphics` and `...graphics-template` 68 of Part 1 appendix C are accepted. `...text-web` and `...formula-template` 69 are registered but no §2.2.x allows them, so they are reported. 70- `formulacheck.py <outdir> <formulas.json>` parses the `style:condition`, 71 `table:condition`, `table:expression`, `table:formula` and `text:formula` 72 values (Part 1 §2.2.1 D.3, §2.2.4 D/E) with the OpenFormula grammar (Part 2 73 chapter 5) and the condition syntaxes of Part 1 §19.468 and §19.596. 74 - Syntax only; no semantics, function arity or type checking. 75 - A prefix bound to another namespace is out of Part 2's reach: `should:` in 76 general, `shall:` for `table:formula` inside `office:spreadsheet` (§2.2.4 D). 77 - Non-ASCII name characters use Unicode categories instead of the XML 1.0 78 4th edition tables; an IRI is checked by character class, not the full 79 RFC 3987 grammar; `openformula-functions.txt` holds the Part 2 function 80 names. 81 - The condition syntaxes follow the prose of Part 1, which has no grammar: 82 `,` between arguments, lowercase ` and `, `!=` as the relational operator. 83 - `formula-cases.tsv` is the table of good and bad values that `selftest.py` 84 runs. 85- `crosscheck.py <outdir> <val.json> <cross.json>` runs the Apache ODF Toolkit 86 `odfvalidator` 1.1.8 (`fetch-odfvalidator.sh`; MSV instead of jing, the same 87 OASIS schemas) in conformance mode (`-c`) and extended mode (`-e`) and 88 compares each stream with the `12` and `12x` results of `validate.py`. 89 - Error counts differ between engines; it compares whether a stream has 90 any error and, in `-c` mode, where its first error is. 91 - It exits if odfvalidator stops before finishing every package. Its 92 failures are silent (an invalid schema prints "Validation aborted"; a 93 crash on one package aborts the run): crashes are skipped and listed. 94 - odfvalidator's own schema lookup fails on current JDKs, so the schemas are 95 passed explicitly; its jar has no MathML 2 schema, so a schema that accepts 96 anything stands in (as `validate.py` skips MathML). 97 - Known leniencies: it truncates a `draw:points` over 2048 characters and 98 adapts old OOo namespaces. It is stricter than Part 3 §3.2 in wanting 99 `META-INF/documentsignatures.xml` listed in the manifest. It checks ID 100 uniqueness, which `jing -i` does not. 101- `fixtures/doctypes/` (`run.sh`, `make.py`, `derive-chart.py`), `fixtures/encrypted/`, `fixtures/signed/`, `fixtures/base/`: documents 102 of the kinds the corpus lacks (templates, master, Writer/Web, Impress as Draw, Math, chart, encrypted, signed, Base), saved 103 in strict and extended mode under `runs/task12/gen-*`. `decrypt.py <in-dir> <out-dir> <password>` writes plain copies of 104 encrypted packages for the schema checkers; `pkgcheck.py` must still run on the real ones. Signing needs a visible 105 (`-invisible`) office because `-headless` cancels the dialog: see the header of `fixtures/signed/make.py`. 106- `fixtures/{writer,calc,draw,chart,styles-text,styles-gfx,leak}/make.py`, run with `fixtures/gen-run.sh <area> <strict|ext> <port> <abs-outdir>`: 107 documents built through UNO to exercise the elements, attributes and enumerated values that `cov_report.py` lists as gaps 108 (`leak` classifies the names AOO emits that the schema does not define). Use an absolute `<outdir>`. 109- `cov_schema.py`, `cov_source.py`, `cov_corpus.py`, `cov_report.py`: schema-coverage report. The schema universe 110 (ODF 1.2 elements, element/attribute pairs, enumerated values), the names the export code in `main/` can emit (static 111 C++ scan; a lower bound, table-driven names are missed) and what a corpus exercises are joined by namespace URI and 112 local name into the gaps worth generating documents for. Outputs live under `runs/`. 113- `idcheck.py <outdir> <ids.json>` parses every stream namespace-aware and 114 checks `xml:id` syntax and uniqueness per stream (ODF 1.2 Part 1 §2.2.1 115 D.1, D.2). `jing -i` does not check ID uniqueness. 116 117## The regression gate 118 119 python3 -I gate.py <tag> --prev runs/gate-<earlier tag> # everything 120 python3 -I gate.py <tag> --build xmloff,sw --prev ... # rebuild first 121 python3 -I gate.py smoke --limit 8 --prev ... --stages corpus,validate,dump,diff 122 123`gate.py` runs the stages below in `runs/gate-<tag>/`, records `results.json`, 124skips stages already done (`--force` reruns) and exits 0 only if everything 125that ran passed. 126 127| stage | does | passes if | 128|---|---|---| 129| `build` | `build.pl` and `deliver.pl` for the `--build` modules (environment script: `ODFV_BUILD_ENV`) | exit codes 0 | 130| `refresh` | copies `--libs` from the solver into `app/` (`refresh-app.sh`) | copies identical | 131| `corpus` | saves the job list in `ext` and `strict` mode | no crash, every job accounted for, OK/FAIL counts equal `--prev` | 132| `validate` | `validate.py`, `report.py` and the four checkers per mode | kinds not worse than `baseline/<mode>.kinds.json` or than `--prev` (checked both ways); all four checkers ran; no `shall` finding from the id, package and document-type checkers; formula findings not above `--prev`; a baseline or `--prev` exists | 133| `dump` | reloads every saved package with `dump7.py` | no office failure; hangs and crashes are listed | 134| `diff` | `diff7.py` on the dumps and `cmp7.py` on the packages against `--prev` | every difference is a document in `gate-volatile.tsv` | 135| `fixtures` | `gate-fixtures.tsv`: saves in `v00 v10 v11 ext strict`, runs the reload checks, validates | saved, checked, and unchanged against `--prev`; the same checker rules as `validate` (formula findings are compared per mode with `--prev`) | 136 137- Inputs are frozen: without `--jobs` the list of `--prev` is reused (outputs 138 are rewritten), so the same documents are saved when files under `main/` 139 change. A later run's `--prev` may be any earlier gate directory or a 140 directory with the same layout (`<mode>/out`, `<mode>/dump`). 141- The dump stage kills the office only when the dump makes no progress for 142 `--stall` seconds (default 300), then writes a `HANG` marker for that 143 document alone. A long batch that keeps progressing is never killed. 144- `gate-volatile.tsv` names the documents whose dumps differ between runs 145 without an export change (volatile Calc formulas, redline timestamps, the 146 `0238` loader order, the intermittent Basic library loss of `0048`). A new 147 entry needs a reason. 148- The first fixtures run has nothing to compare with: its mismatch pattern is 149 only recorded. By design, strict, ODF 1.0 and ODF 1.1 mode fail the 14 150 reload rows that need gradient or transparency fills. 151 152## Checking a change 153 154 python3 -I compare.py baseline/strict.kinds.json runs/new/strict.kinds.json 155 156`compare.py` exits 1 if any kind is new or any kind got worse. 157 158To test a rebuilt module without repackaging, copy its libraries from the 159solver into the app copy: 160 161 ./refresh-app.sh libxo.dylib 162 163`roundtrip.py <port> <file> <checks.json>` loads a document and compares UNO 164expressions with expected values; `--capture <out.json>` records them instead. 165 166## Formula evaluation baseline (calcval) 167 168 ./start.sh calcval 2792 169 ./runpy.sh calcval.py 2792 calcval-cases.tsv runs/phase4/baseline.json 170 python3 -I calcval.py --compare runs/phase4/baseline.json runs/phase4/new.json 171 172`calcval.py` opens a new Calc document, writes the data area described in the 173header of `calcval-cases.tsv`, and for each case enters the formula in a fresh 174scratch cell (`cell`, `cell@N`) or range (`array:CxR`), recalculates, records 175the result type, `getError()`, `getString()`, `getValue()` and the formula as 176read back, then clears the scratch area. `expected` is the value the ODF 1.3 177Part 4 text requires (`-` records the current behaviour only); the status of 178each case is `match`, `mismatch`, `record` or `exception`. The run prints 179every mismatch, which are the current deviations from ODF 1.3. `--compare` 180lists each case whose recorded result changed (exit 1 if any). RECEIVED and 181BESSELK/BESSELY are Analysis add-ins and are entered by programmatic name. 182 183## Notes 184 185- AOO has no `-convert-to`; conversion goes over UNO (`convert.py`). 186- jing must run with `-i`, otherwise it rejects the ODF schemas themselves. 187- A jing fatal error can hide later errors. Such lines are reported as 188 `UNPARSED` kinds, and `compare.py` fails on any new `UNPARSED` kind. 189- jing stops at a fatal error (a stream that is not well-formed); 190 `validate.py` records it and carries on with the remaining files. 191- `setup-app.sh` renames an existing `app/OpenOffice.app` to 192 `OpenOffice.app.old.<timestamp>` instead of deleting it. 193