xref: /trunk/test/odf-validate/README.md (revision 0e16a31d78a652c56c5c0fc79c752f48b9048df9)

# ODF validation harness

This harness has OpenOffice save a corpus of documents as ODF, checks every
saved package against the OASIS ODF 1.2, ODF 1.3 and ODF 1.4 RELAX NG schemas with
jing, and reports the errors grouped by kind. Comparing those reports against
a baseline shows whether an export change fixed errors or introduced new ones.

It runs on macOS only (zsh scripts, `hdiutil`, `.app` layout).

## Requirements

- An AOO macOS build (`.dmg`).
- The Python that the office's pyuno links against. Check with
  `otool -L <app>/Contents/program/libpyuno.dylib`, then set `ODFV_PYTHON`.
- `jing.jar` (set `ODFV_JING`) and `java`; `ODFV_ODFVALIDATOR` and `ODFV_JAVA` for `crosscheck.py`.
- `gtimeout` (GNU coreutils), used by `drive.sh`.

Defaults for `ODFV_APP`, `ODFV_PYTHON`, `ODFV_JING` and `ODFV_SOLVER` are in
`env.sh`.

## Setup

    ./fetch-schemas.sh                 # OASIS schemas, checksum-verified, into schema/
    ./setup-app.sh <path-to-AOO.dmg>   # copies OpenOffice.app into app/
    ./fetch-odfvalidator.sh            # optional: odfvalidator for crosscheck.py
    python3 -I selftest.py             # validator negative controls

The schemas are not committed because they carry OASIS copyright terms.

## Running a corpus

    ./make-jobs.sh runs/base_strict/out > runs/base_strict/jobs.tsv
    ./drive.sh strict 2013 runs/base_strict/jobs.tsv runs/base_strict/done.tsv
    python3 -I validate.py runs/base_strict/out runs/base_strict/val.json
    python3 -I report.py runs/base_strict/val.json baseline/strict.kinds.json 12

The steps:
- `make-jobs.sh` collects the test documents under `main/` (doc, docx, rtf,
  xls, ppt, sx*, od*).
- `drive.sh <ext|strict> <port> <jobs> <done>` saves them in ODF
  1.2 Extended (`DefaultVersion=3`) or strict ODF 1.2 (`DefaultVersion=4`).
  - It runs a headless office on the given port, using a profile under `runs/`.
  - It resumes where it left off, kills a hung office and continues, and
    stops at the first crash.
- `validate.py` checks content, styles, meta, settings and manifest streams
  against 1.2 as written, against 1.3 (`13`) and 1.4 (`14`) with
  `office:version` rewritten, and against 1.2 and 1.3 as extended documents
  (`12x`, `13x`): `foreign.py` first removes
  foreign elements and attributes as Part 1 §2.2.2 B.1 and §3.17 describe (the
  manifest is left as written). Use `report.py <val.json> <kinds.json> 12x` to
  measure what is wrong apart from foreign content.
  Embedded objects whose `content.xml` is MathML are skipped (ODF 1.2 Part 1
  §2.2.1 B.4).
- `report.py` groups the errors by normalized message.
- `pkgcheck.py <outdir> <pkg.json>` checks the ODF 1.2 Part 3 package rules
  (§2.2.1, 3.2, 3.3, 3.4.1): zip methods, `mimetype` first / stored / no extra
  field / equal to the `/` media type, one manifest entry per file, META-INF
  contents, `manifest:size` and STORED on encrypted entries, and signature
  files (well-formed, root, `OpenDocument-v1.2-os-dsig-schema.rng`). Findings
  are `shall:` or `should:`; only `shall` findings are non-conformances.
- `doctypecheck.py <outdir> <dt.json>` checks Part 1 §2.2.1 A.2/E and
  §2.2.3-2.2.10: the `mimetype` file (or `office:mimetype` of a flat file), the
  root element and the `office:body` child must agree on one document type.
  Embedded sub documents are compared with their manifest media type
  (`should:` findings). Two deviations from the literal text of §2.2.5: it
  spells the drawing values `application.vnd.oasis.opendocument.graphics` and
  `...graphic-template`; the registered `...graphics` and `...graphics-template`
  of Part 1 appendix C are accepted. `...text-web` and `...formula-template`
  are registered but no §2.2.x allows them, so they are reported.
- `formulacheck.py <outdir> <formulas.json>` parses the `style:condition`,
  `table:condition`, `table:expression`, `table:formula` and `text:formula`
  values (Part 1 §2.2.1 D.3, §2.2.4 D/E) with the OpenFormula grammar (Part 2
  chapter 5) and the condition syntaxes of Part 1 §19.468 and §19.596.
  - Syntax only; no semantics, function arity or type checking.
  - A prefix bound to another namespace is out of Part 2's reach: `should:` in
    general, `shall:` for `table:formula` inside `office:spreadsheet` (§2.2.4 D).
  - Non-ASCII name characters use Unicode categories instead of the XML 1.0
    4th edition tables; an IRI is checked by character class, not the full
    RFC 3987 grammar; `openformula-functions.txt` holds the Part 2 function
    names.
  - The condition syntaxes follow the prose of Part 1, which has no grammar:
    `,` between arguments, lowercase ` and `, `!=` as the relational operator.
  - `formula-cases.tsv` is the table of good and bad values that `selftest.py`
    runs.
- `crosscheck.py <outdir> <val.json> <cross.json>` runs the Apache ODF Toolkit
  `odfvalidator` 1.1.8 (`fetch-odfvalidator.sh`; MSV instead of jing, the same
  OASIS schemas) in conformance mode (`-c`) and extended mode (`-e`) and
  compares each stream with the `12` and `12x` results of `validate.py`.
  - Error counts differ between engines; it compares whether a stream has
    any error and, in `-c` mode, where its first error is.
  - It exits if odfvalidator stops before finishing every package. Its
    failures are silent (an invalid schema prints "Validation aborted"; a
    crash on one package aborts the run): crashes are skipped and listed.
  - odfvalidator's own schema lookup fails on current JDKs, so the schemas are
    passed explicitly; its jar has no MathML 2 schema, so a schema that accepts
    anything stands in (as `validate.py` skips MathML).
  - Known leniencies: it truncates a `draw:points` over 2048 characters and
    adapts old OOo namespaces. It is stricter than Part 3 §3.2 in wanting
    `META-INF/documentsignatures.xml` listed in the manifest. It checks ID
    uniqueness, which `jing -i` does not.
- `fixtures/doctypes/` (`run.sh`, `make.py`, `derive-chart.py`), `fixtures/encrypted/`, `fixtures/signed/`, `fixtures/base/`: documents
  of the kinds the corpus lacks (templates, master, Writer/Web, Impress as Draw, Math, chart, encrypted, signed, Base), saved
  in strict and extended mode under `runs/task12/gen-*`. `decrypt.py <in-dir> <out-dir> <password>` writes plain copies of
  encrypted packages for the schema checkers; `pkgcheck.py` must still run on the real ones. Signing needs a visible
  (`-invisible`) office because `-headless` cancels the dialog: see the header of `fixtures/signed/make.py`.
- `fixtures/{writer,calc,draw,chart,styles-text,styles-gfx,leak}/make.py`, run with `fixtures/gen-run.sh <area> <strict|ext> <port> <abs-outdir>`:
  documents built through UNO to exercise the elements, attributes and enumerated values that `cov_report.py` lists as gaps
  (`leak` classifies the names AOO emits that the schema does not define). Use an absolute `<outdir>`.
- `cov_schema.py`, `cov_source.py`, `cov_corpus.py`, `cov_report.py`: schema-coverage report. The schema universe
  (ODF 1.2 elements, element/attribute pairs, enumerated values), the names the export code in `main/` can emit (static
  C++ scan; a lower bound, table-driven names are missed) and what a corpus exercises are joined by namespace URI and
  local name into the gaps worth generating documents for. Outputs live under `runs/`.
- `idcheck.py <outdir> <ids.json>` parses every stream namespace-aware and
  checks `xml:id` syntax and uniqueness per stream (ODF 1.2 Part 1 §2.2.1
  D.1, D.2). `jing -i` does not check ID uniqueness.

## The regression gate

    python3 -I gate.py <tag> --prev runs/gate-<earlier tag>      # everything
    python3 -I gate.py <tag> --build xmloff,sw --prev ...        # rebuild first
    python3 -I gate.py smoke --limit 8 --prev ... --stages corpus,validate,dump,diff

`gate.py` runs the stages below in `runs/gate-<tag>/`, records `results.json`,
skips stages already done (`--force` reruns) and exits 0 only if everything
that ran passed.

| stage | does | passes if |
|---|---|---|
| `build` | `build.pl` and `deliver.pl` for the `--build` modules (environment script: `ODFV_BUILD_ENV`) | exit codes 0 |
| `refresh` | copies `--libs` from the solver into `app/` (`refresh-app.sh`) | copies identical |
| `corpus` | saves the job list in `ext` and `strict` mode | no crash, every job accounted for, OK/FAIL counts equal `--prev` |
| `validate` | `validate.py`, `report.py` and the four checkers per mode | kinds not worse than `baseline/<mode>.kinds.json` or than `--prev` (checked both ways); all four checkers ran; no `shall` finding from the id, package and document-type checkers; formula findings not above `--prev`; a baseline or `--prev` exists |
| `dump` | reloads every saved package with `dump7.py` | no office failure; hangs and crashes are listed |
| `diff` | `diff7.py` on the dumps and `cmp7.py` on the packages against `--prev` | every difference is a document in `gate-volatile.tsv` |
| `fixtures` | `gate-fixtures.tsv`: saves in `v00 v10 v11 ext strict`, runs the reload checks, validates | saved, checked, and unchanged against `--prev`; the same checker rules as `validate` (formula findings are compared per mode with `--prev`) |

- Inputs are frozen: without `--jobs` the list of `--prev` is reused (outputs
  are rewritten), so the same documents are saved when files under `main/`
  change. A later run's `--prev` may be any earlier gate directory or a
  directory with the same layout (`<mode>/out`, `<mode>/dump`).
- The dump stage kills the office only when the dump makes no progress for
  `--stall` seconds (default 300), then writes a `HANG` marker for that
  document alone. A long batch that keeps progressing is never killed.
- `gate-volatile.tsv` names the documents whose dumps differ between runs
  without an export change (volatile Calc formulas, redline timestamps, the
  `0238` loader order, the intermittent Basic library loss of `0048`). A new
  entry needs a reason.
- The first fixtures run has nothing to compare with: its mismatch pattern is
  only recorded. By design, strict, ODF 1.0 and ODF 1.1 mode fail the 14
  reload rows that need gradient or transparency fills.

## Checking a change

    python3 -I compare.py baseline/strict.kinds.json runs/new/strict.kinds.json

`compare.py` exits 1 if any kind is new or any kind got worse.

To test a rebuilt module without repackaging, copy its libraries from the
solver into the app copy:

    ./refresh-app.sh libxo.dylib

`roundtrip.py <port> <file> <checks.json>` loads a document and compares UNO
expressions with expected values; `--capture <out.json>` records them instead.

## Formula evaluation baseline (calcval)

    ./start.sh calcval 2792
    ./runpy.sh calcval.py 2792 calcval-cases.tsv runs/phase4/baseline.json
    python3 -I calcval.py --compare runs/phase4/baseline.json runs/phase4/new.json

`calcval.py` opens a new Calc document, writes the data area described in the
header of `calcval-cases.tsv`, and for each case enters the formula in a fresh
scratch cell (`cell`, `cell@N`) or range (`array:CxR`), recalculates, records
the result type, `getError()`, `getString()`, `getValue()` and the formula as
read back, then clears the scratch area. `expected` is the value the ODF 1.3
Part 4 text requires (`-` records the current behaviour only); the status of
each case is `match`, `mismatch`, `record` or `exception`. The run prints
every mismatch, which are the current deviations from ODF 1.3. `--compare`
lists each case whose recorded result changed (exit 1 if any). RECEIVED and
BESSELK/BESSELY are Analysis add-ins and are entered by programmatic name.

## Notes

- AOO has no `-convert-to`; conversion goes over UNO (`convert.py`).
- jing must run with `-i`, otherwise it rejects the ODF schemas themselves.
- A jing fatal error can hide later errors. Such lines are reported as
  `UNPARSED` kinds, and `compare.py` fails on any new `UNPARSED` kind.
- jing stops at a fatal error (a stream that is not well-formed);
  `validate.py` records it and carries on with the remaining files.
- `setup-app.sh` renames an existing `app/OpenOffice.app` to
  `OpenOffice.app.old.<timestamp>` instead of deleting it.