1<!-- 2 Licensed to the Apache Software Foundation (ASF) under one 3 or more contributor license agreements. See the NOTICE file 4 distributed with this work for additional information 5 regarding copyright ownership. The ASF licenses this file 6 to you under the Apache License, Version 2.0 (the 7 "License"); you may not use this file except in compliance 8 with the License. You may obtain a copy of the License at 9 10 http://www.apache.org/licenses/LICENSE-2.0 11 12 Unless required by applicable law or agreed to in writing, 13 software distributed under the License is distributed on an 14 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 15 KIND, either express or implied. See the License for the 16 specific language governing permissions and limitations 17 under the License. 18--> 19 20# Using the suite 21 22Run everything from this directory. `python3 -m pytest` is enough; `uv` is not required. 23 24## Choosing the office 25 26An office is only started when you ask for one, with `AOO_OFFICE` or `--office`. Both take any of: 27 28- an `.app` bundle (macOS) 29- the install root 30- the `program` directory 31- the `soffice` launcher 32 33Builds before 5.0 are refused. Without an office, the tests that need one skip and say why; add `--require-office` to make that a failure. 34 35The suite never touches your own profile. Each ODF save mode gets its own private profile and a headless office under pytest's temporary directory. The offices are stopped at the end, and the session fails if one is still running. 36 37## Tools for the schema checks 38 39The ODF schema checks need Java, jing and the OASIS schemas. Fetch them once: 40 41 PYTHONPATH=src python3 -m aootest.fetch # download what is missing 42 PYTHONPATH=src python3 -m aootest.fetch --list # show what is cached 43 PYTHONPATH=src python3 -m aootest.fetch --offline # download nothing; exit 1 if something is missing 44 45Each download is checked against its pinned sha256 before it is used. The cache is `~/Library/Caches/aootest` on macOS, `$XDG_CACHE_HOME/aootest` or `~/.cache/aootest` on Linux and `%LOCALAPPDATA%\aootest\Cache` on Windows; `AOO_TEST_CACHE` overrides it. To use your own copies instead, set `AOO_JING`, `AOO_ODFVALIDATOR`, `AOO_SCHEMA_DIR` or `AOO_JAVA`. 46 47## Running 48 49 python3 -m pytest -m selftest # checkers against negative controls, no office 50 python3 -m pytest tests/harness # install and worker handling, no office 51 python3 -m pytest -m smoke # quick office checks 52 python3 -m pytest -m "not slow" # all of the above 53 python3 -m pytest tests/odf # save modes and the other families, slow 54 python3 -m pytest -m crash # known crashes of the office, kept out of the default run 55 python3 -m pytest -k convert # any pytest selection works 56 57## Options 58 59| Option | Environment | Meaning | 60|---|---|---| 61| `--office PATH` | `AOO_OFFICE` | The install to test. | 62| `--uno-python PATH` | `AOO_UNO_PYTHON` | A Python that can import `uno`. Skips the search; if it fails the probe, that is an error. | 63| `--connect socket\|pipe` | | How the worker reaches the office. Default `socket`. | 64| `--max-restarts N` | | Office restarts allowed per save mode before the session aborts. Default 5. | 65| `--start-timeout S` | | Seconds an office may take to start and accept a connection. Default 240; starts have taken from 5 to 90 seconds on a busy desktop. | 66| `--require-office` | | Fail instead of skip when there is no usable office. | 67| `--require-odf13` | | Fail instead of skip the ODF 1.3 save modes when the build cannot write 1.3. | 68| `--require-odf14` | | Fail instead of skip the ODF 1.4 save modes when the build cannot write 1.4. | 69| `--documents DIR` | | Extra ODF documents to run through every save mode. | 70| `--shipped-limit N` | | How many shipped templates and documents to use. Default 40. | 71| `--twin-corpus DIR` | `AOO_TWIN_CORPUS` | Corpus for the import twin tests. | 72| `--strict-expect` | | A test that an expectation row says should fail must fail. | 73 74## Pyuno 75 76The tests talk to the office through a small worker process that runs under a Python able to import `uno`. The suite tries these in order and keeps the first that imports `uno`: the install's own Python (`program/python`, shipped by Linux and FreeBSD builds), the Python that `libpyuno` is linked against, the Python running pytest, then `python3.N` on the path. A macOS build that links a system Python needs that Python installed. If none works, the office tests skip and list what was tried. `--uno-python` names one directly. 77 78## Source documents 79 80The save-mode tests run over documents that are not stored in this repository: 81 82- Templates and documents the install ships under `share/template` and `share/samples`, up to `--shipped-limit`. File types and directories are interleaved, so a small limit still spreads across both, and the same install always gives the same list. 83- One new document per application (Writer, Calc, Impress, Draw), created by the office. 84- Documents the worker builds through the API (`src/aootest_worker/ops/generate.py`): page fills, other fills, ODF 1.3 features. They are made in ODF 1.2 extended mode, which keeps every feature, then saved in each mode and read back; the values each read-back must give are in `data/checks/`. 85- Documents made by the large scripts of the ODF harness (`src/aootest_worker/fixtures/`, run by `ops/scripts.py`): Writer, Calc, Draw and Impress features, charts, style properties, document types. `data/built-files.tsv` lists what each script makes. Each script runs in an office of its own, which is closed afterwards; a script that stalls is run once more. They carry no read-back checks, so they add the save-mode items only. 86- Every ODF document under `--documents DIR`, if given. 87 88`data/built-excluded.tsv` names the documents of those scripts that are left out of the default run because converting them kills or stalls the office (Calc documents with tracked changes, Base documents). `python3 -m pytest -m crash` runs each of them through every save mode in a fresh office; expect crash reports, and on macOS the "unexpectedly quit" prompt for each crash, to which `Don't Reopen` is the answer. Known deviations of the other documents are rows in `data/expect/`. 89 90Each document is saved once per mode: strict ODF 1.2 (save mode 4), 1.2 extended (3), 1.3 (5), 1.3 extended (6), 1.4 (7), 1.4 extended (8) and the product default. Builds that predate the ODF 1.3 work ignore modes 5 and 6, and builds that predate the ODF 1.4 work ignore modes 7 and 8; those items skip, or fail with `--require-odf13` or `--require-odf14`. 91 92## Other families 93 94- **Error kinds of the extended modes.** The 1.2, 1.3 and 1.4 extended modes write extension content on purpose, so their output is not asserted to be schema valid. `tests/odf/test_kinds.py` compares the errors against the ODF 1.2 (1.2 extended), 1.3 (1.3 extended) and 1.4 (1.4 extended) schemas, and the same with foreign content removed, with `data/baseline/`: a new kind of error, a higher count, a kind found in more documents, or a baseline document that is no longer covered fails the item; fewer errors pass. The baseline covers the documents the suite builds or creates, not the shipped ones, and was written with the trunk dev app, so another build can differ. After a change that is meant to alter the errors, run `python3 -m pytest tests/odf/test_kinds.py --write-kinds` and read the diff of `data/baseline/` before committing it. 95 96- **Cross-check.** odfvalidator checks the same saved files as jing; the two must agree on which streams have errors. 97- **Stability.** Each document is saved twice; `content.xml` and `styles.xml` must match apart from the cached value of date and time fields. 98- **Encrypted.** Documents saved with a password under each combination of `UseSHA1InODF12` and `UseBlowfishInODF12`, in ODF 1.2 strict and extended mode. The suite's own decryptor (`aootest/checkers/decrypt.py`) must read them with the right password, reject a wrong one, and get the same content as the unencrypted copy. 99- **Formulas.** `data/calcval-cases.tsv` holds about 300 OpenFormula cases with the result ODF 1.3 and 1.4 Part 4 require; each is evaluated in Calc. 100 101## Import twins 102 103The twin tests load documents saved as ODF 1.2 and 1.3 from the same source by another producer and check that AOO imports each pair alike. They need a corpus, which is not part of the suite: a directory with `v12`, `v13`, `v12ext` and `v13ext` subdirectories holding the same file names. Point `--twin-corpus` or `AOO_TWIN_CORPUS` at it; without one the items skip. Two controls run with it: an unchanged copy must compare equal and a copy with one changed word must compare different. 104 105## Expected failures 106 107Known problems are rows in `data/expect/*.tsv`: 108 109 test_name<TAB>xfail|flaky|skip<TAB>reason 110 111`test_name` is the pytest item name including its parameters, and `*` matches any characters. The reason states what is known. An `xfail` row that starts passing shows as XPASS; with `--strict-expect` that fails the run. A `flaky` row is for a test that fails only in some runs and is never strict. At the end of a full run the suite reports rows that matched no test. 112 113## Reading results 114 115- A skip always carries a reason: no office, no tools, a mode the build cannot write. 116- A build that predates the ODF conformance work fails schema checks and some formula cases (the installed 5.1.0 does). Those are results about the build, not suite errors. 117- A hung or crashed office is recorded against the document that caused it, the office is restarted, and the run continues. Too many restarts abort the session. 118