# Extended tests

 > NOTE: This is for developers

A large number (current 3081) of unique shareware Wise installers have
been found. It is `70.19 GiB` in size, so make sure you have that free space.

These tests won't extract anything to disk, it uses `--list`,  `--raw-verify`,
and `--verify` for the dev version of REWise and do a `--verify` test with
REWise v0.2.0 for comparison. Along with that it tries to analyze errors and
comment them.

These scripts require Python 3, so make sure you have that installed.

 > TIP: Use `csvlens`, `libreoffice` or whatever to view the .csv files.

## How to

### Initial setup

 1. If you don't have a REWise v0.2.0 binary then download and compile
    [REWise v0.2.0](https://codeberg.org/CYBERDEV/REWise/archive/v0.2.0.tar.gz).
 2. Compile the REWise dev version with `make debug`, it should be
    in `../../` (from this README.md file, by default).
 3. Copy `config.py.template` to `config.py` and edit `REWISE_02_PATH`
    so it points to your REWise v0.2.0 binary. When you already have
    downloaded the shareware installers somewhere else then default
    you want to edit `DL_PATH`.
 4. Download the installers; make sure you have enough free space and take in
    mind that this will take a while:

    ```
    python test.py download sources/shareware.csv
    ```

    See the `cache/dl.log` file for download/file errors.

 5. Run the initial test; gather info about the installers, run tests against
    REWise v0.2.0 and the REWise dev version. The input will be a source .csv
    and the output will be a result .csv file that will be our base. Later
    tests will skip the REWise v0.2.0 tests because they will be the
    same anyway.

    ```
    python test.py init sources/shareware.csv ./results.csv
    ```


### Testing local changes

So you made some code changes and recompiled REWise with `make debug`
(`rewise` bin should be in `../../`), now it's time to see if it fixed
or broke any installer.

First thing is to re-run the tests against the freshly compiled REWise
bin:

```
python test.py create-diff ./results.csv ./new_results.csv
```

To print the installers that it fixed/broke:

```
python test.py print-diff ./results.csv ./new_results.csv

Fixed (0)
--------------------------------

Broke (2)
--------------------------------
c7aa23ed03082d671a1e2bf776361d685b51f1acfa9525a636f15e25d681084e setup.exe
ff9c38732523f39432479a1f5625a7bf089fab036cc5c7d89f288df1bf56e306 AdminPro_DEMO.exe
```

To print stats of the new results:

```
python test.py print-stats ./new_results.csv

| DATE                | OK   | ERR  | LOK  | LERR | ROK  | RERR | v02  | v02E | VERSION
| :------------------ | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :------
| NE                  |   96 |    6 |  100 |    2 |  101 |    1 |    0 |  102 | Unknown
| 1998-10-02 22:36:51 |   78 |    4 |   78 |    4 |   82 |    0 |   58 |   24 | Unknown
| 1998-10-23 23:23:16 |    2 |    0 |    2 |    0 |    2 |    0 |    2 |    0 | Unknown
| 1998-11-09 21:17:09 |  176 |   22 |  179 |   19 |  198 |    0 |  132 |   66 | InstallMaster 7
| 1998-12-03 23:11:32 |   43 |    0 |   43 |    0 |   43 |    0 |   28 |   15 | Unknown
| 1999-02-05 23:07:52 |  214 |   19 |  233 |    0 |  232 |    1 |  132 |  101 | Unknown
| 1999-04-05 18:07:26 |    4 |    0 |    4 |    0 |    4 |    0 |    4 |    0 | Unknown
| 1999-05-21 22:48:48 |  406 |   33 |  420 |   19 |  432 |    7 |  267 |  172 | Unknown
| 1999-08-17 17:25:48 |  238 |   24 |  250 |   12 |  254 |    8 |  143 |  119 | Unknown
| 2000-04-25 16:37:12 |  894 |   49 |  914 |   29 |  937 |    6 |  514 |  429 | InstallMaster 8
| 2001-08-13 19:13:38 |  319 |    8 |  325 |    2 |  323 |    4 |  222 |  105 | Unknown
| 2001-10-25 21:47:11 |  418 |   24 |  433 |    9 |  429 |   13 |  246 |  196 | Installation System 9
| 2031-05-20 22:54:01 |    4 |    0 |    4 |    0 |    4 |    0 |    4 |    0 | Unknown
| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | -------
| Total:         3081 | 2892 |  189 | 2985 |   96 | 3041 |   40 | 1752 | 1329 | 

PK Total    : 438
PK OK       : 432
PK ERROR    : 6
PK Raw OK   : 438
PK Raw ERROR: 0

Comments:
 80 Without comment
 23 Negative script deflate offset
 21 Possible weird stuff at WiseScriptHeader.unknown_22 '9190A09A878B'.
 14 Possible weird stuff at WiseScriptHeader.unknown_22 '958A928F8C8B'.
 10 Possible weird stuff at WiseScriptHeader.unknown_22 'B890BE889E86'.
  6 Possible weird stuff at WiseScriptHeader.unknown_22 'B6BF88BF96BF'.
  6 Possible weird stuff at WiseScriptHeader.unknown_22 'A8B0ADB3BBB6'.
  5 'Initializing Wise Installation Wizard...' found in the PE b'.data\x00\x00\x00' section.
  3 Possible weird stuff at WiseScriptHeader.unknown_22 'BCADA6CEA5CD'.
  3 Possible weird stuff at WiseScriptHeader.unknown_22 'C8CACBC8C6CC'.
  2 Possible weird stuff at WiseScriptHeader.unknown_22 '99939E9D919E'.
  2 0x18 mismatch, 0x18 00 08 ..
  2 Possible weird stuff at WiseScriptHeader.unknown_22 'DB8F9E8C8C88'.
  2 Possible weird stuff at WiseScriptHeader.unknown_22 'ADBF90BF93BF'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 '8588C9C6C88B'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 'AB969C949A8B'.
  1 0x18 mismatch, there is OP 0x00 right after 0x18
  1 Script deflate offset is absurd high
  1 Possible weird stuff at WiseScriptHeader.unknown_22 'CC918B90929D'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 'B4AAACB6B5AC'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 '9D8D9E919B96'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 '9C909B9A989A'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 '8C929E9393D1'.
  1 Overlay header unknown2 != 0x8000 but 0x0000.

Verify errors:
 75 WS_ERROR_VALUE     ERROR: parseWiseScriptHeader failed to read languageTexts.
 46 WS_ERROR_INFLATE   ERROR: Inflated size is larger then expected
 25 WS_ERROR_SEEK      ERROR: Failed seek to file offset
  9 ERROR_VALUE        ERROR: Advertised WiseScript.bin size looks insane.
  7 WS_ERROR_INFLATE   ERROR: zlib: invalid code lengths set
  4 WS_ERROR_INFLATE   ERROR: zlib: invalid distance too far back
  4 WS_ERROR_INFLATE   ERROR: zlib: invalid stored block lengths
  3 WS_ERROR_PK_HEADER ERROR: PK signature unknown
  3 WS_ERROR_VALUE     ERROR: wiseScriptParsePath expected '%' at path start
  2 WS_ERROR_VALUE     ERROR: parseWiseScript unknown OP: 3A
  2 WS_ERROR_INFLATE   ERROR: zlib: invalid block type
  2 ERROR_VALUE        ERROR: Advertised .dib size looks insane.
  1 WS_ERROR_BRANCHING ERROR: OP 0x23 LANG but not in LANG branch
  1 WS_ERROR_INFLATE   ERROR: zlib: invalid literal/length code
  1 WS_ERROR_VALUE     ERROR: This is a multi language installer but not all languages have their name set
  1 WS_ERROR_VALUE     ERROR: parseWiseScript unknown OP: 49
  1 WS_ERROR_INFLATE   ERROR: zlib: invalid distance code
  1 WS_ERROR_BRANCHING ERROR: OP 0x23 LANG but currently not in a branch
  1 ERROR_INFLATE      ERROR: zlib: invalid stored block lengths
```

Further inspection of the results file can be done using something that can
view .csv files.


### Write debug scripts

I've written a lot of small scripts for debugging specific stuff, like printing
a list of all PE build dates and the overlay offset(s) observed. Such things
are very specific and I didn't find it worth including, but here is a example:

```python
import os
from installers import installerStatsCsvIt

found = {}
for installer in installerStatsCsvIt("new_results.csv"):
  if installer.isNe:
    continue

  fdate = installer.formattedDate()
  if fdate not in found:
    found.update({fdate: []});

  hexoverlay = hex(installer.overlay)
  if hexoverlay not in found[fdate]:
     found[fdate].append(hexoverlay)

for pebuild, offsets in sorted(found.items()):
  print(pebuild, offsets)
```

## Files

### test.py

```help
usage: test.py [-h] {download,init,update,create-diff,filter-invalid,update-comments,print-diff,print-stats,result-to-source} ...

options:
  -h, --help            show this help message and exit

COMMANDS:
  Note: use -h after a command to get more help on that command

  {download,init,update,create-diff,filter-invalid,update-comments,print-diff,print-stats,result-to-source}
    download            Download installers from the given SOURCE file when needed
    init                Run initial test, the output will be written to the given RESULT file. The initial test does include testing against REWise v0.2
    update              Add new installers from given SOURCE and re-run tests for all installers, excluding the REWise v0.2 tests. To only add new and run tests for new installers give the '--only-new' argument.
    create-diff         Re-run tests of first given RESULT, ouput to second given RESULT file.
    filter-invalid      Filter out results that appear to be invalid files / not Wise installers.
    update-comments     Update comments for given RESULT file.
    print-diff          Compare two RESULT files and print the installers that it fixed/broke.
    print-stats         Print some stats for the given RESULT file.
    result-to-source    Convert RESULT file into SOURCE file.
```

#### `module` analyze.py

Analyze errors and add comments.

#### `module` dl.py

For downloading files.

#### `module` sources.py

Handle reading source .csv files.

The columns:

 1. B3 (Blake 3)
 2. MD5
 3. Filesize in bytes
 4. Download URL

#### `module` installers.py

Handle reading/writing result .csv files.

#### `module` rewise.py

This module handles testing and output parsing of REWise current dev and
REWise v0.2.0.

