REWise/tests/extended

Extended tests

NOTE: This is for developers

A large number (current 3081) of unique shareware Wise installers have been found. It is 70.19 GiB in size, so make sure you have that free space.

These tests won't extract anything to disk, it uses --list, --raw-verify, and --verify for the dev version of REWise and do a --verify test with REWise v0.2.0 for comparison. Along with that it tries to analyze errors and comment them.

These scripts require Python 3, so make sure you have that installed.

TIP: Use csvlens, libreoffice or whatever to view the .csv files.

How to

Initial setup

1. If you don't have a REWise v0.2.0 binary then download and compile REWise v0.2.0.
2. Compile the REWise dev version with make debug, it should be in ../../ (from this README.md file, by default).
3. Copy config.py.template to config.py and edit REWISE_02_PATH so it points to your REWise v0.2.0 binary. When you already have downloaded the shareware installers somewhere else then default you want to edit DL_PATH.

4. Download the installers; make sure you have enough free space and take in mind that this will take a while:

   python test.py download sources/shareware.csv
   

   See the cache/dl.log file for download/file errors.

5. Run the initial test; gather info about the installers, run tests against REWise v0.2.0 and the REWise dev version. The input will be a source .csv and the output will be a result .csv file that will be our base. Later tests will skip the REWise v0.2.0 tests because they will be the same anyway.

   python test.py init sources/shareware.csv ./results.csv
   

Testing local changes

So you made some code changes and recompiled REWise with make debug (rewise bin should be in ../../), now it's time to see if it fixed or broke any installer.

First thing is to re-run the tests against the freshly compiled REWise bin:

python test.py create-diff ./results.csv ./new_results.csv

To print the installers that it fixed/broke:

python test.py print-diff ./results.csv ./new_results.csv

Fixed (0)
--------------------------------

Broke (2)
--------------------------------
c7aa23ed03082d671a1e2bf776361d685b51f1acfa9525a636f15e25d681084e setup.exe
ff9c38732523f39432479a1f5625a7bf089fab036cc5c7d89f288df1bf56e306 AdminPro_DEMO.exe

To print stats of the new results:

python test.py print-stats ./new_results.csv

| DATE                | OK   | ERR  | LOK  | LERR | ROK  | RERR | v02  | v02E | VERSION
| :------------------ | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | :------
| NE                  |   96 |    6 |  100 |    2 |  101 |    1 |    0 |  102 | Unknown
| 1998-10-02 22:36:51 |   78 |    4 |   78 |    4 |   82 |    0 |   58 |   24 | Unknown
| 1998-10-23 23:23:16 |    2 |    0 |    2 |    0 |    2 |    0 |    2 |    0 | Unknown
| 1998-11-09 21:17:09 |  176 |   22 |  179 |   19 |  198 |    0 |  132 |   66 | InstallMaster 7
| 1998-12-03 23:11:32 |   43 |    0 |   43 |    0 |   43 |    0 |   28 |   15 | Unknown
| 1999-02-05 23:07:52 |  214 |   19 |  233 |    0 |  232 |    1 |  132 |  101 | Unknown
| 1999-04-05 18:07:26 |    4 |    0 |    4 |    0 |    4 |    0 |    4 |    0 | Unknown
| 1999-05-21 22:48:48 |  406 |   33 |  420 |   19 |  432 |    7 |  267 |  172 | Unknown
| 1999-08-17 17:25:48 |  238 |   24 |  250 |   12 |  254 |    8 |  143 |  119 | Unknown
| 2000-04-25 16:37:12 |  894 |   49 |  914 |   29 |  937 |    6 |  514 |  429 | InstallMaster 8
| 2001-08-13 19:13:38 |  319 |    8 |  325 |    2 |  323 |    4 |  222 |  105 | Unknown
| 2001-10-25 21:47:11 |  418 |   24 |  433 |    9 |  429 |   13 |  246 |  196 | Installation System 9
| 2031-05-20 22:54:01 |    4 |    0 |    4 |    0 |    4 |    0 |    4 |    0 | Unknown
| ------------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | -------
| Total:         3081 | 2892 |  189 | 2985 |   96 | 3041 |   40 | 1752 | 1329 | 

PK Total    : 438
PK OK       : 432
PK ERROR    : 6
PK Raw OK   : 438
PK Raw ERROR: 0

Comments:
 80 Without comment
 23 Negative script deflate offset
 21 Possible weird stuff at WiseScriptHeader.unknown_22 '9190A09A878B'.
 14 Possible weird stuff at WiseScriptHeader.unknown_22 '958A928F8C8B'.
 10 Possible weird stuff at WiseScriptHeader.unknown_22 'B890BE889E86'.
  6 Possible weird stuff at WiseScriptHeader.unknown_22 'B6BF88BF96BF'.
  6 Possible weird stuff at WiseScriptHeader.unknown_22 'A8B0ADB3BBB6'.
  5 'Initializing Wise Installation Wizard...' found in the PE b'.data\x00\x00\x00' section.
  3 Possible weird stuff at WiseScriptHeader.unknown_22 'BCADA6CEA5CD'.
  3 Possible weird stuff at WiseScriptHeader.unknown_22 'C8CACBC8C6CC'.
  2 Possible weird stuff at WiseScriptHeader.unknown_22 '99939E9D919E'.
  2 0x18 mismatch, 0x18 00 08 ..
  2 Possible weird stuff at WiseScriptHeader.unknown_22 'DB8F9E8C8C88'.
  2 Possible weird stuff at WiseScriptHeader.unknown_22 'ADBF90BF93BF'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 '8588C9C6C88B'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 'AB969C949A8B'.
  1 0x18 mismatch, there is OP 0x00 right after 0x18
  1 Script deflate offset is absurd high
  1 Possible weird stuff at WiseScriptHeader.unknown_22 'CC918B90929D'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 'B4AAACB6B5AC'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 '9D8D9E919B96'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 '9C909B9A989A'.
  1 Possible weird stuff at WiseScriptHeader.unknown_22 '8C929E9393D1'.
  1 Overlay header unknown2 != 0x8000 but 0x0000.

Verify errors:
 75 WS_ERROR_VALUE     ERROR: parseWiseScriptHeader failed to read languageTexts.
 46 WS_ERROR_INFLATE   ERROR: Inflated size is larger then expected
 25 WS_ERROR_SEEK      ERROR: Failed seek to file offset
  9 ERROR_VALUE        ERROR: Advertised WiseScript.bin size looks insane.
  7 WS_ERROR_INFLATE   ERROR: zlib: invalid code lengths set
  4 WS_ERROR_INFLATE   ERROR: zlib: invalid distance too far back
  4 WS_ERROR_INFLATE   ERROR: zlib: invalid stored block lengths
  3 WS_ERROR_PK_HEADER ERROR: PK signature unknown
  3 WS_ERROR_VALUE     ERROR: wiseScriptParsePath expected '%' at path start
  2 WS_ERROR_VALUE     ERROR: parseWiseScript unknown OP: 3A
  2 WS_ERROR_INFLATE   ERROR: zlib: invalid block type
  2 ERROR_VALUE        ERROR: Advertised .dib size looks insane.
  1 WS_ERROR_BRANCHING ERROR: OP 0x23 LANG but not in LANG branch
  1 WS_ERROR_INFLATE   ERROR: zlib: invalid literal/length code
  1 WS_ERROR_VALUE     ERROR: This is a multi language installer but not all languages have their name set
  1 WS_ERROR_VALUE     ERROR: parseWiseScript unknown OP: 49
  1 WS_ERROR_INFLATE   ERROR: zlib: invalid distance code
  1 WS_ERROR_BRANCHING ERROR: OP 0x23 LANG but currently not in a branch
  1 ERROR_INFLATE      ERROR: zlib: invalid stored block lengths

Further inspection of the results file can be done using something that can view .csv files.

Write debug scripts

I've written a lot of small scripts for debugging specific stuff, like printing a list of all PE build dates and the overlay offset(s) observed. Such things are very specific and I didn't find it worth including, but here is a example:

import os
from installers import installerStatsCsvIt

found = {}
for installer in installerStatsCsvIt("new_results.csv"):
  if installer.isNe:
    continue

  fdate = installer.formattedDate()
  if fdate not in found:
    found.update({fdate: []});

  hexoverlay = hex(installer.overlay)
  if hexoverlay not in found[fdate]:
     found[fdate].append(hexoverlay)

for pebuild, offsets in sorted(found.items()):
  print(pebuild, offsets)

Files

test.py

usage: test.py [-h] {download,init,update,create-diff,filter-invalid,update-comments,print-diff,print-stats,result-to-source} ...

options:
  -h, --help            show this help message and exit

COMMANDS:
  Note: use -h after a command to get more help on that command

  {download,init,update,create-diff,filter-invalid,update-comments,print-diff,print-stats,result-to-source}
    download            Download installers from the given SOURCE file when needed
    init                Run initial test, the output will be written to the given RESULT file. The initial test does include testing against REWise v0.2
    update              Add new installers from given SOURCE and re-run tests for all installers, excluding the REWise v0.2 tests. To only add new and run tests for new installers give the '--only-new' argument.
    create-diff         Re-run tests of first given RESULT, ouput to second given RESULT file.
    filter-invalid      Filter out results that appear to be invalid files / not Wise installers.
    update-comments     Update comments for given RESULT file.
    print-diff          Compare two RESULT files and print the installers that it fixed/broke.
    print-stats         Print some stats for the given RESULT file.
    result-to-source    Convert RESULT file into SOURCE file.

module analyze.py

Analyze errors and add comments.

module dl.py

For downloading files.

module sources.py

Handle reading source .csv files.

The columns:

1. B3 (Blake 3)
2. MD5
3. Filesize in bytes
4. Download URL

module installers.py

Handle reading/writing result .csv files.

module rewise.py

This module handles testing and output parsing of REWise current dev and REWise v0.2.0.