 drak@kaverne
|
49b7c21978
use par-map to crawl
|
4 anni fa |
|
drak@kaverne
|
8c7b3fc40b
download-by-date-hint: add yearly debugging output
|
8 anni fa |
|
drak@kaverne
|
5e7ef85dd5
FIX: use the month of the date as month, not the day
|
8 anni fa |
|
drak@kaverne
|
d8ea6c9568
datehint: only 10 downloads per processor
|
8 anni fa |
|
drak@kaverne
|
57b52c7ad3
add commented-out example for crawling one seed
|
8 anni fa |
|
drak@kaverne
|
3ab8234dd8
always access the latest version
|
8 anni fa |
|
drak@kaverne
|
fa4fea714c
month starts at 0
|
8 anni fa |
|
drak@kaverne
|
cbfcc38e14
remove limitation to 10 known IDs
|
8 anni fa |
|
drak@kaverne
|
49c5a2cdd7
add status for dl by date hint
|
8 anni fa |
|
drak@kaverne
|
2b7372f28d
avoid weeks earlier than the date in the yearly date hint
|
8 anni fa |
|
drak@kaverne
|
f07225a528
fix stack overflow: replace flatten with map lambda map
|
8 anni fa |
|
drak@kaverne
|
c864340ed7
flatten the return value of the crawled wot
|
8 anni fa |
|
drak@kaverne
|
ce0ba716ae
allow passing a key as seed-id
|
8 anni fa |
|
drak@kaverne
|
56f17e968e
fix: forgot to give string-take the argument
|
8 anni fa |
|
drak@kaverne
|
d1f9344dfe
debug: wot-uri-key broke when getting a key.
|
8 anni fa |
|
drak@kaverne
|
1b695c033a
crawn: only redownload if #:redownload #t
|
8 anni fa |
|
drak@kaverne
|
886544a5ab
clearer if
|
8 anni fa |
|
drak@kaverne
|
bd17affe4a
comment out debug output
|
8 anni fa |
|
drak@kaverne
|
b297e40a5c
crawl the whole WoT.
|
8 anni fa |
|
drak@kaverne
|
a8963c08f6
Use par-map + fix crawl-wot breakage
|
8 anni fa |
|
drak@kaverne
|
8b8824ccf2
Refactor + crawl the full WoT starting at the seed-id
|
8 anni fa |
|
drak@kaverne
|
39822b1d73
Crawling the full WoT by date hint works
|
8 anni fa |
|
drak@kaverne
|
5b93eb8336
Provide more output.
|
8 anni fa |
|
drak@kaverne
|
50ec3fccdd
Crawl all versions and save with the date from the weekly date hint.
|
8 anni fa |
|
drak@kaverne
|
8f714b5460
hopefully the final fix
|
8 anni fa |
|
drak@kaverne
|
85f117ebd2
workaround for a bug I don’t understand.
|
8 anni fa |
|
drak@kaverne
|
1cdc7381fd
add: first argument gives seed
|
8 anni fa |
|
drak@kaverne
|
3dd9b02de4
mostly working crawler
|
8 anni fa |
|
drak@kaverne
|
0ca356080b
now actually working crawler
|
8 anni fa |
|
drak@kaverne
|
7640811104
working WoT crawler
|
8 anni fa |
