-
-
Notifications
You must be signed in to change notification settings - Fork 73
/
Changelog
360 lines (307 loc) · 12.5 KB
/
Changelog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
1.13.0:
* FIX: Many akwards scenarios around Redis (@uriesk #1756)
* FIX: Wrongly downloaded hidden media files (@pavel-karatsiuba #1780 #849)
* FIX: Incompatibility scenario in hacking the start module (@pavel-karatsiuba #1783)
* FIX: Bad ordering in Javascript module loading at reading time (@pavel-karatsiuba #1766)
* FIX: Unpredictive behaviour depending the order of --format (@pavel-karatsiuba #1826)
* FIX: Broken auto-redirect in a few Wiktionaries (@pavel-karatsiuba #1811)
* FIX: Mosaic selection welcome page with irrelevant pictures (@pavel-karatsiuba #1794)
* FIX: Broken logo downloading because of protocol mismatch (@pavel-karatsiuba #1800)
* UPDATE: Improved container image documentation (@kelson42 #1827)
* UPDATE: Container image using latest versions of Node.js & Redis (@kelson42 #1814)
* UPDATE: Container images only pushed to ghcr.io (@rgaudin #1812)
* UPDATE: Better Axios error reporting (@FledgeXu #1795)
* NEW: Better checking of command line arguments (@pavel-karatsiuba #1823 #1806)
* NEW: Introduce option --customZimLanguage (@kelson42 #1841)
* NEW: Synchronization of UI translations with Translatewiki.net (@kelson42 #116)
* NEW: Threshold specification to the --verbose option (@pavel-karatsiuba #1785)
1.12.1:
* FIX: Bad behaviour because of wrong mime-type in S3 (@kelson42 #1772)
* FIX: Unefficient CI steps (@kelson42 1770)
1.12.0:
* NEW: Add --articleListToIgnore option (@uriesk #1706)
* NEW: Add --longDescription option (@pavel-karatsiuba #1755)
* NEW: Remove support of LZMA (@kelson42 #1643)
* NEW: Drop support of Node.js 14 (@kelson42 #1671)
* NEW: Revamping of interactions with Redis (@uriesk #1719 #1720)
* NEW: Use Jest in place of Tape for automated tests (@pavel-karatsiuba #1716)
* NEW: Introduce/migrate to usage of ESM (@pavel-karatsiuba #1688)
* NEW: Use Eslint instead Tslint (@pavel-karatsiuba #1738 #1747)
* FIX: Simplisitc HTTP retry strategy for downloading files (@uriesk #1751)
* FIX: Broken support of OGA audio files (@pavel-karatsiuba#1763)
* FIX: Problematic inline Javascript to comply with CSP (@pavel-karatsiuba #1752)
* FIX: Partly broken WEBP related mime-type mgmt (@uriesk 1723)
* FIX: Broken show/hide toggle in Wikipedia infoboxes (@zhuowei #1665)
* FIX: Handle properly Fandom images src URL (@uriesk #1712)
* FIX: Upgrade most of the dependences (@uriesk @pavel-karatsiuba #1715)
1.11.12:
* FIX: ZIM illustration now always in PNG format
* FIX: Somehow cropped ZIM illustration
* NEW: Add support for mw:File typeof
* NEW: Use keep-alive TCP flag
* UPDATE: Redis 8 & Node 18 in Docker image
1.11.11:
* FIX: Workaround wrong Wep mime-type returns sporadically by Mediawiki
* UPDATE: Better speed computation
1.11.10:
* FIX: Videos in with too low resolution
* UPDATE: Allow to have JSON error 'missingarticle' in module dependence retrieval
* UPDATE: Stop allow Mediawiki instance with misconfigured SSL
1.11.9:
* FIX: Thumbnail URL retrieval with new edge cases
* FIX: Crash in case of badly formated URL in HTML
* UPDATE: Better video picker - respect the HTML element size
* UPDATE: Reduce number of parallel requests by 30%
* UPDATE: HTTP request timeout increased to 120 seconds
1.11.8
* FIX: Redirect retrieval if lots of them
* FIX: Webp polyfill concurency problems
1.11.7
* NEWS: Better ingest Windows encoded article lists
* UPDATE: Use libzim 2.4.4
1.11.6
* FIX: Fix local Parsoid upstream URL config (regression)
1.11.5
* FIX: Better handle of score pictures
* FIX: Desktop API detection (broken in case of redirected welcome page)
* FIX: Handling of articles deleted during the scrape
* FIX: Mediawiki detection crashing if very old Mediawiki
* FIX: Crash if article name is 'constructor'
1.11.4
* NEW: Check properly for unsupported Mediawiki versions
* FIX: Remove surnumerous underscores in M/Flavour
* FIX: Various bugs around ResourceLoader
* FIX: Edge-case bugs around API detection
* FIX: Buggy links for article starting with a /
* UPDATE: Stops when an article details retrieval fails
1.11.3
* FIX: Partly broken Webp image conversion
* FIX: HTTP error status code because wrong 'accept' HTTP request header
* FIX: Bad exit code in some error cases
* UPDATE: Remove a few additional Parsoid images attributes
* NEW: Allow S3 unencrypted connections
1.11.2
* FIX: Local Parsoid for HTTP only instances
* FIX: Buggy sub-page URL rewritting
* FIX: Buggy empty paragraph removal
* FIX: Private Mediawiki login/scraping
* FIX: VisualEditor buggy detection
* FIX: WikiHiero pictures scraping
* NEW: Better --customMainPage mgmt
* NEW: Simplify CSS/JS/Media paths a bit
* NEW: Always show first single section
* NEW: Use latest libzim
1.11.1
* FIX: Buggy API end-points detection
* FIX: Weak image mime-type assignement
* FIX: Partly missing revision ids
* UPDATE: Article details retrieval speed-up
* UPDATE: Use Node.js 14 in Docker
1.11.0
* FIX: multiple bugs around redirects
* FIX: multiple regression if local Parsoid is used
* NEW: image lazy loading
* NEW: option to convert png/jpg images to WebP
1.10.12:
* NEW: Add ability to make ZIM files using Zstandard compression
1.10.11:
* UPDATE: Use node-libzim 2.4.2
* FIX: first section scraping
* FIX: custom plugin for Wiktionary
* FIX: ZIM filename date
* FIX: broken scraping with local Parsoid
1.10.10:
* UPDATE: Small memory usage improvement
* FIX: Regression by article retrieval (from namespace)
1.10.9:
* FIX: Support of video subtitles
* FIX: Run generating ZIM files with different dates in filename
* UPDATE: Better default standard Redis client conf
* UPDATE: Use node-libzim version 2.2.4
* UPDATE: Optimisation of the S3 workflow
* UPDATE: Various revamping to simplify code base
* DEL: Custom download cache
1.10.8:
* UPDATE: Use node-libzim version 2.2.3
1.10.7:
* FIX: Improving parsoid/source select logic
* FIX: Better errors and timeouts handling
* FIX: AWS S3 build errors and logging
* FIX: title on main pages and footer links
* FIX: scraping articles with question marks
* UPDATE: Add support for node v12+
* UPDATE: Migrate to node-libzim v2.2.2
* UPDATE: Migrate to Redis 6 and
* UPDATE: Migrate to sharp
* UPDATE: Remove odd metadata and article dependencies requests
* UPDATE: Add support for parametrized translatable strings + add Spanish translation
* UPDATE: Simplify automated tests call
1.10.6:
* FIX: Hotfix for TypeScript build config
1.10.5:
* FIX: Fix docker build
1.10.4:
* FIX: Optimizations/fixes for better scaling (#1025)
* FIX: Refactoring / making the code clean
1.10.3:
* FIX: Better logging while fetching the articles
1.10.2:
* FIX: Fix broken npm package
1.10.1:
* NEW: Support "http" prefixed URLs for --optimisationCacheUrl
1.10.0:
* NEW: Support of S3 --optimisationCacheUrl
1.9.13:
* NEW: Docker image has an embedded Redis daemon
* DEL: --keepHtml and --format=nozim
* UPDATE: Rename --useCache in --useDownloadCache
* UPDATE: Rename --cacheDirectory in --downloadCacheDirectory
1.9.12:
* UPDATE: Release documentation
1.9.11:
* FIX: broken '_details' tag values (#980)
* FIX: non-empty sections removal (#986)
* NEW: --osTmpDir optional argument (#966)
* NEW: --flavour for specifying pre/post processing scripts
1.9.10:
* NEW: nothing
1.9.9:
* FIX: lib imports
1.9.8:
* UPDATE: update libzim
* UPDATE: update zimcheck
* UPDATE: KaiOS Rendering Hack (#931)
* UPDATE: Use execa instead of execPromise (#948)
* UPDATE: Add canonical meta
* UPDATE: Keep inline mathml HTML (#164)
* UPDATE: Replace Kartographer maplink numbers (#871)
* FIX: Wikivoyage DE rendering (#873)
* FIX: Remove duplicate article redirects (#889)
* FIX: No-JS mobile rendering (#962)
1.9.7:
* NEW: Add 'Flavour' ZIM metadata
* NEW: revamped tag system
1.9.6:
* FEATURE: Add format aliasing option
* UPDATE: Remove skipCacheCleaning parameter
* FIX: Welcome page detection & rendering
1.9.5:
* UPDATE: Update node-libzim
* UPDATE: Cache logic (only use FS)
* UPDATE: Remove JSON request cache
* UPDATE: Specify tile layout for screens ~8096px
* FIX: Welcome page '/' rewriting
* FIX: Empty paragraph removal
1.9.4:
* UPDATE: README improvements (#880)
* UPDATE: Smarter and more efficient article fetching
* UPDATE: remove hard-coded font-sizes for ehadings (#825)
* UPDATE: write categories to 'U' namespace
* UPDATE: redis handling improvements (#448)
* UPDATE: improve code quality (#687)
* UPDATE: improve testing
* FIX: implement scraping of <audio> elements
* FIX: Thumbnail image fetching on nopic (#867)
1.9.3:
* FIX: coordinate retrieval continue bug
* FEATURE: add throttled keepalive log (#791)
* FEATURE: add scraper metadata to ZIM file (#787)
* UPDATE: only collect relevant redirects (#790)
* UPDATE: disable quirks mode for article html (#717)
* UPDATE: improve no-image articlelist home page (#759)
* UPDATE: subsections open by default (#786)
1.9.2:
* FIX: favicon writing logic
* FIX: special uris no longer re-written (#771)
* FIX: prefer remote Parsoid for fallbacks (#767)
* FIX: redirect-writing issues (#769)
* FIX: --help option (#768)
* UPDATE: improve external link handling
* UPDATE: stop retrying 404s (#600)
1.9.1:
* UPDATE: Update node-libzim (and libzim) to improve memory usage (#706)
* UPDATE: Implement basic category scraping (#15)
* UPDATE: Increase default scrape "speed" (#744)
* UPDATE: Report on failed article downloads ()
* UPDATE: Retry all failed media downloads at the end of scrape (#723)
* UPDATE: Re-introduce --useCache option (#700)
* UPDATE: Reduce system dependencies (#681)
* UPDATE: Use native <details> element for collapsible sections in ZIM files (#677)
* FIX: Update article title logic (#605)
* FIX: Logging format improvements (#662)
* FIX: Redis clearing improved (#732, #731, #448)
* FIX: Text only article list main page background colour fixed (#728)
* FIX: Resource relative url improvement (#726)
1.8.6:
* UPDATE: Parallel processing improvements
* UPDATE: Increased Parsoid and MCS limits #710
* UPDATE: Switched to native `<details>` element for collapsible sections #677
* FIX: Re-introduced automatic ZIM tagging #709
1.8.5:
* FIX: Thumbnail nopic quality improvements #693
* FIX: Add error handling for "URI malformed" error #697
* FIX: Add checks to stop "normalized of undefined" errors #698
* FIX: Clear file store between dumps #699
* UPDATE: Re-implement useCache for downloader #700
1.8.4:
* FIX: Article title logic #605
* FIX: Log formatting issue #662
* FIX: Clear Redis when exiting #448
* UPDATE: Moved "convert" utility to a node library #681
1.8.3:
* FIX: Parsoid wt2html limit increased #664
* FIX: Parsoid request timeout increased # 660
* FIX: Logging bug #662
* FIX: Remote article list fixed #661
1.8.2:
* UPDATE: Improved dump performance
* FIX: Reduce memory usage
* FIX: #654 Duplicate jsConfigVars.js
* FIX: #645 Welcome page mirroring
* FIX: #643 --customMainPage
1.8.1:
* FIX: ZIM filename language code #624
1.8.0:
* UPDATE: Removed 'tmp' files and directory #448 #575
* UPDATE: Write ZIM files directly (Using Libzim) #184
* UPDATE: Removed --deflateTmpHTML and --tmpDirectory arguments #575 #576
* UPDATE: Implemented better request backoff #496
* UPDATE: Change file names/paths #278
* UPDATE: Removed --writeHtmlRedirects argument #506
* UPDATE: Removed --localMCS option (automatically detect) #490
* UPDATE: Updated documentation #423
* FIX: Other stability, logging and error handling fixes
1.7.1:
* FIX: Broken Docker image #534
1.7.0:
* UPDATE: Deprecated desktop scraping (not --mobileLayout)
* UPDATE: Improve the article list welcome page #449
* UPDATE: Add string translation mechanism #16
* UPDATE: Fetch hundreds of redirects at a time instead of one #325
* UPDATE: Integrated MCS alongside Parsoid #388 #347 #404
* UPDATE: Introduce automated testing #146
* UPDATE: Support remote articleList and favicon #457
* UPDATE: Added --useCache option #263
1.6.0:
* FIX: Template referencing #421
* FIX: Generated files are more no-js friendly #324
* FIX: Added top level section when it doesn't exist in API Response #180
* FIX: Fixed --addNamespace #338
* FIX: CSS Image issue #418
* UPDATE: Implemented PDF downloads #338
* UPDATE: Added mwmatrixoffliner tests
* UPDATE: Added git revision to version log #426
* UPDATE: Normalised Resources and Templates
* UPDATE: Strip HTML from title tags #159
* UPDATE: Improved Error Handling
* UPDATE: Converted to TypeScript #386
1.5.0:
* FIX: Favicon related crashes #347 #354
* FIX: Improved error logging when invalid logo information is returned from the server #363
* UPDATE: Updated parsoid version to latest security patch (not latest Parsoid)
1.4.1:
* FIX: Fix support of nodejs versions with V8 core #281
* FIX: Missing external link icon with desktop layout #279
* FIX: Sporadic wrong redirects #258
* FIX: Remove empty tags #259
* FIX: Remove sections sur-numerous braces #205