diff --git a/cache/priorities.xml.example b/cache/priorities.xml.example index 397697ac..ee795259 100644 --- a/cache/priorities.xml.example +++ b/cache/priorities.xml.example @@ -106,4 +106,9 @@ esgamelist screenscraper + + import + esgamelist + screenscraper + diff --git a/docs/CACHE.md b/docs/CACHE.md index d40f78ae..ffd8a4c5 100644 --- a/docs/CACHE.md +++ b/docs/CACHE.md @@ -10,7 +10,7 @@ The default base folder for all of Skyscrapers' locally cached data is in the `/ **Resource and scraping module priorities** -There is ONE file that you can and should edit inside each of the `/home//.skyscraper/cache/` folders. That file is called `priorities.xml` and decides the scraper priority of resources for each resource type. For instance, if you know that `thegamesdb` always provides the best `descriptions` for games, you'd add an `` node with a `thegamesdb` subnode. You can have multiple `` nodes, Skyscraper will then prefer the topmost source when generating a game list. If the topmost isn't found it'll prioritize the next one and so forth. Any source that isn't listed with an `` node will be prioritized using timestamps for when each resource was added to the cache. So you don't _have_ to add all of them. +There is ONE file that you can and should edit inside each of the `/home//.skyscraper/cache/` folders. That file is called `priorities.xml` and decides the scraper priority of resources for each resource type. For instance, if you know that `thegamesdb` always provides the best `descriptions` for games, you'd add an `` node with a `thegamesdb` subnode. You can have multiple `` nodes, Skyscraper will then prefer the topmost source when generating a game list. If the topmost isn't found it'll prioritize the next one and so forth. Any source that isn't listed with an `` node will be prioritized using timestamps (newest wins) for when each resource was added to the cache. So you don't _have_ to add all of them. Skyscraper provides the example file `/home//.skyscraper/cache/priorities.xml.example`. Please don't edit this file manually, as it will be overwritten when you update Skyscraper. When a platform is scraped for the first time, it will automatically copy the example file to `/home//.skyscraper/cache//priorities.xml` unless it already exists. You can of course also copy the file yourself before scraping a platform. If you do so, be sure to remove the `.example` part of the filename so it's just called `priorities.xml`. diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index ca084859..00a52b51 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,13 +1,20 @@ ## Changes +### Version 3.12.0 (TBA) + +- Added: Support for scraping of PDF manuals (Modules screenscraper, import and + esgamelist) and gamelist output with these manuals for frontends (ES-DE + Frontend, some EmulationStation variants). See configurations options + [`manuals=true`](CONFIGINI.md#manuals) and + [`gameListVariants=enable-manuals`](CONFIGINI.md#gamelistvariants). Thanks for + the initial PR, @pandino + ### Version 3.11.0 (2024-04-15) - Added: Support for EmulationStation Desktop Edition (ES-DE Frontend). Use - [`frontend=esde`](http://localhost:8000/skyscraper/CONFIGINI/#frontend) in - `config.ini` and see - [documentation](http://localhost:8000/skyscraper/FRONTENDS/#emulationstation-desktop-edition-es-de) - on the default settings. Thanks for the hints and for testing, @maxexcloo, - @Nargash + [`frontend=esde`](CONFIGINI.md#frontend) in `config.ini` and see + [documentation](FRONTENDS.md#emulationstation-desktop-edition-es-de) on the + default settings. Thanks for the hints and for testing, @maxexcloo, @Nargash - Added: Entries in [`aliasMap.csv`](https://github.com/Gemba/skyscraper/blob/master/aliasMap.csv) are now also applicable for Screenscraper. Thanks, @retrobit. diff --git a/docs/CLIHELP.md b/docs/CLIHELP.md index 3197913a..caabe089 100644 --- a/docs/CLIHELP.md +++ b/docs/CLIHELP.md @@ -442,6 +442,10 @@ This flag forces Skyscraper to use the filename (excluding extension) instead of When gathering data from any of the scraping modules many potential entries will be returned. Normally Skyscraper chooses the best entry for you. But should you wish to choose the best entry yourself, you can enable this flag. Skyscraper will then list the returned entries and let you choose which one is the best one. +#### manuals + +By default Skyscraper doesn't scrape and cache game manuals resources because not all scraping sites provide this data and also only some frontends support PDF display of these game manuals. You can enable it by using this flag. Consider setting this in [`config.ini`](CONFIGINI.md#manuals) instead. + #### nobrackets Use this flag to disable any bracket notes when generating the game list. It will disable notes such as `(Europe)` and `[AGA]` completely. This flag is only relevant when generating the game list. It makes no difference when gathering data into the resource cache. Consider setting this in [`config.ini`](CONFIGINI.md#brackets) instead. @@ -518,6 +522,10 @@ Only relevant when generating an EmulationStation, a Retrobat or a Pegasus game When generating gamelists, skip processing covers that already exist in the media output folder. +#### skipexistingmanuals + +When generating gamelists, skip copying manuals that already exist in the media output folder. + #### skipexistingmarquees When generating gamelists, skip processing marquees that already exist in the media output folder. diff --git a/docs/CONFIGINI.md b/docs/CONFIGINI.md index ba4de8a4..75ccf84d 100644 --- a/docs/CONFIGINI.md +++ b/docs/CONFIGINI.md @@ -83,6 +83,7 @@ This is an alphabetical index of all configuration options including the section | [frontend](CONFIGINI.md#frontend) | Y | | | | | [gameListBackup](CONFIGINI.md#gamelistbackup) | Y | | Y | | | [gameListFolder](CONFIGINI.md#gamelistfolder) | Y | Y | Y | | +| [gameListVariants](CONFIGINI.md#gamelistvariants) | | | Y | | | [hints](CONFIGINI.md#hints) | Y | | | | | [importFolder](CONFIGINI.md#importfolder) | Y | Y | | | | [includeFrom](CONFIGINI.md#includefrom) | Y | Y | | | @@ -93,6 +94,7 @@ This is an alphabetical index of all configuration options including the section | [lang](CONFIGINI.md#lang) | Y | Y | | | | [langPrios](CONFIGINI.md#langprios) | Y | Y | | | | [launch](CONFIGINI.md#launch) | Y | Y | Y | | +| [manuals](CONFIGINI.md#manuals) | Y | Y | | | | [maxFails](CONFIGINI.md#maxfails) | Y | | | | | [maxLength](CONFIGINI.md#maxlength) | Y | Y | Y | Y | | [mediaFolder](CONFIGINI.md#mediafolder) | Y | Y | Y | | @@ -959,3 +961,28 @@ However, folder data is not cached by Skyscraper, which means if you delete your Default value: false Allowed in sections: Only for frontends `[emulationstation]`, `[esde]` or `[retrobat]` + +--- + +#### manuals + +By default Skyscraper doesn't scrape and cache game manuals resources because not all scraping sites provide this data and also only some frontends support PDF display of these game manuals. If enabled Skyscraper will collect game manuals for the scraping modules that provide this data. For frontend ES-DE no further option must be set to enable the output of the PDF manuals to the appropriate folder. For other EmulationStation forks see also option [gameListVariants](CONFIGINI.md#gamelistvariants). + +Default value: false +Allowed in sections: `[main]`, `[]` + +--- + +#### gameListVariants + +This is a comma separated list of options for the different gamelist variants used by the various EmulationStation forks. Currently only `enable-manuals` is evaluated as variant: It generates `` entries in the gamelist for the game manuals scraped or found in the cache, if also the manuals configuration option is enabled. This option is not needed for the ES-DE frontend to output game manuals. + +**Example(s)** + +```ini +[emulationstation] +gameListVariants="enable-manuals" +``` + +Default value: unset +Allowed in sections: Only for frontend `[emulationstation]` diff --git a/docs/IMPORT.md b/docs/IMPORT.md index 4459280d..901fe6f9 100644 --- a/docs/IMPORT.md +++ b/docs/IMPORT.md @@ -10,12 +10,13 @@ The following describes how to import your own custom textual, artwork and / or Be sure to also check the `--cache edit` option [here](CLIHELP.md#--cache-editnewtype). -### Images and Videos +### Images, Videos and Game Manuals To import videos or images into the resource cache, use the following procedure: - Name your image or video file with the _exact_ base name of the rom you wish to connect it to. Example: `Bubble Bobble.nes` will import images with a filename of `Bubble Bobble.jpg` or `Bubble Bobble.png` or other well-known image formats. As long as the base name is an _exact_ match. Same goes for video files. I recommend only making use of well-known video formats since Skyscraper imports them directly without conversion (unless you convert them as described [here](CONFIGINI.md#videoconvertcommand)), so they need to be supported directly by the frontend you plan to use. -- Place all of your images or videos in the `/home//.skyscraper/import/screenshots`, `covers`, `wheels`, `marquees` or `videos` folders. +- Game manuals are expected to use PDF format and have the extension `.pdf`. The base name must match the ROM file, thus the game manual of the example is `Bubble Bobble.pdf`. +- Place all of your images, videos or game manuals in the `/home//.skyscraper/import//screenshots`, `covers`, `wheels`, `marquees`, `videos` or `manuals` folders. - Now run Skyscraper with `Skyscraper -p -s import`. If you named your files correctly, they will now be imported. Look for the green 'YES' in the output at the rom(s) you've placed files for. This will tell you if it succeeded or not. - The data is now imported into the resource cache. To make use of if read [here](#how-to-actually-use-the-data). diff --git a/docs/SCRAPINGMODULES.md b/docs/SCRAPINGMODULES.md index 9e80bd62..32822829 100644 --- a/docs/SCRAPINGMODULES.md +++ b/docs/SCRAPINGMODULES.md @@ -18,14 +18,14 @@ Below follows a description of all scraping modules. - API request limit: _20k per day for registered users_ - Thread limit: _1 or more depending on user credentials_ - Platform support: _[Check list under "Systémes"](https://www.screenscraper.fr)_ or see `screenscraper_platforms.json` sibling to your `config.ini` -- Media support: _`cover`, `screenshot`, `wheel`, `marquee`, `video`_ +- Media support: _`cover`, `screenshot`, `wheel`, `manual`, `marquee`, `video`_ - Example use: `Skyscraper -p snes -s screenscraper` ScreenScraper is probably the most versatile and complete retro gaming database out there. It searches for games using either the checksums of the files or by comparing the _exact_ file name to entries in their database. It can be used for gathering data for pretty much all platforms, but it does have issues with platforms that are ISO based. Still, even for those platforms, it does locate some games. -It has the best support for the `wheel` and `marquee` artwork types of any of the databases, and also contains videos for a lot of the games. +It has the best support for the `wheel` and `marquee` artwork types of any of the databases, and also contains videos and manuals for a lot of the games. I strongly recommend supporting them by contributing data to the database, or by supporting them with a bit of money. This can also give you more threads to scrape with. diff --git a/src/abstractfrontend.h b/src/abstractfrontend.h index 619d270d..a740d44d 100644 --- a/src/abstractfrontend.h +++ b/src/abstractfrontend.h @@ -59,6 +59,7 @@ class AbstractFrontend : public QObject { virtual QString getMarqueesFolder() { return QString(); }; virtual QString getTexturesFolder() { return QString(); }; virtual QString getVideosFolder() { return QString(); }; + virtual QString getManualsFolder() { return QString(); }; virtual void sortEntries(QList &gameEntries); protected: diff --git a/src/abstractscraper.cpp b/src/abstractscraper.cpp index f09e0c4a..baeabd17 100644 --- a/src/abstractscraper.cpp +++ b/src/abstractscraper.cpp @@ -149,11 +149,17 @@ void AbstractScraper::populateGameEntry(GameEntry &game) { getVideo(game); } break; + case MANUAL: + if (config->manuals) { + getManual(game); + } + break; default:; } } } +// TODO: openretro and worldofspectrum void AbstractScraper::getDescription(GameEntry &game) { if (descriptionPre.isEmpty()) { return; @@ -176,6 +182,7 @@ void AbstractScraper::getDescription(GameEntry &game) { game.description = StrTools::stripHtmlTags(game.description); } +// TODO: openretro and worldofspectrum void AbstractScraper::getDeveloper(GameEntry &game) { for (const auto &nom : developerPre) { if (!checkNom(nom)) { @@ -188,6 +195,7 @@ void AbstractScraper::getDeveloper(GameEntry &game) { game.developer = data.left(data.indexOf(developerPost.toUtf8())); } +// TODO: openretro and worldofspectrum void AbstractScraper::getPublisher(GameEntry &game) { if (publisherPre.isEmpty()) { return; @@ -203,6 +211,7 @@ void AbstractScraper::getPublisher(GameEntry &game) { game.publisher = data.left(data.indexOf(publisherPost.toUtf8())); } +// TODO: openretro and worldofspectrum void AbstractScraper::getPlayers(GameEntry &game) { if (playersPre.isEmpty()) { return; @@ -218,6 +227,7 @@ void AbstractScraper::getPlayers(GameEntry &game) { game.players = data.left(data.indexOf(playersPost.toUtf8())); } +// TODO: only for html scrape modules (currently none) void AbstractScraper::getAges(GameEntry &game) { if (agesPre.isEmpty()) { return; @@ -233,6 +243,7 @@ void AbstractScraper::getAges(GameEntry &game) { game.ages = data.left(data.indexOf(agesPost.toUtf8())); } +// TODO: openretro and worldofspectrum void AbstractScraper::getTags(GameEntry &game) { if (tagsPre.isEmpty()) { return; @@ -248,6 +259,7 @@ void AbstractScraper::getTags(GameEntry &game) { game.tags = data.left(data.indexOf(tagsPost.toUtf8())); } +// TODO: openretro and worldofspectrum void AbstractScraper::getRating(GameEntry &game) { if (ratingPre.isEmpty()) { return; @@ -270,6 +282,7 @@ void AbstractScraper::getRating(GameEntry &game) { } } +// TODO: openretro and worldofspectrum void AbstractScraper::getReleaseDate(GameEntry &game) { if (releaseDatePre.isEmpty()) { return; @@ -286,6 +299,7 @@ void AbstractScraper::getReleaseDate(GameEntry &game) { data.left(data.indexOf(releaseDatePost.toUtf8())).simplified(); } +// TODO: openretro and worldofspectrum void AbstractScraper::getCover(GameEntry &game) { if (coverPre.isEmpty()) { return; @@ -312,6 +326,7 @@ void AbstractScraper::getCover(GameEntry &game) { } } +// TODO: openretro only void AbstractScraper::getScreenshot(GameEntry &game) { if (screenshotPre.isEmpty()) { return; @@ -340,6 +355,7 @@ void AbstractScraper::getScreenshot(GameEntry &game) { } } +// TODO: only for html scrape modules (currently none) void AbstractScraper::getWheel(GameEntry &game) { if (wheelPre.isEmpty()) { return; @@ -366,6 +382,7 @@ void AbstractScraper::getWheel(GameEntry &game) { } } +// TODO: openretro only void AbstractScraper::getMarquee(GameEntry &game) { if (marqueePre.isEmpty()) { return; @@ -392,6 +409,7 @@ void AbstractScraper::getMarquee(GameEntry &game) { } } +// TODO: only for html scrape modules (currently none) void AbstractScraper::getTexture(GameEntry &game) { if (texturePre.isEmpty()) { return; @@ -421,6 +439,7 @@ void AbstractScraper::getTexture(GameEntry &game) { } } +// TODO: only for html scrape modules (currently none) void AbstractScraper::getVideo(GameEntry &game) { if (videoPre.isEmpty()) { return; diff --git a/src/abstractscraper.h b/src/abstractscraper.h index fd98f625..2e0a6584 100644 --- a/src/abstractscraper.h +++ b/src/abstractscraper.h @@ -82,6 +82,7 @@ class AbstractScraper : public QObject { virtual void getTexture(GameEntry &game); virtual void getTitle(GameEntry &); virtual void getVideo(GameEntry &game); + virtual void getManual(GameEntry &game) { (void)game; }; virtual void nomNom(const QString nom, bool including = true); bool checkNom(const QString nom); diff --git a/src/arcadedb.cpp b/src/arcadedb.cpp index c29b75c3..ca7779e4 100644 --- a/src/arcadedb.cpp +++ b/src/arcadedb.cpp @@ -173,7 +173,7 @@ void ArcadeDB::getVideo(GameEntry &game) { game.videoData.length() > 4096) { game.videoFormat = "mp4"; } else { - game.videoData = ""; + game.videoData = QByteArray(); } } diff --git a/src/cache.cpp b/src/cache.cpp index 02153a69..30a61c61 100644 --- a/src/cache.cpp +++ b/src/cache.cpp @@ -47,12 +47,16 @@ static inline QStringList txtTypes(bool useGenres = true) { return txtTypes; } -static inline QStringList binTypes(bool withVideo = true) { +static inline QStringList binTypes(bool withVideo = true, + bool withManual = true) { QStringList binTypes = {"cover", "screenshot", "wheel", "marquee", "texture"}; if (withVideo) { binTypes.append("video"); } + if (withManual) { + binTypes.append("manual"); + } return binTypes; }; @@ -62,7 +66,7 @@ bool Cache::createFolders(const QString &scraper) { if (scraper != "cache") { for (auto f : binTypes()) { if (!cacheDir.mkpath( - QString("%1/%2s/%3") + QString("%1/%2s/%3") // plural 's' .arg(cacheDir.absolutePath(), f, scraper))) { return false; } @@ -276,6 +280,13 @@ void Cache::printPriorities(QString cacheId) { printf("\033[1;32mYES\033[0m' (%s)\n", game.videoSrc.toStdString().c_str()); } + printf("Manual: '"); + if (game.manualSrc.isEmpty()) { + printf("\033[1;31mNO\033[0m' ()\n"); + } else { + printf("\033[1;32mYES\033[0m' (%s)\n", + game.manualSrc.toStdString().c_str()); + } printf("Description: (%s)\n'\033[1;32m%s\033[0m'", (game.descriptionSrc.isEmpty() ? QString("\033[1;31mmissing\033[0m") : game.descriptionSrc) @@ -898,7 +909,7 @@ void Cache::assembleReport(const Settings &config, const QString filter) { } else if (missingOption == "textual") { resTypeList += txtTypes(false); } else if (missingOption == "artwork") { - resTypeList += binTypes(false); // w/o 'video' + resTypeList += binTypes(false, false); // w/o 'video' or 'manual' } else if (missingOption == "media") { resTypeList += binTypes(); } else { @@ -942,6 +953,7 @@ void Cache::assembleReport(const Settings &config, const QString filter) { printf(" \033[1;32mmarquee\033[0m\n"); printf(" \033[1;32mtexture\033[0m\n"); printf(" \033[1;32mvideo\033[0m\n"); + printf(" \033[1;32mmanual\033[0m\n"); printf("\n"); return; } @@ -1149,6 +1161,7 @@ void Cache::showStats(int verbosity) { int marquees = 0; int textures = 0; int videos = 0; + int manuals = 0; for (QMap::iterator it = resCountsMap.begin(); it != resCountsMap.end(); ++it) { titles += it.value().titles; @@ -1167,6 +1180,7 @@ void Cache::showStats(int verbosity) { marquees += it.value().marquees; textures += it.value().textures; videos += it.value().videos; + manuals += it.value().manuals; } printf(" Titles : %d\n", titles); printf(" Platforms : %d\n", platforms); @@ -1184,6 +1198,7 @@ void Cache::showStats(int verbosity) { printf(" Marquees : %d\n", marquees); printf(" textures : %d\n", textures); printf(" Videos : %d\n", videos); + printf(" Manuals : %d\n", manuals); } else if (verbosity > 1) { for (QMap::iterator it = resCountsMap.begin(); it != resCountsMap.end(); ++it) { @@ -1204,6 +1219,7 @@ void Cache::showStats(int verbosity) { printf(" Marquees : %d\n", it.value().marquees); printf(" textures : %d\n", it.value().textures); printf(" Videos : %d\n", it.value().videos); + printf(" Manuals : %d\n", it.value().manuals); } } printf("\n"); @@ -1242,6 +1258,8 @@ void Cache::addToResCounts(const QString source, const QString type) { resCountsMap[source].textures++; } else if (type == "video") { resCountsMap[source].videos++; + } else if (type == "manual") { + resCountsMap[source].manuals++; } } @@ -1366,6 +1384,7 @@ void Cache::validate() { return; } + // TODO: refactor QDir coversDir(cacheDir.absolutePath() + "/covers", "*.*", QDir::Name, QDir::Files); QDir screenshotsDir(cacheDir.absolutePath() + "/screenshots", "*.*", @@ -1378,6 +1397,8 @@ void Cache::validate() { QDir::Files); QDir videosDir(cacheDir.absolutePath() + "/videos", "*.*", QDir::Name, QDir::Files); + QDir manualsDir(cacheDir.absolutePath() + "/manuals", "*.*", QDir::Name, + QDir::Files); QDirIterator coversDirIt(coversDir.absolutePath(), QDir::Files | QDir::NoDotAndDotDot, @@ -1403,6 +1424,10 @@ void Cache::validate() { QDir::Files | QDir::NoDotAndDotDot, QDirIterator::Subdirectories); + QDirIterator manualsDirIt(manualsDir.absolutePath(), + QDir::Files | QDir::NoDotAndDotDot, + QDirIterator::Subdirectories); + int filesDeleted = 0; int filesNoDelete = 0; @@ -1412,6 +1437,7 @@ void Cache::validate() { verifyFiles(marqueesDirIt, filesDeleted, filesNoDelete, "marquee"); verifyFiles(texturesDirIt, filesDeleted, filesNoDelete, "texture"); verifyFiles(videosDirIt, filesDeleted, filesNoDelete, "video"); + verifyFiles(manualsDirIt, filesDeleted, filesNoDelete, "manual"); if (filesDeleted == 0 && filesNoDelete == 0) { printf("No inconsistencies found in the database. :)\n\n"); @@ -1579,12 +1605,20 @@ void Cache::addResources(GameEntry &entry, const Settings &config, resource.value = entry.releaseDate; addResource(resource, entry, cacheAbsolutePath, config, output); } - if (entry.videoData != "" && entry.videoFormat != "") { + // TODO: refactoring (check if file extensions are needed at all, + // backward comppability) + if (!entry.videoData.isEmpty() && entry.videoFormat != "") { resource.type = "video"; resource.value = "videos/" + entry.source + "/" + entry.cacheId + "." + entry.videoFormat; addResource(resource, entry, cacheAbsolutePath, config, output); } + if (!entry.manualData.isEmpty()) { + resource.type = "manual"; + resource.value = + "manuals/" + entry.source + "/" + entry.cacheId; + addResource(resource, entry, cacheAbsolutePath, config, output); + } if (!entry.coverData.isNull() && config.cacheCovers) { resource.type = "cover"; resource.value = "covers/" + entry.source + "/" + entry.cacheId; @@ -1637,7 +1671,7 @@ void Cache::addResource(Resource &resource, GameEntry &entry, if (notFound) { bool okToAppend = true; QString cacheFile = cacheAbsolutePath + "/" + resource.value; - if (binTypes(false).contains(resource.type)) { + if (binTypes(false, false).contains(resource.type)) { QByteArray *imageData = nullptr; if (resource.type == "cover") { imageData = &entry.coverData; @@ -1730,10 +1764,20 @@ void Cache::addResource(Resource &resource, GameEntry &entry, "in '/home//.skyscraper/config.ini.'"); okToAppend = false; } + } else if (resource.type == "manual") { + QFile f(cacheFile); + if (f.open(QIODevice::WriteOnly)) { + f.write(entry.manualData); + f.close(); + } else { + output.append("Error writing file: '" + f.fileName() + + "' to cache. Please check permissions."); + okToAppend = false; + } } if (okToAppend) { - if (binTypes(false).contains(resource.type)) { + if (binTypes(false, false).contains(resource.type)) { // Remove old style cache image if it exists if (QFile::exists(cacheFile + ".png")) { QFile::remove(cacheFile + ".png"); @@ -1937,18 +1981,6 @@ void Cache::fillBlanks(GameEntry &entry, const QString scraper) { QString source = ""; QByteArray data; if (fillType(type, matchingResources, result, source)) { - if (type == "video") { - QFileInfo info(cacheDir.absolutePath() + "/" + result); - QFile f(info.absoluteFilePath()); - if (f.open(QIODevice::ReadOnly)) { - entry.videoData = f.readAll(); - f.close(); - entry.videoFormat = info.suffix(); - entry.videoFile = info.absoluteFilePath(); - entry.videoSrc = source; - } - continue; - } QFile f(cacheDir.absolutePath() + "/" + result); if (f.open(QIODevice::ReadOnly)) { data = f.readAll(); @@ -1969,6 +2001,21 @@ void Cache::fillBlanks(GameEntry &entry, const QString scraper) { } else if (type == "texture") { entry.textureData = data; entry.textureSrc = source; + } else if (type == "video" && !data.isEmpty()) { + // video is not part of artwork.xml / compositor.cpp + // set filename here + entry.videoData = data; + entry.videoSrc = source; + QFileInfo info(f); + entry.videoFormat = info.suffix(); + entry.videoFile = info.absoluteFilePath(); + } else if (type == "manual" && !data.isEmpty()) { + // manual is not part of artwork.xml / compositor.cpp + // set filename here + entry.manualData = data; + entry.manualSrc = source; + QFileInfo info(f); + entry.manualFile = info.absoluteFilePath(); } } } diff --git a/src/cache.h b/src/cache.h index 761f7533..e6a99f72 100644 --- a/src/cache.h +++ b/src/cache.h @@ -63,6 +63,7 @@ struct ResCounts { int marquees; int textures; int videos; + int manuals; }; class Cache { diff --git a/src/cli.cpp b/src/cli.cpp index ceae88d8..77fbad40 100644 --- a/src/cli.cpp +++ b/src/cli.cpp @@ -427,6 +427,9 @@ QMap Cli::getSubCommandOpts(const QString subCmd) { {"skipexistingcovers", "When generating gamelists, skip processing covers that already " "exist in the media output folder."}, + {"skipexistingmanuals", + "When generating gamelists, skip processing manuals that already " + "exist in the media output folder."}, {"skipexistingmarquees", "When generating gamelists, skip processing marquees that already " "exist in the media output folder."}, @@ -466,6 +469,9 @@ QMap Cli::getSubCommandOpts(const QString subCmd) { {"videos", "Enables scraping and caching of videos for the scraping modules " "that support them. Beware, this takes up a lot of disk space!"}, + {"manuals", + "Enables scraping and caching of manuals for the scraping modules " + "that support them."}, }; } return m; diff --git a/src/compositor.cpp b/src/compositor.cpp index 3381256b..0e1fc997 100644 --- a/src/compositor.cpp +++ b/src/compositor.cpp @@ -532,6 +532,7 @@ void Compositor::processChildLayers(GameEntry &game, Layer &layer) { QString Compositor::getSubpath(const QString &absPath) { QString subPath = "."; + // only esde expects media files in same subpath as game file if (config->frontend == "esde") { QDir inputDir = QDir(config->inputFolder); QFileInfo entryInfo(absPath); diff --git a/src/config.cpp b/src/config.cpp index b25f6935..f588e8b5 100644 --- a/src/config.cpp +++ b/src/config.cpp @@ -72,6 +72,7 @@ void Config::setupUserConfig() { skyDir.mkpath("import/marquees"); skyDir.mkpath("import/textures"); skyDir.mkpath("import/videos"); + skyDir.mkpath("import/manuals"); // Create resources folder skyDir.mkpath("resources"); @@ -141,8 +142,7 @@ void Config::setupUserConfig() { {"artwork.xml", QPair("", FileOp::CREATE_DIST)}, {"peas.json", QPair("", FileOp::CREATE_DIST)}, {"platforms_idmap.csv", - QPair("", FileOp::CREATE_DIST)} - }; + QPair("", FileOp::CREATE_DIST)}}; for (auto src : configFiles.keys()) { QString dest = configFiles.value(src).first; diff --git a/src/emulationstation.cpp b/src/emulationstation.cpp index 6a593d1c..ee5c44cf 100644 --- a/src/emulationstation.cpp +++ b/src/emulationstation.cpp @@ -128,6 +128,7 @@ void EmulationStation::preserveFromOld(GameEntry &entry) { entry.marqueeFile = oldEntry.marqueeFile; entry.textureFile = oldEntry.textureFile; entry.videoFile = oldEntry.videoFile; + // entry.manualFile on type folder does not make sense } break; } @@ -225,7 +226,8 @@ void EmulationStation::assembleList(QString &finalOutput, if (entry.isFolder && !config->addFolders && !existingInGamelist(entry)) { qDebug() << "addFolders is false, directory not added (but may be " - "preserved): " << entry.path; + "preserved): " + << entry.path; continue; } @@ -386,6 +388,10 @@ QStringList EmulationStation::createEsVariantXml(const GameEntry &entry) { vidFile = ""; } l.append(elem("video", vidFile, addEmptyElem, true)); + + if (config->manuals && config->gameListVariants.contains("enable-manuals") && !entry.manualSrc.isEmpty()) { + l.append(elem("manual", entry.manualFile, false, true)); + } return l; } @@ -424,3 +430,7 @@ QString EmulationStation::getTexturesFolder() { QString EmulationStation::getVideosFolder() { return config->mediaFolder % "/videos"; } + +QString EmulationStation::getManualsFolder() { + return config->mediaFolder % "/manuals"; +} diff --git a/src/emulationstation.h b/src/emulationstation.h index b74c63a8..3f1e0a1b 100644 --- a/src/emulationstation.h +++ b/src/emulationstation.h @@ -54,10 +54,11 @@ class EmulationStation : public AbstractFrontend { QString getMarqueesFolder() override; QString getTexturesFolder() override; QString getVideosFolder() override; + QString getManualsFolder() override; protected: virtual QStringList createEsVariantXml(const GameEntry &entry); - virtual QStringList extraGamelistTags(bool isFolder); + virtual QStringList extraGamelistTags(bool isFolder /* ignored on RP ES */); virtual GameEntry::Format gamelistFormat() { return GameEntry::Format::RETROPIE; }; diff --git a/src/esde.cpp b/src/esde.cpp index bd054abd..dbaa0f34 100644 --- a/src/esde.cpp +++ b/src/esde.cpp @@ -53,19 +53,3 @@ QString Esde::getGameListFolder() { QString Esde::getMediaFolder() { return baseFolder() % "/downloaded_media/" % config->platform; } - -QString Esde::getCoversFolder() { return config->mediaFolder % "/covers"; } - -QString Esde::getScreenshotsFolder() { - return config->mediaFolder % "/screenshots"; -} - -QString Esde::getWheelsFolder() { return config->mediaFolder % "/wheels"; } - -QString Esde::getMarqueesFolder() { return config->mediaFolder % "/marquees"; } - -QString Esde::getVideosFolder() { return config->mediaFolder % "/videos"; } - -QString Esde::getTexturesFolder() { - return config->mediaFolder % "/textures"; /* not used in esde */ -} diff --git a/src/esde.h b/src/esde.h index 40416480..784311dc 100644 --- a/src/esde.h +++ b/src/esde.h @@ -32,12 +32,6 @@ class Esde : public EmulationStation { QString getInputFolder() override; QString getGameListFolder() override; QString getMediaFolder() override; - QString getCoversFolder() override; - QString getScreenshotsFolder() override; - QString getWheelsFolder() override; - QString getMarqueesFolder() override; - QString getTexturesFolder() override; - QString getVideosFolder() override; protected: QStringList createEsVariantXml(const GameEntry &entry); diff --git a/src/esgamelist.cpp b/src/esgamelist.cpp index 35781b3c..e9794f4c 100644 --- a/src/esgamelist.cpp +++ b/src/esgamelist.cpp @@ -81,43 +81,39 @@ void ESGameList::getGameData(GameEntry &game) { game.description = gameNode.firstChildElement("desc").text(); if (config->cacheMarquees) { game.marqueeData = - loadImageData(gameNode.firstChildElement("marquee").text()); + loadBinaryData(gameNode.firstChildElement("marquee").text()); } if (config->cacheCovers) { game.coverData = - loadImageData(gameNode.firstChildElement("thumbnail").text()); + loadBinaryData(gameNode.firstChildElement("thumbnail").text()); } if (config->cacheScreenshots) { game.screenshotData = - loadImageData(gameNode.firstChildElement("image").text()); + loadBinaryData(gameNode.firstChildElement("image").text()); + } + if (config->manuals) { + game.manualData = + loadBinaryData(gameNode.firstChildElement("manual").text()); } if (config->videos) { loadVideoData(game, gameNode.firstChildElement("video").text()); } } -QByteArray ESGameList::loadImageData(const QString fileName) { - QFile imageFile(getAbsoluteFileName(fileName)); - if (imageFile.open(QIODevice::ReadOnly)) { - QByteArray imageData = imageFile.readAll(); - imageFile.close(); - return imageData; +QByteArray ESGameList::loadBinaryData(const QString fileName) { + QFile binFile(getAbsoluteFileName(fileName)); + if (binFile.open(QIODevice::ReadOnly)) { + QByteArray data = binFile.readAll(); + binFile.close(); + return data; } return QByteArray(); } void ESGameList::loadVideoData(GameEntry &game, const QString fileName) { - QString absoluteFileName = getAbsoluteFileName(fileName); - if (absoluteFileName.isEmpty()) - return; - - QFile videoFile(absoluteFileName); - if (videoFile.open(QIODevice::ReadOnly)) { - game.videoData = videoFile.readAll(); - if (game.videoData.size() > 4096) { - game.videoFormat = QFileInfo(absoluteFileName).suffix(); - } - videoFile.close(); + game.videoData = loadBinaryData(fileName); + if (game.videoData.size() > 4096) { + game.videoFormat = QFileInfo(getAbsoluteFileName(fileName)).suffix(); } } diff --git a/src/esgamelist.h b/src/esgamelist.h index 9ef61ea7..01054b4f 100644 --- a/src/esgamelist.h +++ b/src/esgamelist.h @@ -42,7 +42,7 @@ class ESGameList : public AbstractScraper { void getSearchResults(QList &gameEntries, QString searchName, QString platform) override; void getGameData(GameEntry &game) override; - QByteArray loadImageData(const QString fileName); + QByteArray loadBinaryData(const QString fileName); void loadVideoData(GameEntry &game, const QString fileName); QString getAbsoluteFileName(const QString fileName); diff --git a/src/gameentry.cpp b/src/gameentry.cpp index cd8a35c3..3b664b05 100644 --- a/src/gameentry.cpp +++ b/src/gameentry.cpp @@ -27,12 +27,15 @@ GameEntry::GameEntry() {} -void GameEntry::calculateCompleteness(bool videoEnabled) { +void GameEntry::calculateCompleteness(bool videoEnabled, bool manualEnabled) { completeness = 100.0; int noOfTypes = 13; if (videoEnabled) { noOfTypes += 1; } + if (manualEnabled) { + noOfTypes += 1; + } double valuePerType = completeness / (double)noOfTypes; if (title.isEmpty()) { completeness -= valuePerType; @@ -79,6 +82,9 @@ void GameEntry::calculateCompleteness(bool videoEnabled) { if (videoEnabled && videoFormat.isEmpty()) { completeness -= valuePerType; } + if (manualEnabled && manualData.isEmpty()) { + completeness -= valuePerType; + } } int GameEntry::getCompleteness() const { return (int)completeness; } @@ -88,5 +94,6 @@ void GameEntry::resetMedia() { screenshotData = QByteArray(); wheelData = QByteArray(); marqueeData = QByteArray(); - videoData = ""; + videoData = QByteArray(); + manualData = QByteArray(); } diff --git a/src/gameentry.h b/src/gameentry.h index 7c9b88d4..bd506987 100644 --- a/src/gameentry.h +++ b/src/gameentry.h @@ -47,7 +47,8 @@ enum : int { MARQUEE, AGES, TITLE, - TEXTURE + TEXTURE, + MANUAL }; class GameEntry { @@ -56,7 +57,7 @@ class GameEntry { GameEntry(); - void calculateCompleteness(bool videoEnabled = false); + void calculateCompleteness(bool videoEnabled = false, bool manualEnabled = false); int getCompleteness() const; void resetMedia(); @@ -100,9 +101,12 @@ class GameEntry { QByteArray textureData = QByteArray(); QString textureFile = ""; QString textureSrc = ""; - QByteArray videoData = ""; // TODO: change to QByteArray() + QByteArray videoData = QByteArray(); QString videoFile = ""; QString videoSrc = ""; + QByteArray manualData = QByteArray(); + QString manualFile = ""; + QString manualSrc = ""; // internal int searchMatch = 0; diff --git a/src/importscraper.cpp b/src/importscraper.cpp index 8ef9fbf6..bd572ba1 100644 --- a/src/importscraper.cpp +++ b/src/importscraper.cpp @@ -40,6 +40,7 @@ ImportScraper::ImportScraper(Settings *config, fetchOrder.append(MARQUEE); fetchOrder.append(TEXTURE); fetchOrder.append(VIDEO); + fetchOrder.append(MANUAL); fetchOrder.append(RELEASEDATE); fetchOrder.append(TAGS); fetchOrder.append(PLAYERS); @@ -65,6 +66,9 @@ ImportScraper::ImportScraper(Settings *config, videos = QDir(config->importFolder + "/videos", "*.*", QDir::Name, QDir::Files | QDir::NoDotAndDotDot) .entryInfoList(); + manuals = QDir(config->importFolder + "/manuals", "*.*", QDir::Name, + QDir::Files | QDir::NoDotAndDotDot) + .entryInfoList(); textual = QDir(config->importFolder + "/textual", "*.*", QDir::Name, QDir::Files | QDir::NoDotAndDotDot) .entryInfoList(); @@ -89,20 +93,21 @@ void ImportScraper::runPasses(QList &gameEntries, wheelFile = ""; marqueeFile = ""; videoFile = ""; + manualFile = ""; GameEntry game; - bool textualFound = + bool any = checkType(info.completeBaseName(), textual, textualFile); - bool screenshotFound = + any |= checkType(info.completeBaseName(), screenshots, screenshotFile); - bool coverFound = checkType(info.completeBaseName(), covers, coverFile); - bool wheelFound = checkType(info.completeBaseName(), wheels, wheelFile); - bool marqueeFound = + any |= checkType(info.completeBaseName(), covers, coverFile); + any |= checkType(info.completeBaseName(), wheels, wheelFile); + any |= checkType(info.completeBaseName(), marquees, marqueeFile); - bool textureFound = + any |= checkType(info.completeBaseName(), textures, textureFile); - bool videoFound = checkType(info.completeBaseName(), videos, videoFile); - if (textualFound || screenshotFound || coverFound || wheelFound || - marqueeFound || textureFound || videoFound) { + any |= checkType(info.completeBaseName(), videos, videoFile); + any |= checkType(info.completeBaseName(), manuals, manualFile); + if (any) { game.title = info.completeBaseName(); game.platform = config->platform; gameEntries.append(game); @@ -113,6 +118,7 @@ QString ImportScraper::getCompareTitle(const QFileInfo &info) { return info.completeBaseName(); } +// TODO: Refactor void ImportScraper::getCover(GameEntry &game) { if (!coverFile.isEmpty()) { QFile f(coverFile); @@ -175,6 +181,17 @@ void ImportScraper::getVideo(GameEntry &game) { } } +void ImportScraper::getManual(GameEntry &game) { + if (!manualFile.isEmpty()) { + QFile f(manualFile); + if (f.open(QIODevice::ReadOnly)) { + game.manualData = f.readAll(); + f.close(); + } + } +} + +// TODO: Refactor void ImportScraper::getAges(GameEntry &game) { if (isXml) { game.ages = getElementText(agesPre); diff --git a/src/importscraper.h b/src/importscraper.h index d6b77d41..626aaa76 100644 --- a/src/importscraper.h +++ b/src/importscraper.h @@ -54,6 +54,7 @@ class ImportScraper : public AbstractScraper { void getTexture(GameEntry &game) override; void getTitle(GameEntry &game) override; void getVideo(GameEntry &game) override; + void getManual(GameEntry &game) override; void getWheel(GameEntry &game) override; private: @@ -82,6 +83,7 @@ class ImportScraper : public AbstractScraper { QList marquees; QList textures; QList videos; + QList manuals; QString textualFile = ""; QString coverFile = ""; QString screenshotFile = ""; @@ -89,6 +91,7 @@ class ImportScraper : public AbstractScraper { QString marqueeFile = ""; QString textureFile = ""; QString videoFile = ""; + QString manualFile = ""; // true if definition.dat is XML style bool isXml; diff --git a/src/main.cpp b/src/main.cpp old mode 100755 new mode 100644 diff --git a/src/nametools.h b/src/nametools.h index 62410e1b..af1cc7f5 100644 --- a/src/nametools.h +++ b/src/nametools.h @@ -33,7 +33,8 @@ class NameTools : public QObject { public: - static QString getScummName(const QFileInfo &info, const QString baseName, const QString scummIni); + static QString getScummName(const QFileInfo &info, const QString baseName, + const QString scummIni); static QString getNameWithSpaces(const QString baseName); static QString getUrlQueryName(const QString baseName, const int words = -1, const QString spaceChar = "+"); diff --git a/src/scraperworker.cpp b/src/scraperworker.cpp index 7d592864..a35b21fa 100644 --- a/src/scraperworker.cpp +++ b/src/scraperworker.cpp @@ -28,6 +28,7 @@ #include "arcadedb.h" #include "compositor.h" #include "esgamelist.h" +#include "gameentry.h" #include "igdb.h" #include "importscraper.h" #include "localscraper.h" @@ -265,43 +266,11 @@ void ScraperWorker::run() { if (!config.pretend && config.scraper == "cache") { // Process all artwork compositor.saveAll(game, info.completeBaseName()); - if (config.videos && game.videoFormat != "" && - !game.videoFile.isEmpty() && QFile::exists(game.videoFile)) { - - QString videoDst = - info.completeBaseName() % "." % game.videoFormat; - QString subPath = compositor.getSubpath(game.path); - if (subPath != ".") { - videoDst = subPath % "/" % videoDst; - QFileInfo fi = - QFileInfo(config.videosFolder % "/" % videoDst); - if (!QDir().mkpath(fi.absolutePath())) { - qWarning() - << "Path could not be created" << fi.absolutePath() - << " Check file permissions, gamelist binary data " - "maybe incomplete."; - } - } - videoDst = config.videosFolder % "/" % videoDst; - - if (!(config.skipExistingVideos && QFile::exists(videoDst))) { - // Copy or symlink videos as requested - QFile::remove(videoDst); - if (config.symlink) { - if (!QFile::link(game.videoFile, videoDst)) { - game.videoFormat = ""; - } - } else { - QFile videoFile(videoDst); - if (videoFile.open(QIODevice::WriteOnly)) { - videoFile.write(game.videoData); - videoFile.close(); - } else { - game.videoFormat = ""; - } - } - } - } + // extra media files (not part of compositor) + const QString baseName = info.completeBaseName(); + const QString subPath = compositor.getSubpath(game.path); + copyMedia("video", baseName, subPath, game); + copyMedia("manual", baseName, subPath, game); } // Add all resources to the cache @@ -338,6 +307,8 @@ void ScraperWorker::run() { game.videoFile = StrTools::xmlUnescape(config.videosFolder + "/" + info.completeBaseName() + "." + game.videoFormat); + game.manualFile = StrTools::xmlUnescape( + config.manualsFolder + "/" + info.completeBaseName() + ".pdf"); game.description = StrTools::xmlUnescape(game.description); if (config.tidyDesc) { bool skipBangs = game.title.contains("!!"); @@ -467,6 +438,13 @@ void ScraperWorker::run() { : " (size exceeded, uncached)")) + " (" + game.videoSrc + ")\n"); } + if (config.manuals) { + output.append( + "Manual: " + + QString((game.manualData.isEmpty() ? "\033[1;31mNO" + : "\033[1;32mYES")) + + "\033[0m (" + game.manualSrc + ")\n"); + } output.append("\nDescription: (" + game.descriptionSrc + ")\n'\033[1;32m" + game.description.left(config.maxLength) + "\033[0m'\n"); @@ -477,7 +455,7 @@ void ScraperWorker::run() { if (!forceEnd) { forceEnd = limitReached(output); } - game.calculateCompleteness(); + game.calculateCompleteness(config.videos, config.manuals); game.resetMedia(); emit entryReady(game, output, debug); if (forceEnd) { @@ -785,6 +763,62 @@ GameEntry ScraperWorker::getEntryFromUser(const QList &gameEntries, return suggestedGame; } +void ScraperWorker::copyMedia(const QString &mediaType, + const QString &completeBaseName, + const QString &subPath, GameEntry &game) { + + const bool isVideoType = mediaType == "video"; + + const QString fmt = isVideoType ? game.videoFormat : "pdf"; + const QString fn = isVideoType ? game.videoFile : game.manualFile; + const QByteArray data = isVideoType ? game.videoData : game.manualData; + const bool mediaTypeEnabled = isVideoType ? config.videos : config.manuals; + const bool skipExisting = + isVideoType ? config.skipExistingVideos : config.skipExistingManuals; + const QString mediaTypeFolder = + isVideoType ? config.videosFolder : config.manualsFolder; + + bool noCopy = true; + if (mediaTypeEnabled && fmt != "" && !fn.isEmpty() && QFile::exists(fn)) { + QString absMediaFn = completeBaseName % "." % fmt; + if (subPath != ".") { + absMediaFn = subPath % "/" % absMediaFn; + QFileInfo fi = QFileInfo(mediaTypeFolder % "/" % absMediaFn); + if (!QDir().mkpath(fi.absolutePath())) { + qWarning() << "Path could not be created" << fi.absolutePath() + << " Check file permissions, gamelist binary data " + "maybe incomplete."; + } + } + absMediaFn = mediaTypeFolder % "/" % absMediaFn; + + if (!(skipExisting && QFile::exists(absMediaFn))) { + QFile::remove(absMediaFn); + if (config.symlink && isVideoType) { + // symlink + if (QFile::link(fn, absMediaFn)) { + noCopy = false; + } + } else { + // copy + QFile fh(absMediaFn); + if (fh.open(QIODevice::WriteOnly)) { + fh.write(data); + fh.close(); + noCopy = false; + } + } + } + } + if (noCopy) { + if (isVideoType) { + game.videoFormat = ""; + } else { + game.manualData = QByteArray(); + game.manualFile = ""; + } + } +} // --- Console colors --- // Black 0;30 Dark Gray 1;30 // Red 0;31 Light Red 1;31 diff --git a/src/scraperworker.h b/src/scraperworker.h index 3a532b49..f8c5c0a6 100644 --- a/src/scraperworker.h +++ b/src/scraperworker.h @@ -76,12 +76,17 @@ class ScraperWorker : public QObject { const int &lowestDistance); bool limitReached(QString &output); + void copyMedia(const QString &mediaType, const QString &completeBaseName, + const QString &subPath, GameEntry &game); + // TODO: Remove, replaced with AbstractScraper::MatchType + /* QStringList const directMatchScrapers = {"cache", "import", "arcadedb", "screenscraper", "esgamelist"}; QStringList const searchBasedScrapers = {"thegamesdb", "mobygames", "igdb", "worldofspectrum"}; QStringList const variableScrapers = {"openretro"}; + */ }; #endif // SCRAPERWORKER_H diff --git a/src/screenscraper.cpp b/src/screenscraper.cpp index b0cf1c64..6d0ec6bb 100644 --- a/src/screenscraper.cpp +++ b/src/screenscraper.cpp @@ -63,6 +63,7 @@ ScreenScraper::ScreenScraper(Settings *config, fetchOrder.append(MARQUEE); fetchOrder.append(TEXTURE); fetchOrder.append(VIDEO); + fetchOrder.append(MANUAL); } void ScreenScraper::getSearchResults(QList &gameEntries, @@ -352,6 +353,44 @@ QByteArray ScreenScraper::downloadMedia(const QString &url) { return QByteArray(); } +void ScreenScraper::downloadBinary(const QString &url, const QString &type, + GameEntry &game) { + bool isVideoType = type == "video"; + for (int retries = 0; retries < RETRIESMAX; ++retries) { + limiter.exec(); + netComm->request(url); + q.exec(); + if (isVideoType) { + game.videoData = netComm->getData(); + } else { + game.manualData = netComm->getData(); + } + QByteArray contentType = netComm->getContentType(); + if (netComm->getError(config->verbosity) == QNetworkReply::NoError) { + // Make sure received data is actually a video file/pdf file + if (isVideoType) { + if (contentType.contains("video/") && + game.videoData.size() > 4096) { + game.videoFormat = contentType.mid( + contentType.indexOf("/") + 1, + contentType.length() - contentType.indexOf("/") + 1); + break; + } + } else { + if (contentType.contains("application/pdf")) { + break; + } + } + } else { + if (isVideoType) { + game.videoData = QByteArray(); + } else { + game.manualData = QByteArray(); + } + } + } +} + void ScreenScraper::getCover(GameEntry &game) { QString url = ""; if (config->platform == "arcade" || config->platform == "fba" || @@ -400,28 +439,16 @@ void ScreenScraper::getVideo(GameEntry &game) { types.append("video"); QString url = getJsonText(jsonObj["medias"].toArray(), NONE, types); if (!url.isEmpty()) { - bool moveOn = true; - for (int retries = 0; retries < RETRIESMAX; ++retries) { - limiter.exec(); - netComm->request(url); - q.exec(); - game.videoData = netComm->getData(); - // Make sure received data is actually a video file - QByteArray contentType = netComm->getContentType(); - if (netComm->getError(config->verbosity) == - QNetworkReply::NoError && - contentType.contains("video/") && - game.videoData.size() > 4096) { - game.videoFormat = contentType.mid( - contentType.indexOf("/") + 1, - contentType.length() - contentType.indexOf("/") + 1); - } else { - game.videoData = ""; - moveOn = false; - } - if (moveOn) - break; - } + downloadBinary(url, types.last(), game); + } +} + +void ScreenScraper::getManual(GameEntry &game) { + QStringList types; + types.append("manuel"); + QString url = getJsonText(jsonObj["medias"].toArray(), REGION, types); + if (!url.isEmpty()) { + downloadBinary(url, types.last(), game); } } diff --git a/src/screenscraper.h b/src/screenscraper.h index 5a4ad587..2ecb97b4 100644 --- a/src/screenscraper.h +++ b/src/screenscraper.h @@ -65,10 +65,13 @@ class ScreenScraper : public AbstractScraper { void getMarquee(GameEntry &game) override; void getTexture(GameEntry &game) override; void getVideo(GameEntry &game) override; + void getManual(GameEntry &game) override; QString getJsonText(QJsonArray array, int attr, QList types = QList()); QByteArray downloadMedia(const QString &url); + void downloadBinary(const QString &url, const QString &type, + GameEntry &game); QString getUrlOrTextPropertyValue(const QJsonObject &jsonVal, const QString &key, const QString &matchValue); diff --git a/src/settings.cpp b/src/settings.cpp index 98dda291..f74851a8 100644 --- a/src/settings.cpp +++ b/src/settings.cpp @@ -212,6 +212,17 @@ void RuntimeCfg::applyConfigIni(CfgType type, QSettings *settings, config->importFolder = v; continue; } + if (k == "gameListVariants") { + if (config->frontend == "emulationstation") { + config->gameListVariants = v; + } else { + printf( + "\033[1;33mParameter %s is ignored. Only " + "applicable with frontend=emulationstation.\n\033[0m", + k.toUtf8().constData()); + } + continue; + } if (k == "includeFiles" || k == "includePattern") { if (k == "includeFiles") { printf("\033[1;33mParameter %s is deprecated! " @@ -428,6 +439,10 @@ void RuntimeCfg::applyConfigIni(CfgType type, QSettings *settings, config->videos = v; continue; } + if (k == "manuals") { + config->manuals = v; + continue; + } } else if (conv == "int") { bool intOk; int v = ss.toInt(&intOk); @@ -642,6 +657,8 @@ void RuntimeCfg::setFlag(const QString flag) { config->relativePaths = true; } else if (flag == "skipexistingcovers") { config->skipExistingCovers = true; + } else if (flag == "skipexistingmanuals") { + config->skipExistingManuals = true; } else if (flag == "skipexistingmarquees") { config->skipExistingMarquees = true; } else if (flag == "skipexistingscreenshots") { @@ -666,6 +683,8 @@ void RuntimeCfg::setFlag(const QString flag) { config->unpack = true; } else if (flag == "videos") { config->videos = true; + } else if (flag == "manuals") { + config->manuals = true; } else if (flag == "notidydesc") { config->tidyDesc = false; } else { diff --git a/src/settings.h b/src/settings.h index cb5623bf..42ec1f52 100644 --- a/src/settings.h +++ b/src/settings.h @@ -54,13 +54,14 @@ struct Settings { QString mediaFolder = ""; // Next two only relevant for EmulationStation/ES-DE bool mediaFolderHidden = false; // EmulationStation only - bool addFolders = false; // EmulationStation and ES-DE + bool addFolders = false; // EmulationStation and ES-DE QString screenshotsFolder = ""; QString coversFolder = ""; QString wheelsFolder = ""; QString marqueesFolder = ""; QString texturesFolder = ""; QString videosFolder = ""; + QString manualsFolder = ""; QString importFolder = "import"; QString nameTemplate = ""; int doneThreads = 0; @@ -112,6 +113,8 @@ struct Settings { int romLimit = -1; bool videos = false; + bool manuals = false; + QString gameListVariants = ""; bool videoPreferNormalized = true; int videoSizeLimit = 100 * 1000 * 1000; QString videoConvertCommand = ""; @@ -130,6 +133,7 @@ struct Settings { bool skipExistingMarquees = false; bool skipExistingTextures = false; bool cacheTextures = true; + bool skipExistingManuals = false; QString user = ""; QString password = ""; @@ -201,6 +205,7 @@ class RuntimeCfg : public QObject { {"gameListBackup", QPair("bool", CfgType::MAIN | CfgType::FRONTEND )}, {"gamelistFolder", QPair("str", CfgType::MAIN | CfgType::PLATFORM | CfgType::FRONTEND )}, {"gameListFolder", QPair("str", CfgType::MAIN | CfgType::PLATFORM | CfgType::FRONTEND )}, + {"gameListVariants", QPair("str", CfgType::FRONTEND )}, {"hints", QPair("bool", CfgType::MAIN )}, {"importFolder", QPair("str", CfgType::MAIN | CfgType::PLATFORM )}, {"includeFiles", QPair("str", CfgType::MAIN | CfgType::PLATFORM | CfgType::FRONTEND )}, @@ -212,6 +217,7 @@ class RuntimeCfg : public QObject { {"lang", QPair("str", CfgType::MAIN | CfgType::PLATFORM )}, {"langPrios", QPair("str", CfgType::MAIN | CfgType::PLATFORM )}, {"launch", QPair("str", CfgType::MAIN | CfgType::PLATFORM | CfgType::FRONTEND )}, + {"manuals", QPair("bool", CfgType::MAIN | CfgType::PLATFORM )}, {"maxFails", QPair("int", CfgType::MAIN )}, {"maxLength", QPair("int", CfgType::MAIN | CfgType::PLATFORM | CfgType::FRONTEND | CfgType::SCRAPER )}, {"mediaFolder", QPair("str", CfgType::MAIN | CfgType::PLATFORM | CfgType::FRONTEND )}, diff --git a/src/skyscraper.cpp b/src/skyscraper.cpp old mode 100755 new mode 100644 index ef4f38f3..0f803f24 --- a/src/skyscraper.cpp +++ b/src/skyscraper.cpp @@ -102,6 +102,10 @@ void Skyscraper::run() { printf("Videos folder: '\033[1;32m%s\033[0m'\n", config.videosFolder.toStdString().c_str()); } + if (config.manuals) { + printf("Manuals folder: '\033[1;32m%s\033[0m'\n", + config.manualsFolder.toStdString().c_str()); + } printf("Cache folder: '\033[1;32m%s\033[0m'\n", config.cacheFolder.toStdString().c_str()); if (config.scraper == "import") { @@ -229,8 +233,9 @@ void Skyscraper::run() { } config.inputFolder = inputDir.absolutePath(); - bool isCacheScraper = config.scraper == "cache" && !config.pretend; + const bool isCacheScraper = config.scraper == "cache" && !config.pretend; + // TODO: Repeating code: refactor &config.gameListFolder, isCacheScraper QDir gameListDir(config.gameListFolder); if (isCacheScraper) { checkForFolder(gameListDir); @@ -274,6 +279,13 @@ void Skyscraper::run() { } config.videosFolder = videosDir.absolutePath(); } + if (config.manuals) { + QDir manualsDir(config.manualsFolder); + if (isCacheScraper) { + checkForFolder(manualsDir); + } + config.manualsFolder = manualsDir.absolutePath(); + } QDir importDir(config.importFolder); checkForFolder(importDir, false); @@ -815,16 +827,18 @@ void Skyscraper::loadConfig(const QCommandLineParser &parser) { config.mediaFolder = rtConf->concatPath(config.gameListFolder, mf); } } + // only resolve after config.mediaFolder is set config.coversFolder = frontend->getCoversFolder(); config.screenshotsFolder = frontend->getScreenshotsFolder(); config.wheelsFolder = frontend->getWheelsFolder(); config.marqueesFolder = frontend->getMarqueesFolder(); config.texturesFolder = frontend->getTexturesFolder(); config.videosFolder = frontend->getVideosFolder(); + config.manualsFolder = frontend->getManualsFolder(); // Choose default scraper for chosen platform if none has been set yet if (config.scraper.isEmpty()) { - // TODO: is always "cache" + // TODO: is always "cache", set hardwired config.scraper = Platform::get().getDefaultScraper(); } diff --git a/src/xmlreader.cpp b/src/xmlreader.cpp index 804460e5..9da1429a 100644 --- a/src/xmlreader.cpp +++ b/src/xmlreader.cpp @@ -97,6 +97,8 @@ void XmlReader::addEntries(const QDomNodeList &nodes, if (!entry.videoFile.isEmpty()) { entry.videoFormat = "fromxml"; } + entry.manualFile = + makeAbsolute(node.firstChildElement("manual").text(), inputFolder); for (const auto &t : gamelistExtraTags) { entry.setEsExtra(t, node.firstChildElement(t).text()); diff --git a/supplementary/bash-completion/Skyscraper.bash b/supplementary/bash-completion/Skyscraper.bash index 742536c2..2b6051f6 100644 --- a/supplementary/bash-completion/Skyscraper.bash +++ b/supplementary/bash-completion/Skyscraper.bash @@ -78,7 +78,7 @@ _skyscraper() { ;; '-f') # frontends - mapfile -t COMPREPLY < <(compgen -W "emulationstation pegasus retrobat attractmode" -- "$cur") + mapfile -t COMPREPLY < <(compgen -W "emulationstation esde pegasus retrobat attractmode" -- "$cur") return 0 ;; '-t') diff --git a/test/settings/config_test.ini b/test/settings/config_test.ini index bd4a5b19..eb9143a8 100644 --- a/test/settings/config_test.ini +++ b/test/settings/config_test.ini @@ -55,6 +55,7 @@ jpgQuality="99" lang="fr" langPrios="fr,en,de,es" launch="launch__pega_main_launch" +manuals="false" maxFails="101" maxLength="4242" mediaFolder="/home/pi/RetroPie/roms/test" @@ -174,6 +175,7 @@ cacheWheels="true" cacheMarquees="true" cacheTextures="true" cacheRefresh="1" +manuals="true" videos="true" videoSizeLimit="42" videoConvertCommand="ffmpeg -i %i -y -pix_fmt yuv420p -t 00:00:10 -c:v libx264 -crf 23 -c:a aac -b:a 64k -vf scale=640:480:force_original_aspect_ratio=decrease,pad=640:480:(ow-iw)/2:(oh-ih)/2,setsar=1 %o" diff --git a/test/settings/settings_test.cpp b/test/settings/settings_test.cpp index 2ae2847e..abe91b94 100644 --- a/test/settings/settings_test.cpp +++ b/test/settings/settings_test.cpp @@ -107,6 +107,8 @@ void TestSettings::configIniMain() { QCOMPARE(config.lang, exp); exp = settings.value("langPrios"); QCOMPARE(config.langPriosStr, exp); + exp = settings.value("manuals"); + QCOMPARE(config.manuals, exp); exp = settings.value("maxFails"); QCOMPARE(config.maxFails, exp); exp = settings.value("maxLength"); @@ -258,6 +260,8 @@ void TestSettings::configIniPlatform() { QCOMPARE(config.lang, exp); exp = settings.value("langPrios"); QCOMPARE(config.langPriosStr, exp); + exp = settings.value("manuals"); + QCOMPARE(config.manuals, exp); exp = settings.value("maxLength"); QCOMPARE(config.maxLength, exp); exp = settings.value("mediaFolder"); @@ -369,6 +373,8 @@ void TestSettings::configIniFrontend() { QCOMPARE(config.includePattern, exp); exp = settings.value("maxLength"); QCOMPARE(config.maxLength, exp); + exp = settings.value("manuals"); + QCOMPARE(config.manuals, exp); exp = settings.value("mediaFolder"); QCOMPARE(config.mediaFolder, exp); exp = settings.value("mediaFolderHidden"); @@ -458,6 +464,8 @@ void TestSettings::configIniScraper() { QCOMPARE(config.interactive, exp); exp = settings.value("jpgQuality"); QCOMPARE(config.jpgQuality, exp); + exp = settings.value("manuals"); + QCOMPARE(config.manuals, exp); exp = settings.value("maxLength"); QCOMPARE(config.maxLength, exp); QCOMPARE(config.minMatch, 65);