The Comic Vine online database cannot be reached on Windows 7 or 8

[Lotran]

Such a problem has already occurred; it was solved with updates. Can anyone suggest what to do?

[cbanack]

I just checked and the Comic Vine API (the online database) is currently online and working for me. I has been known to go down for a few hours every now and then, which might have caused your problem accessing it.

Try it again, if it is still not working for you, odds are good that you have some kind of networking issue/firewall setting on your computer that is stopping programs like the Comic Vine Scraper from accessing the ComicVine API, which is located here: https://comicvine.gamespot.com/api/

[Lotran]

Thanks for the answer. The problem has been going on for two days now. Firewall checked first of all, there is access to the site comicvine, API status "Current API Usage You have no current rate limits. Go forth and query."

[cbanack]

Hmm, you could try running the scraper, and when you get the error dialog, press Ctrl-Shift-L (or maybe Ctrl-Alt-L, or just Ctrl-L, I can't remember exactly.) It should let you save out a text file with a log of your scraping session. The end of that log may provide a clue as to why you can't reach the database.

[n-99]

I've got the same problem, and Comicvine says I'm not limited in any way. None of the shortcuts produce any log of any kind anywhere.

[cbanack]

I just checked my notes, it is definitely Ctrl-Shift-L, make sure when you press it that the scraper has the "keyboard focus" (just click somewhere on the error dialog before you press Ctrl-Shift-L, and it should pop up a dialog to let you save a file, which contains the log.

[Lotran]

Works again, no errors. Thanks for the help

[cbanack]

Glad to hear it.

[n-99]

It again doesn't work, for at least the last three days.

[cbanack]

I checked it this morning and again tonight, and everything seems to be working ok for me.

If you are still unable to connect, there may be something about your computer or network setup that is blocking the scraper's access to the ComicVine API. The most common culprit is a firewall setting that you may have turned on by accident. You could try temporarily disabling your firewall to see if that helps...

[n-99]

There's no firewall (unless there's something weird going on at the ISP level), and I haven't changed anything. Strangely enough, for one of the scrapes, I got in the volume select dialogue, but only for one specific file. For those files that have a volume/issue ID, the progress bar goes one step in the middle before showing me the error message, for others it doesn't. The API key also isn't limited.

[cbanack]

Can you try to get the error, and then press Ctrl-Shift-L while the scraper dialog has focus? It should offer you a chance to save an error log, which should provide more clues as to what's going on. You could post it here, and I can look and see if anything stands out to me.

[kas2113]

I also have the same issue for the last 5 days. No firewall, API status fine and downgrading to 1.00.099 and 1.00.100 produces the same result ("Cannot access online database"). This is the error log

---

CV Scraper Version: 1.0.101
Running As: ComicRack Plugin (CR version 0.9.178)
Cache Directory: C:\Users\user1\AppData\Roaming\Comic Vine Scraper\localCache
Settings File: C:\Users\user1\AppData\Roaming\Comic Vine Scraper\settings.dat

---

[X] Series [X] Volume [X] Number
[X] Title [X] Published [X] Released
[X] Crossovers [X] Publisher [X] Imprint
[X] Writer [X] Penciller [X] Inker
[X] Colorist [X] Letterer [X] Cover Art
[X] Editor [X] Summary [X] Characters
[X] Teams [X] Locations [X] Webpage

[X] Overwrite Existing [ ] Ignore Blanks
[X] Convert Imprints [ ] Autochoose Series
[X] Download Thumbs [X] Preserve Thumbs
[ ] Confirm Issues [X] Rescraping: Notes
[X] Fast Rescrape [X] Rescraping: Tags
[X] Summary Dialog

======> scraping next comic book: 'Darth Vader v3 39.cbz'
searching for series that match 'Darth Vader'...
ERROR OCCURRED CONTACTING COMICVINE. RETRYING...
------------------- PYTHON ERROR ------------------------
Caught DatabaseConnectionError: Comic Vine database could not be reached
url: http://comicvine.gamespot.com/api/search/?api_key=...&client=cvscraper&format=xml&limit=100&resources=volume&field_list=name,start_year,publisher,id,image,count_of_issues&query=darth%20vader
CAUSE: System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Scripting.Interpreter.FuncCallInstruction2.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.LightLambda.Run3[T0,T1,T2,TRet](T0 arg0, T1 arg1, T2 arg2) at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1) at IronPython.Compiler.Ast.CallExpression.Invoke0Instruction.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.LightLambda.Run2[T0,T1,TRet](T0 arg0, T1 arg1) at Microsoft.Scripting.Interpreter.FuncCallInstruction4.Run(InterpretedFrame frame)
at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame)
at Microsoft.Scripting.Interpreter.LightLambda.Run4[T0,T1,T2,T3,TRet](T0 arg0, T1 arg1, T2 arg2, T3 arg3)
at Microsoft.Scripting.Interpreter.DynamicInstruction`4.Run(InterpretedFrame frame)
at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame)
Traceback (most recent call last):
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 142, in scrape
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 257, in _ScrapeEngine__scrape
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 465, in _ScrapeEngine__scrape_book
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\scrapeengine.py", line 777, in _ScrapeEngine__query_series_refs
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\db.py", line 156, in query_series_refs
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvdb.py", line 142, in _query_series_refs
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvdb.py", line 165, in __query_series_refs
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvconnection.py", line 65, in _query_series_ids_dom
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvconnection.py", line 217, in __get_dom
File "C:\Users\user1\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\cvconnection.py", line 174, in __get_dom

wrote debug logfile: cvs-debug-log-2023-10-14.txt

[cbanack]

Can you confirm that this is this still happening for you? If so, I might have an idea what's causing it, this problem here, but that's something I'll have to fix on the code in my end.

[kas2113]

Yes, still unable to reach the comic vine database.

[cbanack]

I'll experiment with it when I have a chance, and see if I can fix it.

[cbanack]

I spent some time looking into this bug (and trying to duplicate it so that I could fix it), and I'm having a very hard time getting it to happen. My original thought about what was going wrong was incorrect so I'm running low on other ideas. The error message you're getting ("System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.") is a pretty generic error that happens when your computer cannot connect to the ComicVine servers for almost any reason, so it's not much help.

Are you using an very old version of windows, perhaps? MAYBE your PC is somehow stuck on an old version of the .NET framework (.NET is needed to run both ComicRack and the Scraper...but it's supposed to get updated by Windows automatically.)

One thing you could try is downloading and installing the ".NET Framework 4.8 runtime" (not the "Developer Pack") from here: https://dotnet.microsoft.com/en-us/download/dotnet-framework/net481

That should at least make sure that you have the latest .NET framework installed, in case you don't. That might have a slim chance of fixing the problem you're experiencing.

[Bloodwah]

Seems to only fail on Windows 7 (still works fine on 10+)

[kas2113]

This is probably the issue since I am on Windows 8.1 and NET Framework 4.8.1 is not supported in my OS. I guess it's time for me to upgrade. Thank you so much for your answer, I appreciate that.

[giotte]

Just adding to confirm that the underlying issue is that older Windows versions do not support the cipher suites required by comicvine website.

If you go to https://www.ssllabs.com/ssltest/index.html and input comicvine.gamespot.com, you'll see that these are the supported ciphers for TLS 1.2 on the comicvine site:
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

Unfortunately, none of these are natively supported by any Windows versions earlier than Windows 10, which you can confirm here: https://learn.microsoft.com/en-us/windows/win32/secauthn/cipher-suites-in-schannel

Even more unfortunately, it doesn't appear possible to manually add cipher suites to the Windows OS; the only way is via Windows updates, which seems unlikely at this point given how old v8.1 and v7 are now. From what I can tell, .NET just relies on whatever is provided by the Schannel SSP, so upgrading the .NET framework won't help. Connecting to the website on older Windows versions is possible via Chrome, Firefox, etc. because these browsers have their own client and do not rely on Windows Schannel.

So, in summary; comicvine recently changed their website to require cipher suites that are not supported in older versions of Windows (earlier than 10), and .NET relies on the OS to provide these ciphers for SSL/TLS connections. I have tried to see if using native python to make the connection to comicvine might be possible (which is easy to do in vanilla python), but so far no luck, as ComicRack uses IronPython, which doesn't seem to use an up-to-date version of OpenSSL and thus does not have the necessary cipher suites. Maybe someone more versed in the details of IronPython / ComicRack can think of a possible solution.

[cbanack]

@giotte : thanks very much for your research and insight here. I agree with everything you've found, it does appear that ComicRack (Gamespot) has "hardened" their servers to require only the latest TLS 1.2 ciphers, which are not installed on windows 7 or 8.

The Comic Vine Scraper uses Ironpython, which in turn uses .NET, which in turn uses Windows OS to obtain the ciphers it can use, so I'm pretty sure the only way to solve the problem would be to somehow get those newer ciphers onto an old versions of Windows. I googled around a bit looking for any way to do this, but I couldn't find one.

Unfortunately, that pretty much means that users of Windows 7/8 will need to upgrade to at least Windows 10 (as you suggested) in order to keep using Comic Vine Scraper.

I will leave this ticket open for a while for further discussion, especially if anyone out there knows a way to get .NET to use these ciphers on Windows 7/8. Barring that, I think there's not much else to be done here. :(

[giotte]

I found a way to allow Comic Vine Scraper (CVS) on Windows 7/8.1 systems to connect to the Comic Vine website by updating the code to use a third party library instead of the built-in .NET WebRequest. The solution is a bit kludgy and this issue probably only affects a small number of users, so I'm not suggesting that CVS be updated. I'm just sharing this in case it's useful to anyone. I've tested this on Windows 7 SP1 x64 and Windows 8.1 x64 using ComicRack v0.0.178 64bit and CVS 1.0.101, but obviously no guarantees here and your mileage may vary, etc.

1. Obtain the CurlThin binaries and place the files in a folder called "CurlThin" and place that folder in the same directory as CVS
Typically the path will look something like C:\Users\USERNAME\AppData\Roaming\cYo\ComicRack\Comic Vine Scraper\CurlThin
CurlThin is a .NET wrapper of libcurl and allows for making TLS/SSL connections using recent cipher suites in IronPython by using clr.AddReferenceToFileAndPath to the DLL files. You need both the CurlThin and CurlThin.Native packages, which you can get from NuGet here and here, after which you can extract and copy the DLLs to the CurlThin folder described above. Then you need to run CurlResources.Init() in an IronPython interpreter, which extracts additional supporting files (libcurl, libcrypto, libssl, and a certificate) and then move those extracted files to the same CurlThin folder with the DLLs. Alternatively, you can obtain all of these files from this github fork here.
CurlThin requires .NET Standard 2.0, so if you don't have it installed you may get errors about "Could not load file or assembly 'netstandard, Version=2.0.0.0, ...". In that case, you can fix it by installing .NET Framework 4.7.1 Runtime, available here

2. Update cvdb.py so that _query_image uses CurlThin to connect to ComicVine
You can see the updates here or download the updated file: https://github.com/giotte/comic-vine-scraper/blob/master/src/py/database/comicvine/cvdb.py

3. Update utils.py so that get_html_string uses CurlThin to connect to ComicVine
You can see the updates here or download the updated file:
https://github.com/giotte/comic-vine-scraper/blob/master/src/py/utils/utils.py

4. Update cvconnection.py by doing a find and replace for all instances of "http://" to "https://"
If you're using the latest version of cvconnection.py on the comic-vine-scraper master branch on github, you don't need to do this last step because these changes have already been made. But if you're using the most recent Release version of Comic Vine Scraper (1.0.101), you'll need to make these updates yourself.

That's it. Just happy that I can continue to use Comic Vine Scraper without needing to upgrade the entire OS!

[cbanack]

This is great, thanks a lot @giotte! When I have some time (and that may be a while), I'll take a closer look at what you have and see if I can create a new version of the scraper that uses CurlThin to support older versions of Windows.

[n-99]

Just tried @giotte 's solution.

My directory is somewhere different,
C:\Users\USERNAME\AppData\Roaming\cYo\ComicRack\Scripts\Comic Vine Scraper\CurlThin
instead of
C:\Users\USERNAME\AppData\Roaming\cYo\ComicRack\Comic Vine Scraper\CurlThin
but that shouldn't matter, I assume.

I tried various combinations of moving the files around, or changing the paths in the two Python files, both relative and absolute. I first get a "file does not exist: PATHTOTTHEFILEFROMTHEPYFILE" error, then "'ScopeStorage' object has no attribute 'cvs_scrape'" on subsequent files until the next restart of the application.