-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add cookie & crumb to requests #1657
Conversation
@ValueRaider This fixes all my issues mentioned in #1649! Please merge this ASAP. |
Code will need fault-tolerance, so failure to get cookie or crumb doesn't stop use. |
Updated with @bot-unit 's crumb fetch #1729 (comment). Also, store crumb in Works great except
|
b7a78e8
to
c9680d1
Compare
I don't know if this is the right place to leave this comment. But I installed and tried this feature branch today from the US and I run into the following error:
|
c9680d1
to
649cc51
Compare
@ValueRaider - i need some guidance what you suggest I try in data.py. |
If I just put the url into my browser I get back an 11 character string. url is from data.py : https://query1.finance.yahoo.com/v1/test/getcrumb So that doesn't seem to be a problem - bug must be elsewhere. Edit: so seems to me _get_crumb() works while _get_crumb_botunit() gets error at line 108 - doesn't seem to have much to do with _get_crumb() or even any crumb best I can tell. The failure occurs parsing this: url = https://guce.yahoo.com/consent Sorry I have no clue what this part of the code is trying to do. As I said in the last message (that got deleted) - this url returns a ton of stuff but none of that includes 'csrfToken' what is csrfToken and why would it be in the .../consent' url? |
(I deleted because you're repeating stack trace above you) |
I don't know if this helps, but I'm located in the US (California), and when I go to https://guce.yahoo.com/consent I get this result:
After the redirect, I'm basically on www.yahoo.com. Here's the content of guce.txt. It doesn't contain the |
I then opened a VPN tunnel to Germany, and went to https://guce.yahoo.com/consent through that. This redirects to https://consent.yahoo.com/v2/collectConsent. See here:
For comparison, here's the source of the returned page: guce.txt. This time I do have the |
It would be helpful to know what we are actually looking for - the code looks for csrfToken which is not in the returned data. grep -i csrfToken /tmp/guce.txt returns nothing - seems to be the source of the "problem" Do you all see anything like that in the returned output? File I get is pretty long - $ wc /tmp/guce.txt |
Remember the goal: mimic what happens when you manually visit finance.yahoo.com. Firefox -> F12 -> Network tab -> F5 |
okay got it - from the US we don't get the guce csrfToken - as you all had already surmised. |
@ValueRaider , @gene-git : I'm knee-deep in
I'll try more stuff later. |
It's look like Yahoo has different politics for cookie for different country. I can test only for europe and if cookie isn't set, they ask to me to be agree with cookies. (redirect to https://guce.yahoo.com/consent) The process look like:
We can try in this order: 2, if not success, then 1 and 2. And 3 is last. How i test, crumb is valid all time while cookie is valid too |
So in my code i am using two functions before doing any yahoo requests (in Session object of Requests):
The first, i create request (https://guce.yahoo.com/consent) for form to agree with cookies, yahoo gives me session id. Then i post my agreement with session id (https://consent.yahoo.com/v2/collectConsent?sessionId=). After that i request cookies with my session id (https://guce.yahoo.com/copyConsent?sessionId=). The last step is get crumb (https://query2.finance.yahoo.com/v1/test/getcrumb) This code is working for me everyday. My software recreate new Session object of requests every day, so i repeat all steps. |
d9f7b6c
to
68bd2b5
Compare
https://query2.finance.yahoo.com/v1/test/getcrumb USA users can check if they need cookies or something else: just delete cookie from browser and then open https://query2.finance.yahoo.com/v1/test/getcrumb |
installed from Git: Works for me (UK) ..... stopped working on .info dictionary again but .get_fast_info() still works! |
Vote if works instead of replying (see top post). @pipreaper We need more info. Is there an error message? And post output with |
As of 4.15am UK time it is working on .info. If it stops working I will post the debug output.
/home/robert/PycharmProjects/get_prospects/env/bin/python /home/robert/PycharmProjects/get_prospects/tests/a_cookie and crumb/test_it.py |
I'm located in Canada, and I was getting the following when trying to call .info File "C:\Users\money\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\models.py", line 1021, in raise_for_status I installed your branch per your instructions, and not only did it solve this problem but it also fixed the cookie & crumb issues reported in many of the "failed to decrypt yahoo data response" issues that started with #1407 and have been reported in several other issues for a while now as well! IMHO the workaround to revert to the much older v6 URL was a bad decision. In any case, all errors I was getting are gone using your branch; it should be merged with the main branch to make it into the next release ASAP! |
All (UK), working well on .history and .info. Problem occurs if I enter an erroneous Ticker (I have a search box in an App). After the erroneous call to 'NEX.L' fetching history fails to retrieve data for following valid ticker entries. |
@pipreaper fixed |
confirm running my code issue has been solved |
National Express (NEX.L) has been renamed Mobico Group Plc (MCG.L) effective from early June 2023. |
@brian-dm100 |
Maybe. I use an 'exception' and enclosed 'while true: try:' to present the input box again until a correct ticker is entered. |
I've moved data fetching into a singleton class, so one session one cookie shared by threads. Can people stress-test |
Add cookie & crumb to requests. Involves several changes: - fetch cookie & crumb, obviously. - two different cookie strategies - one seems to work better in USA, other better outside. - yfinance auto-detects if one strategy fails, and switches to other strategy. - cookie is stored in persistent cache folder, alongside timezones. Refetched after 24 hours. To have this work well with multithreading (yfinance.download()) requires more changes: - all threads share the same cookie, therefore the same session object. Requires thread-safety ... - converted data class to a singleton with "SingletonMeta": - the first init() call initialises data. - but successive calls update its session object - naughty but necessary. - thread locks to avoid deadlocks and race conditions.
30fe3b2
to
4466e57
Compare
No issues for me on yf.download with 46 tickers |
Not a bug, caused by Python 3.8 |
Worked perfectly well for me in Brazil. For both history() and get_info() for 400+ tickers. |
Ima wait until Friday evening passes, that seems to be when Yahoo makes breaking changes. Then official release. |
Using
|
Does this fix
info
404 error? VOTE HEREI stumbled upon simple code to get cookie & crumb from Yahoo: https://stackoverflow.com/questions/76065035/yahoo-finance-v7-api-now-requiring-cookies-pythonIt might fix issues some users are having. Try it and report back (instructions #1080)Update:
Fixed this "basic" strategy, then added second strategy for getting cookie & crumb #1657 (comment)
Hopefully fixes:
Status:
Big change so letting people stress-test this. Concerned by Asia reporting "doesn't fix".
Credits:
@bot-unit
@psychoz971
@Jshen123
anyone else?