Update tor-bridges-ip-port.csv and create python script to parse #18
Conversation
scripts/parse_fallback_dirs.py
Outdated
| csv_writer = csv.writer(csv_file, delimiter=',') | ||
|
|
||
| for l in lines: | ||
| csv_writer.writerow(l) |
There was a problem hiding this comment.
Thanks for also writing a script to parse them!
I guess this does not actually write directly the csv file, because the csv file contains some extra IPs and addresses from other parts of the tor source code.
How did you go abouts merging what is the output of this script with the content of the existing CSV?
It would be great if we had a way to entirely generate the csv file from a script that we can schedule to run on a periodic basis.
There was a problem hiding this comment.
As far as I understand, there are 4 potential sources for hard-coded relays:
- Tor fallback directory mirrors: https://gitweb.torproject.org/tor.git/tree/src/app/config/fallback_dirs.inc
- Tor authorities: https://gitweb.torproject.org/tor.git/tree/src/app/config/auth_dirs.inc
- Tor Browser (Desktop) Bridges: these are kept in a JS preferences file, I don't know where it is
- Tor Browser for Android Bridges: I'm not sure if they use a different set of bridges
There was a problem hiding this comment.
* Tor Browser (Desktop) Bridges: these are kept in a JS preferences file, I don't know where it is
FWIW, they are available here: https://gitweb.torproject.org/builders/tor-browser-build.git/tree/projects/tor-browser/Bundle-Data/PTConfigs/bridge_prefs.js
* Tor Browser for Android Bridges: I'm not sure if they use a different set of bridges
The list is identical to desktop, and available here: https://gitweb.torproject.org/tor-android-service.git/tree/service/src/main/assets/common/bridges.txt
More broadly, we maintain all information related to default bridges here: https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/DefaultBridges
| 81.7.11.186,443,BeastieJoy60,or_port | ||
| 197.231.221.211,443,IPredator,or_port |
There was a problem hiding this comment.
I am not 100% sure we want to be deleting IPs of bridges that we used to test (even if they are now offline).
This bridge in particular still seems to work, so it's probably wise to keep it in there.
There was a problem hiding this comment.
This is an (old) fallback directory mirror. We update the fallback list every 6-12 months as relays fail (or change address or keys). We also rotate fallbacks to evade censorship.
We'd like ooni to test the latest fallbacks, authorities, and Tor Browser bridges. Testing old fallbacks, authorities, and bridges isn't that helpful, because most Tor clients and Tor Browser installs are up to date.
And you'll get spurious failures, because we remove some fallbacks and bridges because they have failed.
|
I have updated the script to merge the new arrays and only delete duplicates. Not sure that's what you want or are comfortable running, but it is a start. |
|
We have since moved to storing the testing targets inside of https://github.com/ooni/sysadmin/blob/master/ansible/roles/probe-services/templates/tor_targets.json, we may want to add some tooling in that repository to update the files inside of that repo and format them as JSON. @NullHypothesis do you have some thoughts on how you would like us to ensure that the testing targets are always fresh? |
Fix issue #16 .