[beta] Presenting GG2TW: A Tool to Convert Google Groups to TW JSON

[flancast90]

Hello All,

Having been part of a discussion on the TW Google Groups on whether or not to migrate from Google Groups (GG) to another service, I became increasingly interested in adapting TW to support a more forum-like structure. As part of this, a major concern among others was importing the data from GG to TW.
I addressed this problem by building GG2TW, a tool that scrapes the TW GG and JSON.parse()'s the data found into fields, finally combining them into an array. Since quickly creating it a few days back, I have steadily worked it into a "semi-reliable" way to handle all of this data, but it is still prone to bugs. I am now happy to release it as the "beta" version of the software, and share a demo at my website subdomain: http://gg2tw.finnsoftware.net/GG2TW/GG2Tid.html.
I would love for some of you more experienced devs here to take a look at it, and to give me feedback on it (this is my first contribution to anything TW-related on GitHub). Already, I am beginning to polish it to get rid of unnecessary wording from the screen-scraping, and re-ordering code to speed-it-up (can take a few mins because of the huge amount of data read and parsed).
For anyone that would like a look through of the code, its repo can be found at https://github.com/flancast90/GG2TW, where the main code is housed at assets/script.js. (Beware: the syntax of the code sucks, is a great example of https://github.com/Droogans/unmaintainable-code :))
I would appreciate, as stated, any comments or concerns, which can be given at the GitHub link mentioned. Feel free to star, fork, download, begin an Issue, etc at that same repo.

Lastly, a known bug that can prevent the code from working is it being run in multiple places. GG issues a CORS allow-origin header which can only be bypassed by the proxy API I'm using for ONE location. If you have already used it on the website, but would like to download and run, a quick fix for this is to clear your browser cache, history, cookies, etc.

[pmario]

Hi, If I try it, it doesn't work for me.
The JSON looks like this: [[[[xxxx]]]] which doesn't import to TW at all. So I needed to remove 3 of those brackets.

All the text fields contain %20 as spaces. %20 stays, if imported to TW.

I did test it with FF Ubuntu 20.04

[flancast90]

That’s strange that the JSON isn’t formatted correctly for you. I’ve tested in most browsers and run the JSON through a validator multiple times. One issue may be unintended characters in the GG conversations, which could malform the JSON. The %20 issue was known to me: it is a result of escaping these same characters in the JSON using Jquery escape().
I am researching an alternative to this, but so far encodeURI has proven ineffective for this. If you have any ideas for escaping []{}“”’, etc. let me know. My idea so far is to just create a replace function that adds \ before each of these chars to escape them.

As said before, though, the program is still filled with bugs, and my many changes on GitHub to the repo can lead to further issues. I appreciate this report: hopefully I encounter the same error so I can check-out what is happening using the console errors.

[flancast90]

Update: Fixed the %20 issues. Also found that it runs much faster on FF, if that's a priority.