Interest in expanding the dataset? (eg community effort)

[nmstoker]

Hello,

This looks like a really impressive effort, thank you for sharing the paper, code, dataset and other details.

Is there interest in continuing to grow the dataset, to cover more sites and/or to have more on existing sites?

If so, it strikes me that it might be possible to build a little volunteer community around such an effort. I know there was pay for (turk?) helpers but I would gladly give it a go on free time.

Likewise perhaps certain major websites might be convinced to donate efforts too, since it would presumably help with accessibility. For instance government sites oftwn want their sites to be easily accessed by the public and in sone cases (eg UK NHS) they have quite consistent design, so effort in one area may open up a large amount of web content.

Anyway, just thought I'd sound you out, as you may have ideas this way yourself (or equally may think it's not something that can be easily supported).

Thanks again for the research! 🙂

[xhluca]

@nmstoker Thanks for the suggestion!

I think this is a great idea. We would love to see this being extended to more websites and settings!

I plan to publish the extension and upload/data processing code at a later stage (i need to clean them up as they are not quite ready yet for public consumption, and we are currently focusing on this chrome app for integrating agents into the browser).

With the extension and app, it's much easier to coordinate an open and decentralized effort in collecting and sharing the data. I would suggest the following steps in your efforts:

1. Collect the data using the extension/upload interface. It's better to work in a pair (one person is the instructor, the other the navigator, use a pseudonym to identify who is the navigator for a given demo).
   • You can focus on specific problems; for example, categories/subcategories not yet covered, website in a different language, etc.
   • Check out the annotators instructions at the end of the Appendix of our paper - it gives a lot of details on the best practices of instructor/navigator.
2. [Optional] Reach out to other community members interested in building a similar dataset, so you can coordinate the collection effort and merge the resulting data.
3. Once you have a good # of demos, you can create two new huggingface datasets: a simpler one like WebLINX, and the complete dataset like WebLINX-full
   • You can call it something similar to the dataset depending on your task, e.g. "WebLINX-UK" for British websites, "GitLINX" for GitHub specific demos, "LongLINX" for longer sessions (e.g. 60+ turns min.), etc. You can use the -full suffix as a convention for the full data, and no suffix for the main version (faster loading). The data card can be in the main version.
   • If you train models, you can add the name of the dataset as postfix, e.g. Sheared-LLaMA-2.7B-longlinx
   • You can publish under your own Huggingface account, or create an organization if you are more than 1 working on it (if you don't have an idea for the org, just name it the same as the dataset you want to create).

I think that once we have a few datasets, we could curate them under a shared community. I reserved huggingface.co/OpenLINX for this, but will wait for a few community-created datasets before formalizing it.

Regarding donations, I think it'd be a great idea! If anyone finds an interesting funding opportunity, feel free to share them in this thread so community members (including ourselves) can apply and receive funds to work on the collection effort (or to coordinate it via professional services such as EsyCommerce).

[xhluca]

PS: I converted the issue into discussion to take advantage of the community functionality on Github

[cry0genic]

Hi @xhluca, great work! I absolutely love it, have been following this space for a while!

A small request (if possible) from my end is: Would it be possible to share the extension's code + upload/data processing code via email? ([email protected])

I read the entire paper end to end, but there's a whole lot of difference between theory and application. Even though I have mostly understood how this is working, it would be a richer experience with the code.

If you cannot, no issue; I'll wait for the official release.

Thanks!

[nmstoker]

Thanks a lot for those tips @xhluca and I'm really pleased to hear you're positive about the community idea.

I'll go over the paper again, in particular the appendix as you mentioned and will keep an eye out for the extension update in the repo.

[xhluca]

Enjoy and let me know if you have questions!

[sivareddyg]

@nmstoker thank you for sharing your thoughts. 100% agree with you. This was in our original plan -- having a repository of skills and websites that models can be extended to, and facilitate a way for interested parties to pay for expansion. We want to work with our data partner to ensure consistency (perhaps at a later stage, this can be done by volunteers).

The main priority right now is to have a browser plugin and run these models to see how they perform on live websites and figure out their weaknesses. A couple of volunteers agreed to take it up but no one has delivered anything yet. We will test these models live after the plugin is ready. After that, we will facilitate a way to contribute to the skill and website expansion.

[majidsh97]

Is there any update about the Chrome extension that records actions?
Thanks for your impressive work.