Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NOINDEX #29

Open
rimutaka opened this issue Feb 2, 2022 · 2 comments
Open

NOINDEX #29

rimutaka opened this issue Feb 2, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@rimutaka
Copy link
Member

rimutaka commented Feb 2, 2022

The current indexing strategy makes no sense. We should let the search engines index anything they like and then review.
@rimutaka rimutaka added the enhancement New feature or request label Feb 2, 2022
rimutaka added a commit that referenced this issue Feb 2, 2022
@rimutaka
Remove NOINDEX from useful content #29
d49e2a8 
* removed LANG from links to packages on profile pages
* allow indexing of dev profiles
* allow indexing any combination of keywords
* allow indexing of related packages
rimutaka added a commit that referenced this issue Feb 3, 2022
@rimutaka
Restore NOINDEX for dev profile pages #29
8ffe203
@rimutaka
Copy link
Member Author

rimutaka commented Feb 3, 2022 •
edited
Loading

The rationale of using NOINDEX on personal profiles is that it's one thing we analyze someone's profile and the other is to make it readily available via a search engine. This feature would need to be controlled by profile owners and should be no-index unless specified otherwise.

Update

The change I made in 8ffe203 is for anonymous profiles. Those are definitely not indexed. GH profiles were indexed and are still indexed. I do not know if it's a good thing or not. The profiles are a gateway to projects. Given how few people complained I'm tempted to leave it as INDEX for now.

@rimutaka
Copy link
Member Author

rimutaka commented Feb 8, 2022

Update

I added a rule to allow blocked bots to fetch robots.txt. Bing finally picked it up and stopped hammering the site.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rimutaka