Use uint64 to hold queue size in message stats ?

[sebastian-nagel]

The number of queued URLs in the message stats is hold in a uint32 type (see urlfrontier.proto) which naturally limits the max. number of URLs to 2^32 resp. 2^31 if there are no unsigned integer types in the target programming language (eg. Java).

What about using uint64 instead?

• because protobuf uses a [varint encoding](https://developers.google.com/protocol-buffers/docs/encoding#varints] this causes no overhead when sending message stats
• in doubt, I would assume that there are distributed crawlers which can handle more than 2 billions queued URL. Of course, there is no need for a queue implementation to handle such large queues.
• open question: use uint64 for all values, including per-queue size, number of queues? Again, in doubt, why not?
  • a sitemap index may list 2.5 billion URLs (50k*50k)
  • there are for sure more than 2 billion host names in the web
• btw., ListQueues takes uint64 as parameter

[jnioche]

thanks @sebastian-nagel
for Stats: yes they could all be uint64 - apart from maybe from inprocess but that wouldn't matter if it did

[jnioche]

ListQueues probably doesn;t need to be a uint64 but it doesn' tmatter really

[sebastian-nagel]

Indeed, you hardly want to see so many queues. But yes, no matter.

[jnioche]

@sebastian-nagel implemented in a21656a