llm-factor: migrate to candle

[karthik2804]

This PR replaces the dependency on rustformers/llm to huggingface/candle. This allows us to run newer models like Llama 3(.1). This now requires the models to be of the safetensors format.

This PR also removes the concept of well-known models. This ensures a consistent directory structure for all models. The rationale is that, with this change, the only group of models initially supported is the Llama family.

Closes #2735

[radu-matei]

Given this is a breaking change, I'd suggest adding the 3.0 label.

[karthik2804]

@radu-matei I do not believe I can add labels in this repository.

[karthik2804]

The test failure does not seem to be related?

[rylev]

You can replace this with:

#[derive(Deserialize)]
struct SafeTensorsJson {
  weight_map: HashMap<String, String>
}

let json: SafeTensorsJson = serde_json::from_reader(&json_file).map_err(candle::Error::wrap)?

[karthik2804]

I reverted this change and the other because it was leading to some off error, where the returned vector was a duplicate of the same thing repeated several times which meant the same files were being loaded over and over which led to consuming large amounts of memory.

[rylev]

Suggested change

	for value in weight_map.values() {
	if let Some(file) = value.as_str() {
	safetensors_files.insert(file.to_string());
	}
	}
	let safetensors_files = safetensors_files
	.iter()
	.map(|v| model_dir.join(v))
	.collect::<Vec<_>>();
	safetensors_files.extend(weight_map.values().map(|v| model_dir.join(v))

This assumes no need to call as_str because of the suggested change above.

[rylev]

Can we document this trait? What about it makes it Cached? Are implementors required to cache results or does it just happen that the current implementors do?

[rylev]

I'm fine with keeping the name, but I personally find the name CachedInferencingModel confusing when implementors aren't required to cache anything. InferencingModel seems like a more appropriate name.

[rylev]

Suggested change

	match self.tokenizer.decode(tokens, true) {
	Ok(str) => Ok(str),
	Err(err) => anyhow::bail!("cannot decode: {err}"),
	}
	self.tokenizer.decode(tokens, true).context("failed to decode token stream")

[karthik2804]

It does look like I cannot do this because tokenizer.decode returns a Result<String, Box<dyn Error + Send + Sync>> which does not seem to be suitable to use context on(?)

[rylev]

I don't fully understand what this check is supposed to be doing. Why do we care about the length of the next text vs the previous, and why do we care whether the last character is alphanumeric?

[karthik2804]

The length check is to see if we have any new tokens. The alphanumeric check is supposed to be to check if we have a valid token to decode. That is what I gather from the python function the docs link to

https://github.com/huggingface/text-generation-inference/blob/5ba53d44a18983a4de32d122f4cb46f4a17d9ef6/server/text_generation_server/models/model.py#L68

[rylev]

The python code is dealing with unfinished utf-8 byte sequences which is not possible at this point in the Rust code. Rust chars are guaranteed to be valid utf-8. The check for alphanumeric chars is checking that the character is A-Z | a-z | 0-9 which does seem to be what we want.

The Tokenizer::decode function returns Strings so I'm guessing somehow the tokenizer crate is taking care of byte sequences that aren't valid utf-8?

[karthik2804]

Here is the relevant rust version from where this is borrowed.
https://github.com/huggingface/candle/blob/6eea45a761fc1636b5e8012d02bdaa93321652ca/candle-examples/src/token_output_stream.rs#L43

[rylev]

I'm fine with keeping the name, but I personally find the name CachedInferencingModel confusing when implementors aren't required to cache anything. InferencingModel seems like a more appropriate name.

[rylev]

The python code is dealing with unfinished utf-8 byte sequences which is not possible at this point in the Rust code. Rust chars are guaranteed to be valid utf-8. The check for alphanumeric chars is checking that the character is A-Z | a-z | 0-9 which does seem to be what we want.

The Tokenizer::decode function returns Strings so I'm guessing somehow the tokenizer crate is taking care of byte sequences that aren't valid utf-8?

[rylev]

🎉