Skip to content

Commit

Permalink
parser: increase prefix length used for detection of CSV dialect
Browse files Browse the repository at this point in the history
When detecting the CSV dialect, our results will get more accurate the more
data we look at. The previous value of 8KiB would sometimes only allow us to
look at one or two rows if the CSV was very wide. Wide CSVs are pretty common
in practice, so this increases that prefix in order to give our dialect
detection a better shot at success.
  • Loading branch information
psFried committed Jan 29, 2024
1 parent 3c567a8 commit 29c5141
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion crates/parser/src/format/character_separated/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ struct CsvParser {
config: AdvancedCsvConfig,
}

const PEEK_PREFIX_LEN: usize = 8096;
const PEEK_PREFIX_LEN: usize = 1 << 16; // 64KiB

impl Parser for CsvParser {
fn parse(&self, content: Input) -> Result<Output, ParseError> {
Expand Down

0 comments on commit 29c5141

Please sign in to comment.