Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flate2::Decompress::decompress consumes more input than actually used #434

Closed
link2xt opened this issue Oct 6, 2024 · 2 comments
Closed

Comments

@link2xt
Copy link

link2xt commented Oct 6, 2024

flate2::Decompress::decompress is documented as "consuming only as much input as needed and writing as much output as possible". However in my tests it consumes the whole input even if output buffer size is not sufficient. Because of this I cannot use flate2 to decode the IMAP stream incrementally using async-compression crate: async-email/async-imap#112

Here is a failing test where output buffer is only 8 bytes long, but the whole input is consumed:

#[test]
fn deflate_decoder_partial() {
    // Decompresses to
    // "* QUOTAROOT INBOX \"User quota\"\r\n* QUOTA \"User quota\" (STORAGE 76 307200)\r\nA0001 OK Getquotaroot completed (0.001 + 0.000 secs).\r\n"
    let input = vec![
        210, 82, 8, 12, 245, 15, 113, 12, 242, 247, 15, 81, 240, 244, 115, 242, 143, 80, 80, 10,
        45, 78, 45, 82, 40, 44, 205, 47, 73, 84, 226, 229, 210, 130, 200, 163, 136, 42, 104, 4,
        135, 248, 7, 57, 186, 187, 42, 152, 155, 41, 24, 27, 152, 27, 25, 24, 104, 242, 114, 57,
        26, 24, 24, 24, 42, 248, 123, 43, 184, 167, 150, 128, 213, 21, 229, 231, 151, 40, 36, 231,
        231, 22, 228, 164, 150, 164, 166, 40, 104, 24, 232, 129, 20, 104, 43, 128, 104, 3, 133,
        226, 212, 228, 98, 77, 61, 94, 46, 0, 0, 0, 0, 255, 255,
    ];

    // Create very small output buffer.
    let mut output = vec![0; 8];

    let zlib_header = false;
    let mut decompress = flate2::Decompress::new(zlib_header);

    let flush_decompress = flate2::FlushDecompress::None;
    let status = decompress.decompress(&input, &mut output, flush_decompress).unwrap();
    assert_eq!(status, flate2::Status::Ok);

    // Should not consume everything, there is not enough space in the buffer for the output.
    assert_ne!(decompress.total_in(), input.len() as u64);
}

Here miniz_oxide::inflate::stream::inflate returns bytes_consumed equal to the whole input size even though there was not enough space in the output buffer:

self.total_in += res.bytes_consumed as u64;

I thought about reporting the bug directly to miniz_oxide, but don't see in its documentation any guarantee that it will consume only as much as needed.

@link2xt
Copy link
Author

link2xt commented Oct 6, 2024

The test passes with --no-default-features --features zlib, so probably should be upstreamed to miniz_oxide.

@link2xt
Copy link
Author

link2xt commented Oct 6, 2024

I upstreamed the issue to miniz_oxide: Frommi/miniz_oxide#158

@rust-lang rust-lang locked and limited conversation to collaborators Oct 7, 2024
@Byron Byron converted this issue into discussion #435 Oct 7, 2024

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant