stat a non-exist object can be expensive due to backtrace

[xxchan]

I noticed stat failed NotFound can take as long as 1s. After investigation, I found the culprit is Error::new.

Capturing a backtrace can be a quite expensive runtime operation [1][2]. Actually in this case (checking whether a file exists), the overhead is completely unnecessary. However, I'm also reluctant to disable backtrace completely because it may be useful in other cases. Any ideas how we can improve that?

log:

2025-01-24T17:39:06.604104+08:00  INFO risingwave_meta::hummock::manager: try reading cluster_id
2025-01-24T17:39:06.60682+08:00 DEBUG opendal::services: service=s3 name=hummock001 path=hummock_001/cluster_id/0: stat started    
2025-01-24T17:39:06.615312+08:00  INFO opendal::types::error: Error::new begin    
2025-01-24T17:39:07.703926+08:00  INFO opendal::types::error: Error::new end    
2025-01-24T17:39:07.704459+08:00  WARN opendal::services: service=s3 name=hummock001 path=hummock_001/cluster_id/0: stat failed NotFound (permanent) at stat, context: { uri: http://127.0.0.1:9301/hummock001/hummock_001/cluster_id/0, response: Parts { status: 404, version: HTTP/1.1, headers: {"accept-ranges": "bytes", "content-length": "0", "server": "MinIO", "strict-transport-security": "max-age=31536000; includeSubDomains", "vary": "Origin", "vary": "Accept-Encoding", "x-amz-id-2": "5e0466557790194f71acb9d44341f26b381bbf7ca42536c367b0a3b121df5c06", "x-amz-request-id": "181D9771E1E90D48", "x-content-type-options": "nosniff", "x-minio-error-code": "NoSuchKey", "x-minio-error-desc": "\"The specified key does not exist.\"", "x-xss-protection": "1; mode=block", "date": "Fri, 24 Jan 2025 09:39:06 GMT"} }, service: s3, path: hummock_001/cluster_id/0 }    
2025-01-24T17:39:07.704479+08:00 DEBUG opendal::services: service=s3 name=hummock001 path=hummock_001/cluster_id/0: stat failed    
2025-01-24T17:39:07.797189+08:00  INFO risingwave_meta::hummock::manager: cluster_id not found, writing cluster_id

The added log:

impl Error {
    /// Create a new Error with error kind and message.
    pub fn new(kind: ErrorKind, message: impl Into<String>) -> Self {
        log::info!("Error::new begin");
        let e = Self {
            kind,
            message: message.into(),

            status: ErrorStatus::Permanent,
            operation: "",
            context: Vec::default(),
            source: None,
            // `Backtrace::capture()` will check if backtrace has been enabled
            // internally. It's zero cost if backtrace is disabled.
            backtrace: Backtrace::capture(),
        };
        log::info!("Error::new end");
        e
    }

[Xuanwo]

I feel that we only need to capture the backtrace when users encounter an Unexpected error. But I'm not very sure about this too. Perhaps we could at least disable backtrace capture for NotFound.