Is there a way to expose a PathBuf struct member as a Python's pathlib.Path member?

[AngheloAlf]

I'm porting a Python library to PyO3 and everything is working fine, up until testing this Python class to Rust:

Python:

@dataclasses.dataclass
class File:
    filepath: pathlib.Path
    # Omitted code

Rust:

#[derive(Debug, Clone)]
#[pyclass(module = "mapfile_parser", unsendable, sequence)]
pub struct File {
    #[pyo3(get, set)]
    pub filepath: PathBuf,
    // Omitted code
}

For some reason the Rusts version converts the filepath member to a str instead of a pathlib.Path when accessing it from Python, which feels a bit inconsistent since the new method takes a PathBuf and making a Python instance errors if I don't pass a pathlib.Path.
Am I doing something wrong?

I'm using PyO3 0.19.0

[davidhewitt]

You could write a custom #[getter] in #[pymethods] for this. I am also open to considering whether we got this conversion wrong and it should create pathlib.Path, but that would be a breaking API change.

[dbarnett]

Could you suggest the ideal implementation for a manual getter like that? I'm looking for the same.

[LilyFoote]

This implementation worked for me:

use std::path::PathBuf;

use pyo3::exceptions::PyValueError;
use pyo3::intern;
use pyo3::prelude::*;
use pyo3::types::PyString;

#[pymodule]
mod pyo3_pathlib {
    use super::*;

    #[derive(Debug, Clone)]
    #[pyclass(unsendable, sequence)]
    pub struct File {
        #[pyo3(set)]
        pub filepath: PathBuf,
    }

    #[pymethods]
    impl File {
        #[new]
        fn new(filepath: PathBuf) -> Self {
            Self { filepath }
        }

        #[getter]
        fn get_filepath(&self, py: Python) -> PyResult<Py<PyAny>> {
            let pathlib = PyModule::import_bound(py, intern!(py, "pathlib"))?;
            let path = pathlib.getattr(intern!(py, "Path"))?;
            let os_filepath = self.filepath.as_os_str();
            let filepath = match os_filepath.to_str() {
                Some(filepath) => filepath,
                None => {
                    let bytes = os_filepath.as_encoded_bytes();
                    let message = format!("filepath is not valid utf-8. Raw bytes: {:?}", bytes);
                    let err = PyValueError::new_err(message);
                    return Err(err);
                }
            };
            let filepath = PyString::new_bound(py, filepath);
            Ok(path.call1((filepath,))?.unbind())
        }
    }
}

mod tests {
    #[test]
    fn test_invalid_utf8() {
        use super::pyo3_pathlib::*;
        use super::*;
        use std::ffi::OsStr;
        use std::os::unix::ffi::OsStrExt;
        use std::path::Path;

        use pyo3::types::PyDict;
        let bytes = vec![195, 40];
        let path = OsStr::from_bytes(&bytes);
        let filepath = Path::new(path).to_path_buf();
        let file = File { filepath };

        pyo3::prepare_freethreaded_python();
        Python::with_gil(|py| {
            let locals = PyDict::new_bound(py);
            locals.set_item("file", file.into_py(py)).unwrap();
            let err = py
                .eval_bound("file.filepath", None, Some(&locals))
                .unwrap_err();
            assert!(err.is_instance_of::<PyValueError>(py));
            assert_eq!(
                err.value_bound(py).to_string(),
                "filepath is not valid utf-8. Raw bytes: [195, 40]"
            );
        });
    }
}

I've tested it manually with some simple cases I expected to work and added this one test for the error path. There may be room for optimisation though.

It may be possible to do better with the error path, but that would require a deeper understanding of how Rust's OsStr encoding works than I have. It's probably not directly compatible with Python's surrogateescape error handler, which appears to be how to handle this on the Python side.

[LilyFoote]

I got a bit further with this, using surrogateescape to fully round-trip Path objects created from arbitrary bytes on the Python side through Rust and back to Python.

The new implementation:

use pyo3::types::{PyBytes, PyDict, PyString};

// snip identical code

        #[getter]
        fn get_filepath<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
            let os_filepath = self.filepath.as_os_str();
            let filepath = match os_filepath.to_str() {
                Some(filepath) => PyString::new_bound(py, filepath),
                None => {
                    let bytes = os_filepath.as_encoded_bytes();
                    let bytes = PyBytes::new_bound(py, bytes);
                    let kwargs = PyDict::new_bound(py);
                    kwargs.set_item("errors", "surrogateescape")?;
                    let filepath = bytes.call_method(intern!(py, "decode"), (), Some(&kwargs))?;
                    filepath.downcast_into()?
                }
            };
            let pathlib = PyModule::import_bound(py, intern!(py, "pathlib"))?;
            let path = pathlib.getattr(intern!(py, "Path"))?;
            path.call1((filepath,))
        }

I tested this thoroughly with hypothesis:

from pathlib import Path

from hypothesis import given
from hypothesis.strategies import binary, text

from pyo3_pathlib import File


@given(text())
def test_pathlib_roundtrip(raw_path):
    path = Path(raw_path)
    file = File(path)
    assert file.filepath == path


@given(binary())
def test_pathlib_roundtrip_binary(raw_path):
    escaped_path = raw_path.decode(errors="surrogateescape")
    path = Path(escaped_path)
    file = File(path)
    assert file.filepath == path

I then wanted to test starting with a PathBuf created in Rust and round-tripping to Python and back to Rust, but ran into problems converting from Bound<'_, PyAny> back to File. I get the same error as in #4337 with this test:

#[cfg(test)]
mod tests {
    use super::pyo3_pathlib::*;
    use super::*;
    use std::ffi::OsStr;
    use std::os::unix::ffi::OsStrExt;
    use std::path::Path;

    #[test]
    fn test_extract_file() {
        pyo3::prepare_freethreaded_python();
        Python::with_gil(|py| {
            let locals = PyDict::new_bound(py);
            let code = "
from pyo3_pathlib import File
from pathlib import Path
file = File(Path('/home/lily/'))
            ";
            py.run_bound(code, None, Some(&locals)).unwrap();
            let file = locals.get_item("file").unwrap().unwrap();
            let file: File = file.extract().unwrap();
        });
    }
}

Maybe I'm doing it wrong, but I couldn't work out the right way from the current documentation, and the existence of #4337 makes me think it might currently not be possible.