Shouldn't missing python fields been defaulted if field is an Option<> in Rust?

[andronat]

I'm surprised that the following example fails with: Error: KeyError: 'b'. Shouldn't field b automatically become None given that it doesn't exist in the Python dict?

Also, is there a way for me to make this field "truly" optional? I'm trying to support some backwards and forwards compatibility on a python API and it's not guaranteed that all past fields will be present. I wanted to use FromPyObject and from_item_all so I would avoid writing for every field a parsing function with #[pyo3(from_py_with = "my_parser")].

Is it possible to universally make optional fields?

    use pyo3::FromPyObject;
    #[derive(FromPyObject, Clone)]
    #[pyo3(from_item_all)]
    struct Test {
        pub a: String,
        pub b: Option<String>,
    }

    #[test]
    fn test_option() -> Result<()> {
        Python::with_gil(|py| {
            let result: Test =
                py.eval("{'a': 'test'}", None, None).wrap_err("eval failed")?.extract()?;
            assert!(result.a == "test");
            assert!(result.b.is_none());
            Ok(())
        })
    }

[davidhewitt]

I think we would need a #[pyo3(default)] attribute similar to serde, I'm certainly open to that.

[andronat]

I would love to contribute! Any spare time to give me some code pointers?

[andronat]

Also, something to highlight here, Serde by default when a field is Option<> and is not available in the struct, it's automatically defaulted to None. Here is an example:

    use serde::Deserialize;

    #[derive(Deserialize, Debug)]
    struct MyData {
        a: String,
        b: Option<String>,
    }

    #[test]
    fn test_serde() -> Result<()> {
        let json = r#"
        {
            "a": "Value of A"
        }"#;

        let result: MyData = serde_json::from_str(json)?;
        println!("Parsed JSON with optional b: {result:?}");
        Ok(())
    }

Output: Parsed JSON with optional b: MyData { a: "Value of A", b: None }

[birkenfeld]

That is probably ok for serde, but feels too implicit for PyO3.

[davidhewitt]

That is probably ok for serde, but feels too implicit for PyO3.

Agreed.

Any spare time to give me some code pointers?

Sorry to have taken a few days to get to this. I think you should start by exploring the existing #[derive(FromPyObject)] code in pyo3-macros-backend/src/frompyobject.rs and also src/impl_/frompyobject.rs

To get a sense of what the existing derive does, it might also be useful to install cargo expand and run cargo expand --test test_frompyobject

[kylebarron]

I also hit this today. I expected that derive(FromPyObject) would allow missing keys as None for an Option.

Did any progress ever get made on this?

[davidhewitt]

I don't think so, opened #4643