Structs or enums for API responses

[Erutuon]

Using serde_json::Value to represent API responses is pretty laborious. It requires lots of .as_object() or .as_str() and then checking that the result is Some(_), or Option::map, or matching on variants of serde_json::Value, etc.

I propose creating custom structs or enums to represent responses. This makes accessing fields in the JSON as simple as accessing fields in the struct or enum. For instance, this example shows a struct that could be used in the Page::text method (and returns a serde_json::Value if the JSON fails to deserialize as RevisionsResponse, though that may not be necessary):

use serde::Deserialize;
use serde_json::Value as JsonValue;
use std::collections::HashMap;
use url::Url;

#[derive(Debug, Deserialize)]
#[serde(untagged)]
enum FallibleDeserialization<T> {
    Success(T),
    Failure(JsonValue)
}

#[derive(Debug, Deserialize)]
#[allow(unused)]
struct RevisionsResponse {
    batchcomplete: bool,
    query: PagesQuery,
}

#[derive(Debug, Deserialize)]
struct PagesQuery {
    pages: Vec<Page>,
}

#[derive(Debug, Deserialize)]
struct Page {
    #[serde(rename = "pageid")]
    id: u32,
    #[serde(rename = "ns")]
    ns: i32,
    title: String,
    revisions: Vec<Revision>,
}

#[derive(Debug, Deserialize)]
struct Revision {
    slots: HashMap<String, RevisionSlot>,
}

#[derive(Debug, Deserialize)]
struct RevisionSlot {
    #[serde(rename = "contentmodel")]
    content_model: String,
    #[serde(rename = "contentformat")]
    content_format: String,
    content: String,
}

#[tokio::main]
async fn main() {
    let mut url: Url = Url::parse("https://en.wiktionary.org/w/api.php").unwrap();
    url.set_query(Some(&serde_urlencoded::to_string(&[
        ("action", "query"),
        ("prop", "revisions"),
        ("titles", "Template:link"),
        ("rvslots", "*"),
        ("rvprop", "content"),
        ("formatversion", "2"),
        ("format", "json"),
    ]).unwrap()));
    let response: FallibleDeserialization<RevisionsResponse> = reqwest::get(url).await.unwrap().json().await.unwrap();
    if let FallibleDeserialization::Success(response) = response {
        for Page { revisions, .. } in response.query.pages {
            for Revision { slots } in revisions {
                let slot = slots.get("main").or_else(|| slots.iter().next().map(|(_, slot)| slot));
                dbg!(slot);
            }
        }
    }
}

Dependencies in Cargo.toml:

reqwest = { version = "0.10", features = ["json"]}
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
serde_urlencoded = "0.6"
tokio = { version = "0.2", features = ["rt-core", "macros"] }
url = "2.1"

To make this possible, the Api methods that currently decode the response as serde_json::Value (ultimately via Api::query_api_json) would need to be generic, so that they could instead deserialize into a more specific struct or enum (something like, in the example above, FallibleDeserialization<RevisionsResponse>).

And Api::get_query_api_json_limit would probably need some way to perform the function of Api::json_merge generically, for the various structs that it would return in place of serde_json::Value. For instance it could be generic over a trait that has a merge method (maybe named MergeableResponse).

Difficulties: Using the Deserialize derive macro will add to compile time. Also it may require trial-and-error to figure out what the schema for the API responses actually is.

[enterprisey]

Maybe a tool for scraping the Action API documentation and converting it into Rust structs and enums? I've always wanted to have type-checked parameters for various queries.

[Erutuon]

@enterprisey, that would be great, but I don't know how to do it: the documentation of the schema of the JSON responses doesn't seem machine-readable enough. So far I've only tried making up the structs and enums manually.

[legoktm]

I played with this quite a bit over the long weekend and came up with a proc_macro that generates a response struct based on query parameters:

#[query(
    list = "logevents",
    leprop = "user|type",
)]
struct Response;

(You could also specify other parameters like lelimit but that doesn't affect the response)

Here's the proof-of-concept code: https://gitlab.com/legoktm/mwapi_responses/ (rather unpolished, but mostly functional)

I think this provides a reasonable balance on allowing flexible permutations of prop parameters while still providing strict typing.

As far as autogeneration, I think we can infer most of the metadata from results, but it might need some manual adjustments (e.g. u32 vs i32).

If people are interested in this I can keep working on it.

[Erutuon]

It seems a great advantage to be able to generate structs automatically. There are a lot of API responses and it takes a lot of effort to create a set of structs with useful types and more Rustic field names.

See for instance the SiteInfo struct in my branch. I renamed fields so that they had underscores between words using #[serde(rename)] (mainpage -> main_page) and created some custom types (CaseSensitivity, BTreeMap<NamespaceId, NamespaceInfo> for the namespaces, MapVec to convert Vec-like maps into Vecs).

I'd consider this ideal, but not feasible for all API responses. Maybe the output of the proc macro could be used as a starting point for a more tailored set of structs.

[legoktm]

FWIW I have kept working on mwapi_responses and have published it as a full crate. Most of it is autogenerated, but some I ended up writing by hand just to keep things simpler (e.g. https://gitlab.com/mwbot-rs/mwbot/-/blob/master/mwapi_responses/src/protection.rs). https://gitlab.com/mwbot-rs/mwbot/-/blob/master/mwbot/src/generators.rs demonstrates some real-world uses of the macro. I think siteinfo is a good candidate for automatic/dynamic generation since the various siprop parameters affect which fields get output.

I tried to outline my vision in https://blog.legoktm.com/2021/11/01/generating-rust-types-for-mediawiki-api-responses.html - basically stay as faithful to the API response as possible, but provide convenience functions to make it more Rust-friendly: https://gitlab.com/mwbot-rs/mwbot/-/issues/2.

[magnusmanske]

Very cool, thanks so much for doing this!