I am taking a look at this. Currently also pandas-like fails to add a column for none, yet without raising an error.

Yet hardest part is mapping column names and ordering of such 😂

😄 thanks for taking a look

🤔 i do think the Polars default output looks a bit odd here, and where I spotted this was in Formulaic, and they wouldn't include the "null" column in the output anyway. wondering if we should add a required drop_nulls keyword argument

i do think the Polars default output looks a bit odd here,

Agree 👌

I opened the PR, but pandas behavior is so inconsistent and hard to remap. Tests are failing because for int values, adding the dummy_na=True flag (which adds a null column at the end, even if there aren't any) seems to trigger some casting to float. Therefore the column names change from a_1, a_2,... to a_1.0, a_2.0,....

Do you think that this should be reported upstream?

Upstream to Polars? Yes I think so that might be good

I meant the pandas behavior that converts int to float for dummy_na=True. Example:

import pandas as pd

data = [1, 2, 1]
pd.get_dummies(pd.Series([1,2,1]), dummy_na=True)

     1.0    2.0    NaN
0   True  False  False
1  False   True  False
2   True  False  False

while:

pd.get_dummies(pd.Series([1,2,1]))

       1      2
0   True  False
1  False   True
2   True  False

What I mean is that by adding the extra argument, the name of the first two columns will change as well even though no nulls/nans are present.

This does not happen with nullable types. I am not sure how we should go about it honestly.

ah i see - well pandas allows for column names to be floats, so I don't think pandas would consider this a bug nor something that needs changing

the string "null" in Polars on the other hand...maybe that should at least be configurable, with a dummy_null argument? If not in Polars, we could at least add it here, given how problematic it seems to be to have dummy_null=True behave consistently across libraries

Edited the snippet above to better explain what I mean for pandas

thanks, but i still think the response would be "this is just what pandas does" (😭 )

Alright, then let's discuss how we should behave 😂
As you can see in #1040 doctests are failing because we have ints in our example. We can either convert_dtypes or use strings in the example. However I would rather add some disclaimer note in the method docstring. WDYT?

i discussed this a bit with polars devs, they're open to having something like dummy_null in Polars

I think the Polars API should be revised further, because currently, if you get

shape: (3, 3)
┌──────┬──────┬───────┐
│ _bar ┆ _foo ┆ _null │
│ ---  ┆ ---  ┆ ---   │
│ u8   ┆ u8   ┆ u8    │
╞══════╪══════╪═══════╡
│ 0    ┆ 1    ┆ 0     │
│ 1    ┆ 0    ┆ 0     │
│ 0    ┆ 0    ┆ 1     │
└──────┴──────┴───────┘

then you have no way to know if it came from

pl.Series(['foo', 'bar', 'null']).to_dummies()

or

pl.Series(['foo', 'bar', None]).to_dummies()

I'm more inclined still to just follow the pandas default here (!), and loudly document that we slightly differ from Polars here (my prediction is that the Polars API will have to change for this function anyway)

Thanks @MarcoGorelli , shall we wait for them to decide on the polars API? IIRC this was needed for formulaic, so you can set the pace for how soon we need this one out.

I added some commits because I figured out of to workaround pandas issue 😂