Skip to content

Add @subset#263

Merged
pdeffebach merged 15 commits into
JuliaData:masterfrom
pdeffebach:add_subset
Jun 29, 2021
Merged

Add @subset#263
pdeffebach merged 15 commits into
JuliaData:masterfrom
pdeffebach:add_subset

Conversation

@pdeffebach

@pdeffebach pdeffebach commented Jun 25, 2021 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This is an initial attempt to add @subset.

I was able to entirely replace @where by calling skipmissing=true as a keyword argument, which is great.

We are in a real bind with regards to keyword arguments. With the move to using :block we can't just support keyword argument handling like @subset(df, ...; skipmissing = true). So I added the flag @skipmissing and re-factored the macro-flags a little bit.

julia> df = DataFrame(a = [1, missing], b = [3, 4]);

julia> @subset df @skipmissing begin 
           :a .== 1
           :b .== 3
       end
1×2 DataFrame
 Row │ a       b     
     │ Int64?  Int64 
─────┼───────────────
   1 │      1      3

But adding tests is still a pain, because the tests have missing so we would have to add @skipmissing everywhere. Is this a time when we should break with the DataFrames API and make skipmissing=true the default? That would make people's lives a bit easier when upgrading from @where to @subset.

@nalimilan This is a design decision, so I would appreciate your input.

@pdeffebach

pdeffebach commented Jun 26, 2021

Copy link
Copy Markdown
Collaborator Author

Okay, I've made skipmissing=true the default with subset and removed the @skipmissing flag. This seems like the easiest way forward. Unlike DataFrames.jl, we will treat missing as false in @subset.

All docs are added and tests added and pass. So this could be merged.

nalimilan
nalimilan reviewed Jun 27, 2021
View reviewed changes

@nalimilan nalimilan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! That makes the API much more consistent with DataFrames.

Regarding skipmissing, I think it would be good to outline a general plan. Should DataFramesMeta automatically skip/propagate missing values everywhere? We discussed adding a keyword argument to do that in DataFrames at JuliaData/DataFrames.jl#2314. It hasn't been implemented at this point, but it would make sense to decide whether we would like to enable it by default eventually in DataFramesMeta.

Comment thread docs/src/index.md Outdated
Comment thread src/macros.jl Outdated
Comment thread src/macros.jl Outdated
Comment thread src/parsing.jl Outdated
Comment thread src/parsing.jl
Comment on lines -347 to +348
create_args_vector(arg) -> vec, wrap_byrow

Normalize a single input to a vector of expressions,
with a `wrap_byrow` flag indicating that the
expressions should operate by row.

If `arg` is a single `:block`, it is unnested.
Otherwise, return a single-element array.
Also removes line numbers.

If `arg` is of the form `@byrow ...`, then
`wrap_byrow` is returned as `true`.
create_args_vector(arg) -> vec, outer_flags

@nalimilan nalimilan Jun 27, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why remove the contents of the docstring?

@pdeffebach pdeffebach Jun 27, 2021

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add a correct docstring.

Comment thread test/dataframes.jl Outdated
@@ -763,13 +705,13 @@ end
@test nrow(d) == 1

d = @where df begin

@nalimilan nalimilan Jun 27, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this to deprecated.jl?

@pdeffebach pdeffebach Jun 27, 2021

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved.

Comment thread test/subset.jl
@@ -0,0 +1,143 @@
module TestSubset

@nalimilan nalimilan Jun 27, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add tests with GroupedDataFrame?

@pdeffebach pdeffebach Jun 27, 2021

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added, ported from @where.

Comment thread test/subset.jl Outdated
pdeffebach and others added 5 commits June 27, 2021 15:32
@pdeffebach
add docstring
f2dde1e
@pdeffebach @nalimilan
Apply suggestions from code review
7c14a20 
Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
@pdeffebach @nalimilan
Apply suggestions from code review
f83161f 
Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
@pdeffebach
Merge branch 'add_subset' of https://github.com/pdeffebach/DataFrames…
430d398 
…Meta.jl into add_subset
@pdeffebach @nalimilan
Update test/subset.jl
6f956dc 
Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
@pdeffebach

pdeffebach commented Jun 27, 2021

Copy link
Copy Markdown
Collaborator Author

w.r.t. missings.

I think that adding transform! with SubDataFrame goes along way to emulating Stata's if syntax. But you are right it doesn't help with missings.

I think something along the lines of This PR in Missings.jl is the solution. Since we are constructing anonymous functions we can just add a spreadmissing(anon) when we need to. I hope we can make it performant. I don't know if it should be default in case people compare the speed to data.table, but I am open to the idea. It may also help people who don't like row-wise since a lot of the benefit of row-wise functions is dealing with missings.

That's a long term strategy. Maybe in the meantime we should just continue to treat missings as false since it's the default behavior with @where currently.

nalimilan
nalimilan reviewed Jun 28, 2021
View reviewed changes
Comment thread src/parsing.jl Outdated
Comment thread src/parsing.jl Outdated
Comment thread test/subset.jl
end

end # module No newline at end of file
@testset "@subset with a grouped data frame" begin

@nalimilan nalimilan Jun 28, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also test @subset! with GroupedDataFrame?

@pdeffebach pdeffebach Jun 28, 2021

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added!

pdeffebach and others added 5 commits June 28, 2021 14:36
@pdeffebach
switching
a75d817
@pdeffebach
@subset! with gd
7991b8c
@pdeffebach @nalimilan
Update src/parsing.jl
fa09626 
Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
@pdeffebach @nalimilan
Update src/parsing.jl
1c485c8 
Co-authored-by: Milan Bouchet-Valat <nalimilan@club.fr>
@pdeffebach
Merge branch 'add_subset' of https://github.com/pdeffebach/DataFrames…
8d55422 
…Meta.jl into add_subset
@pdeffebach

pdeffebach commented Jun 28, 2021

Copy link
Copy Markdown
Collaborator Author

Okay ready for merging.

@pdeffebach

pdeffebach commented Jun 29, 2021

Copy link
Copy Markdown
Collaborator Author

Thank you!

@pdeffebach pdeffebach merged commit fea3dee into JuliaData:master Jun 29, 2021
@pdeffebach pdeffebach deleted the add_subset branch June 29, 2021 13:25
@etpinard etpinard mentioned this pull request Jul 23, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nalimilan nalimilan nalimilan approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pdeffebach @nalimilan