Skip to content

Fix map remove_columns on empty dataset #4021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 29, 2022
Merged

Fix map remove_columns on empty dataset #4021

merged 1 commit into from
Mar 29, 2022

Conversation

lhoestq
Copy link
Member

@lhoestq lhoestq commented Mar 25, 2022

On an empty dataset, the remove_columns parameter of map currently doesn't actually remove the columns:

>>> ds = datasets.load_dataset("glue", "rte")
>>> ds_filtered = ds.filter(lambda x: x["label"] != -1)
>>> ds_mapped = ds_filtered.map(lambda x: x, remove_columns=["label"])
>>> print(repr(ds_mapped.column_names))
{
    'train': ['sentence1', 'sentence2', 'idx'],
    'validation': ['sentence1', 'sentence2', 'idx'],
    'test': ['sentence1', 'sentence2', 'label', 'idx']
}

I fixed this error and updated the tests

@lhoestq lhoestq requested a review from albertvillanova March 25, 2022 13:36
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Mar 25, 2022 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

albertvillanova
albertvillanova approved these changes Mar 29, 2022
View reviewed changes
Copy link
Member

@albertvillanova albertvillanova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Thanks.

@lhoestq lhoestq merged commit 34ff2d7 into master Mar 29, 2022
@lhoestq lhoestq deleted the fix-map-remove_columns-on-empty-dataset branch March 29, 2022 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@albertvillanova albertvillanova albertvillanova approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lhoestq @HuggingFaceDocBuilderDev @albertvillanova