Skip to content

/ pandas

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI add end-of-file-fixer #36826

Merged
merged 13 commits into from Oct 6, 2020
Merged

CI add end-of-file-fixer #36826

merged 13 commits into from Oct 6, 2020

Conversation

@MarcoGorelli
Copy link
Member

@MarcoGorelli MarcoGorelli commented Oct 3, 2020 •
edited

xref #36600 (comment)

ok if you'd fix the EOF issue can merge (and yes good idea to add to the pre-commit hook / separate PR pls)

Note: in addition to adding the end-of-file-fixer pre-commit hook, I applied the following

diff --git a/scripts/generate_pip_deps_from_conda.py b/scripts/generate_pip_deps_from_conda.py
index c417f58f6..c6d00eb58 100755
--- a/scripts/generate_pip_deps_from_conda.py
+++ b/scripts/generate_pip_deps_from_conda.py
@@ -94,7 +94,7 @@ def main(conda_fname, pip_fname, compare=False):
         f"# This file is auto-generated from {fname}, do not modify.\n"
         "# See that file for comments about the need/usage of each dependency.\n\n"
     )
-    pip_content = header + "\n".join(pip_deps)
+    pip_content = header + "\n".join(pip_deps) + "\n"
 
     if compare:
         with open(pip_fname) as pip_fd:

I don't know how many of these we want to apply this to, for now I'm just excluding licenses (which I don't think should be touched?), html, txt, svg,and csv files
MarcoGorelli added 7 commits Oct 3, 2020
@MarcoGorelli
end-of-file-fixer
8a589f0
@MarcoGorelli
remove comment
9171e6e
@MarcoGorelli
exclude licenses, html, and csv
22ecece
@MarcoGorelli
regex
b1e554f
@MarcoGorelli
exclude .txt
7e30f16
@MarcoGorelli
exclude svg
69cd87c
@MarcoGorelli
noop
d5c18ed
@dsaxton
Copy link
Member

@dsaxton dsaxton commented Oct 3, 2020

I actually get an error when I try to use this or the yesqa hook (it's the same error), do you know what this might be about:

(pandas-dev) ➜  pandas git:(end-of-file-fixer) git commit --allow-empty
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Installing environment for https://github.com/asottile/yesqa.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python', '-mvirtualenv', '/Users/danielsaxton/.cache/pre-commit/repowwvsgzcw/py_env-python3.8', '-p', '/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python')
return code: 1
expected return code: 0
stdout:
    ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'

stderr: (none)
Check the log at /Users/danielsaxton/.cache/pre-commit/pre-commit.log

Full log:

version information

pre-commit version: 2.7.1
sys.version:
    3.8.5 | packaged by conda-forge | (default, Sep 24 2020, 16:37:41) 
    [Clang 10.0.1 ]
sys.executable: /Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python
os.name: posix
sys.platform: darwin

error information

An unexpected error has occurred: CalledProcessError: command: ('/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python', '-mvirtualenv', '/Users/danielsaxton/.cache/pre-commit/repowwvsgzcw/py_env-python3.8', '-p', '/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python')
return code: 1
expected return code: 0
stdout:
    ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'
    
stderr: (none)

Traceback (most recent call last):
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/error_handler.py", line 63, in error_handler
    yield
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/main.py", line 362, in main
    return hook_impl(
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/commands/hook_impl.py", line 217, in hook_impl
    return retv | run(config, store, ns)
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/commands/run.py", line 388, in run
    install_hook_envs(hooks, store)
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/repository.py", line 206, in install_hook_envs
    _hook_install(hook)
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/repository.py", line 82, in _hook_install
    lang.install_environment(
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/languages/python.py", line 213, in install_environment
    cmd_output_b(*venv_cmd, cwd='/')
  File "/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pre_commit/util.py", line 157, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python', '-mvirtualenv', '/Users/danielsaxton/.cache/pre-commit/repowwvsgzcw/py_env-python3.8', '-p', '/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python')
return code: 1
expected return code: 0
stdout:
    ModuleNotFoundError: No module named 'virtualenv.seed.via_app_data'
    
stderr: (none)

@MarcoGorelli
Copy link
Member Author

@MarcoGorelli MarcoGorelli commented Oct 3, 2020 •
edited

I've seen that before, yes - I think pip install -U virtualenv?

@dsaxton
Copy link
Member

@dsaxton dsaxton commented Oct 3, 2020

I did that and a different error happens:

(pandas-dev) ➜  pandas git:(dummy-branch) git commit --allow-empty
[INFO] Installing environment for https://github.com/asottile/yesqa.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python', '-mvirtualenv', '/Users/danielsaxton/.cache/pre-commit/repowwvsgzcw/py_env-python3.8', '-p', '/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/bin/python')
return code: 1
expected return code: 0
stdout:
    ImportError: cannot import name 'ensure_file_on_disk' from 'virtualenv.util.zipapp' (/Users/danielsaxton/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/virtualenv/util/zipapp.py)
    
stderr: (none)
Check the log at /Users/danielsaxton/.cache/pre-commit/pre-commit.log

Might be the same issue as here: pypa/virtualenv#1876

@dsaxton
Copy link
Member

@dsaxton dsaxton commented Oct 3, 2020 •
edited

I think I was able to fix it by uninstalling virtualenv in my base conda environment and then reinstalling virtualenv in my pandas-dev environment, but I wonder if this warrants some type of note somewhere for anyone who might have the same problem?

MarcoGorelli added 5 commits Oct 4, 2020
@MarcoGorelli
Merge remote-tracking branch 'upstream/master' into end-of-file-fixer
7ea250e
@MarcoGorelli
add virtualenv to environment
f8cf7bf
@MarcoGorelli
revert
dbc47cd
@MarcoGorelli
document potential virtualenv conflict
0f96f6a
@MarcoGorelli
link to pypa issue
c126373
@dsaxton
dsaxton reviewed Oct 4, 2020
View changes
.pre-commit-config.yaml Outdated
rev: v3.2.0
hooks:
- id: end-of-file-fixer
exclude: '.html$|^LICENSES/|.csv$|.txt$|.svg$'

This comment has been minimized.

Sign in to view
@dsaxton

dsaxton Oct 4, 2020
Member

I wonder if we might actually want to exclude .py files and only check others (e.g., .yml), since I think .py files will already fail black if they're missing the EOF newline?
@MarcoGorelli
Update .pre-commit-config.yaml
523e68a
@jreback
jreback requested changes Oct 5, 2020
View changes
doc/source/development/contributing.rst Show resolved Hide resolved
@jreback jreback added this to the 1.2 milestone Oct 6, 2020
@jreback
jreback approved these changes Oct 6, 2020
View changes
@jreback jreback merged commit 3cf204a into pandas-dev:master Oct 6, 2020
16 checks passed
16 checks passed
@github-actions
Checks
Details
@github-actions
pre-commit
Details
@github-actions
Web and docs
Details
@codecov
codecov/patch Codecov Report
Details
@codecov
codecov/project Codecov Report
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@azure-pipelines
pandas-dev.pandas Build #20201005.25 succeeded
Details
@azure-pipelines
pandas-dev.pandas (Linux py37_locale) Linux py37_locale succeeded
Details
@azure-pipelines
pandas-dev.pandas (Linux py37_locale_slow) Linux py37_locale_slow succeeded
Details
@azure-pipelines
pandas-dev.pandas (Linux py37_minimum_versions) Linux py37_minimum_versions succeeded
Details
@azure-pipelines
pandas-dev.pandas (Linux py37_slow) Linux py37_slow succeeded
Details
@azure-pipelines
pandas-dev.pandas (Linux py38_locale) Linux py38_locale succeeded
Details
@azure-pipelines
pandas-dev.pandas (Linux py38_np_dev) Linux py38_np_dev succeeded
Details
@azure-pipelines
pandas-dev.pandas (Windows py37_np16) Windows py37_np16 succeeded
Details
@azure-pipelines
pandas-dev.pandas (Windows py38_np18) Windows py38_np18 succeeded
Details
@azure-pipelines
pandas-dev.pandas (macOS py37_macos) macOS py37_macos succeeded
Details
@MarcoGorelli MarcoGorelli deleted the MarcoGorelli:end-of-file-fixer branch Oct 7, 2020
kesmit13 added a commit to kesmit13/pandas that referenced this pull request Nov 2, 2020
@MarcoGorelli @kesmit13
CI add end-of-file-fixer (pandas-dev#36826)
023e418
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dsaxton dsaxton

@jreback jreback

Assignees
No one assigned
Labels
CI Code Style
Projects
None yet
Milestone
1.2
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
@MarcoGorelli @dsaxton @jreback @jbrockmendel