Skip to content

shutil.copytree doesn't seem to understand NTFS junctions, can truncates files when tricked #104046

Open
@LazyDodo

Description

@LazyDodo

Bug report

Minimal repro extracted from Blender Issue #99766 , When copying a folder containing a directory junction some odd behavior can be observed.

When called with symlinks=True when copying a folder, the junction will be lost even though the documentation states If symlinks is true, symbolic links in the source tree are represented as symbolic links in the new tree and the metadata of the original links will be copied as far as the platform allows; while junctions vs symlinks generally can lead to a passionate debate which, for the sake of simplicity i'd like to skip over, and rather compare the behavior to the build in copy tools in windows, xcopy also loses the junction, so while it's inconvenient it's not completely unexpected behavior, shutil.copytree gets a pass here in my book.

However... there is some destructive behavior shutil.copytree has xcopy does not, when copying to folders on top of each other with both already containing a junction point pointing to the same folder.

with a directory structure like this

my-app/
├─ source/
│  ├─ Hello.py
├─ Project1/
│  ├─ test[Junction to my-app/source]/
│    ├─ Hello.py
├─ Project2/
│  ├─ test[Junction to my-app/source]/
│    ├─ Hello.py

When calling shutil.copytree to copy Project1 on top of Project2 it will try to create a new file Project2/test/Hello.py which truncates the original source/Hello.py which is a bit more destructive than I would have liked.

Repro:

REM repro.cmd
set WORK_DIR=%~dp0
REM Create some important files
mkdir source
echo print("world") > source/hello.py

REM Create a project with a junction to the source folder
mkdir project1
mklink /J "%WORK_DIR%/project1/test" "%WORK_DIR%/source"

REM Create a second project with a junction to the source folder
mkdir project2
mklink /J "%WORK_DIR%/project2/test" "%WORK_DIR%/source"

dir /s > before.txt

REM Case 1: Copy project1 to a whole new folder.
REM This will copy the files, but will lose the junction even though we asked to copy symlinks.
REM After the copy project3/test/hello.py and source/hello.py will be individual files changes to one do not affect the other.
python.exe -c "import shutil; shutil.copytree(r'%WORK_DIR%project1',r'%WORK_DIR%project3', dirs_exist_ok=True, symlinks=True)"
dir /s > case1.txt

REM Case 2: Copy 2 folders with identical junctions already in place on top of each other.
REM This is much MUCH more dangerous as copytree appears to create a new file for project2/test/hello.py
REM which truncates the original source/hi.py to 0 bytes.
python.exe -c "import shutil; shutil.copytree(r'%WORK_DIR%project1',r'%WORK_DIR%project2', dirs_exist_ok=True, symlinks=True)"
dir /s > case2.txt

While the example is squarely in the "seems far fetched, why would you even do that!?" category, sadly this was extracted from a real life scenario where people did lose their work Blender Issue #99766

Your environment

  • CPython versions tested on: 3.10/3.11
  • Operating system and architecture: Windows 10/X64

Metadata

Metadata

Assignees

No one assigned

    Labels

    OS-windowsstdlibPython modules in the Lib dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions