Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

User-defined virtual path objects #110109

Closed
8 tasks done
barneygale opened this issue Sep 29, 2023 · 2 comments
Closed
8 tasks done

User-defined virtual path objects #110109

barneygale opened this issue Sep 29, 2023 · 2 comments
Assignees
Labels
stdlib Python modules in the Lib dir topic-pathlib type-feature A feature request or enhancement

Comments

@barneygale
Copy link
Contributor

barneygale commented Sep 29, 2023

Proposal

Add a PathBase class that can be subclassed by users to implement "virtual paths", e.g. paths within .zip files or on FTP servers, and publish it as an external PyPI package.

Phase 1: Private support in pathlib

Phase 2: Experimental support in PyPI package

Previous Discussion

https://discuss.python.org/t/make-pathlib-extensible/3428

Linked PRs

@barneygale barneygale added type-feature A feature request or enhancement topic-pathlib labels Sep 29, 2023
@barneygale barneygale self-assigned this Sep 29, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Oct 3, 2023
Use the class under test to create files, directories and symlinks.
barneygale added a commit to barneygale/cpython that referenced this issue Oct 3, 2023
…ther.

This is a very soft deprecation of `PurePath.as_uri()`. We instead document
it as a `Path` method, and add a couple of sentences mentioning that it's
also available in `PurePath`.
barneygale added a commit to barneygale/cpython that referenced this issue Oct 3, 2023
Add `PurePathBase` and `PathBase` abstract base classes.

`PathBase` can be subclassed to implement *virtual paths*: objects
representing paths on virtual filesystems, like archive files or remote
storage systems.
barneygale added a commit to barneygale/cpython that referenced this issue Oct 5, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Oct 5, 2023
- Add fast path to `_split_stack()`
- Skip unnecessarily resolution of the current directory when a relative
  path is given to `resolve()`
- Remove stat and target caches, which slow down most `resolve()` calls in
  practice.
- Slightly refactor code for clarity.
barneygale added a commit to barneygale/cpython that referenced this issue Oct 10, 2023
barneygale added a commit that referenced this issue Oct 10, 2023
Use the class under test to create files, directories and symlinks.
barneygale added a commit to barneygale/cpython that referenced this issue Oct 10, 2023
…classes.

This adds some test coverage to `pathlib._PathBase.walk()`
barneygale added a commit to barneygale/cpython that referenced this issue Oct 11, 2023
Add private `pathlib._PurePathBase` class: a private superclass of both
`PurePath` and `_PathBase`. Unlike `PurePath`, it does not define any of
these special methods: `__fspath__`, `__bytes__`, `__reduce__`, `__hash__`,
`__eq__`, `__lt__`, `__le__`, `__gt__`, `__ge__`.

This is important for supporting *virtual paths*: user subclasses of
`_PathBase` that provide access to archive files, FTP servers, etc. In
these classes, the above methods should be implemented by users only as
appropriate, with due consideration for the hash/equality of any backing
objects, such as file objects or sockets.
barneygale added a commit to barneygale/cpython that referenced this issue Oct 28, 2023
iritkatriel pushed a commit to iritkatriel/cpython that referenced this issue Oct 29, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Nov 12, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Nov 13, 2023
Re-arrange `pathlib.PurePath` methods in source code. No other changes.

The `PurePath` implementations of certain special methods, such as
`__eq__()` and `__hash__()`, are not usually applicable to user subclasses
of `_PathBase`. To facilitate their removal, another patch will split the
`PurePath` class into `_PurePathBase` and `PurePath`, with the latter
providing these special methods.

This patch prepares the ground for splitting `PurePath`. It's similar to
e8d77b0, which preceded splitting `Path`. By churning the methods here,
subsequent patches will be easier to review and less likely to break
things.
barneygale added a commit to barneygale/cpython that referenced this issue Nov 13, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Nov 13, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Nov 13, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Nov 17, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Nov 17, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Dec 16, 2023
Change the value of `pathlib._abc.PurePathBase.pathmod` from `os.path` to
`posixpath`.

User subclasses of `PurePathBase` and `PathBase` previously used the host
OS's path syntax, e.g. backslashes as separators on Windows. This is wrong
in most use cases, and likely to catch developers out unless they test on
both Windows and non-Windows machines.

In this patch we change the default to POSIX syntax, regardless of OS. This
is somewhat arguable (why not make all aspects of syntax abstract and
individually configurable?) but an improvement all the same.

This change has no effect on `PurePath`, `Path`, nor their subclasses. Only
private APIs are affected.
barneygale added a commit to barneygale/cpython that referenced this issue Dec 16, 2023
…tribute

Store the test base directory as a class attribute named `base` rather than
module constants named `BASE`.

The base directory is a local file path, and therefore not ideally suited
to the pathlib ABC tests. In a future commit we'll change its value in
`test_pathlib_abc.py` such that it points to a totally fictitious path.
barneygale added a commit that referenced this issue Dec 17, 2023
#113221)

Store the test base directory as a class attribute named `base` rather than
module constants named `BASE`.

The base directory is a local file path, and therefore not ideally suited
to the pathlib ABC tests. In a future commit we'll change its value in
`test_pathlib_abc.py` such that it points to a totally fictitious path, which 
will help to ensure we're not touching the local filesystem.
barneygale added a commit to barneygale/cpython that referenced this issue Dec 17, 2023
barneygale added a commit to barneygale/cpython that referenced this issue Dec 19, 2023
A regex group with a `?+` possessive quantifier was used to match empty
paths, which were represented as a single dot, preventing the dot from
being matched by other wildcards. This quantifier is only available from
Python 3.11+, but pathlib's `_abc.py` file will be made available as a PyPI
package for Python 3.8+.

This commit adds a new private `_pattern_str` property that works like
`__str__()` but represents empty paths as `''` rather than `'.'`. This
string is used for pattern matching, which removes the need for the
possessive group.

Improves compatibility with older Python; no other change of behaviour.
barneygale added a commit to barneygale/cpython that referenced this issue Dec 21, 2023
`PurePathBase.__repr__()` produces a string like `MyPath('/foo')`. This
repr is incorrect/misleading when a subclass's `__init__()` method is
customized, which I expect to be the very common.

This commit moves the `__repr__()` method to `PurePath`, leaving
`PurePathBase` with the default `object` repr.

No user-facing changes because the `pathlib._abc` module remains private.
barneygale added a commit that referenced this issue Dec 22, 2023
`PurePathBase.__repr__()` produces a string like `MyPath('/foo')`. This
repr is incorrect/misleading when a subclass's `__init__()` method is
customized, which I expect to be the very common.

This commit moves the `__repr__()` method to `PurePath`, leaving
`PurePathBase` with the default `object` repr.

No user-facing changes because the `pathlib._abc` module remains private.
barneygale added a commit to barneygale/cpython that referenced this issue Dec 22, 2023
barneygale added a commit that referenced this issue Dec 22, 2023
Change the value of `pathlib._abc.PurePathBase.pathmod` from `os.path` to
`posixpath`.

User subclasses of `PurePathBase` and `PathBase` previously used the host
OS's path syntax, e.g. backslashes as separators on Windows. This is wrong
in most use cases, and likely to catch developers out unless they test on
both Windows and non-Windows machines.

In this patch we change the default to POSIX syntax, regardless of OS. This
is somewhat arguable (why not make all aspects of syntax abstract and
individually configurable?) but an improvement all the same.

This change has no effect on `PurePath`, `Path`, nor their subclasses. Only
private APIs are affected.
barneygale added a commit to barneygale/cpython that referenced this issue Dec 22, 2023
This very boring patch reduces the number of changes needed in
`test_pathlib_abc.py` when backporting to the external `pathlib_abc`
package.
barneygale added a commit that referenced this issue Dec 22, 2023
…3411)

This very boring patch reduces the number of changes needed in
`test_pathlib_abc.py` when backporting to the external `pathlib_abc`
package.
barneygale added a commit to barneygale/cpython that referenced this issue Dec 22, 2023
Do not use the locale-specific default encoding in `PathBase.read_text()`
and `write_text()`. Locale settings shouldn't influence the operation of
these base classes, which are intended mostly for implementing rich paths
on *nonlocal* filesystems.
barneygale added a commit to barneygale/cpython that referenced this issue Dec 22, 2023
Do not use the locale-specific default encoding in `PathBase.read_text()`
and `write_text()`. Locale settings shouldn't influence the operation of
these base classes, which are intended mostly for implementing rich paths
on *nonlocal* filesystems.
barneygale added a commit to barneygale/cpython that referenced this issue Dec 23, 2023
The `pathlib._abc` module will be made available as a PyPI backport
supporting Python 3.8+. The `warnings._deprecated()` function was only
added last year, and it's private from an external package perspective, so
here we switch to `warnings.warn()` instead.
@barneygale
Copy link
Contributor Author

ryan-duve pushed a commit to ryan-duve/cpython that referenced this issue Dec 26, 2023
…hon#113376)

`PurePathBase.__repr__()` produces a string like `MyPath('/foo')`. This
repr is incorrect/misleading when a subclass's `__init__()` method is
customized, which I expect to be the very common.

This commit moves the `__repr__()` method to `PurePath`, leaving
`PurePathBase` with the default `object` repr.

No user-facing changes because the `pathlib._abc` module remains private.
ryan-duve pushed a commit to ryan-duve/cpython that referenced this issue Dec 26, 2023
…ython#113219)

Change the value of `pathlib._abc.PurePathBase.pathmod` from `os.path` to
`posixpath`.

User subclasses of `PurePathBase` and `PathBase` previously used the host
OS's path syntax, e.g. backslashes as separators on Windows. This is wrong
in most use cases, and likely to catch developers out unless they test on
both Windows and non-Windows machines.

In this patch we change the default to POSIX syntax, regardless of OS. This
is somewhat arguable (why not make all aspects of syntax abstract and
individually configurable?) but an improvement all the same.

This change has no effect on `PurePath`, `Path`, nor their subclasses. Only
private APIs are affected.
ryan-duve pushed a commit to ryan-duve/cpython that referenced this issue Dec 26, 2023
python#113411)

This very boring patch reduces the number of changes needed in
`test_pathlib_abc.py` when backporting to the external `pathlib_abc`
package.
barneygale added a commit that referenced this issue Dec 27, 2023
Do not use the locale-specific default encoding in `PathBase.read_text()`
and `write_text()`. Locale settings shouldn't influence the operation of
these base classes, which are intended mostly for implementing rich paths
on *nonlocal* filesystems.
barneygale added a commit that referenced this issue Dec 27, 2023
The `pathlib._abc` module will be made available as a PyPI backport
supporting Python 3.8+. The `warnings._deprecated()` function was only
added last year, and it's private from an external package perspective, so
here we switch to `warnings.warn()` instead.
@barneygale
Copy link
Contributor Author

pathlib-abc==0.1.0 available now! https://pypi.org/project/pathlib-abc/0.1.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stdlib Python modules in the Lib dir topic-pathlib type-feature A feature request or enhancement
Projects
None yet
Development

No branches or pull requests

2 participants