diff --git a/Makefile b/Makefile index 7bcc836..088eb8d 100644 --- a/Makefile +++ b/Makefile @@ -25,16 +25,16 @@ lint-py: ## lint python with ruff python -m ruff format --check fsspec_python lint-docs: ## lint docs with mdformat and codespell - python -m mdformat --check README.md $(wildcard docs/src) - python -m codespell_lib README.md $(wildcard docs/src) + python -m mdformat --check README.md docs/*.md + python -m codespell_lib --ignore-words-list fo README.md docs/*.md fix-py: ## autoformat python code with ruff python -m ruff check --fix fsspec_python python -m ruff format fsspec_python fix-docs: ## autoformat docs with mdformat and codespell - python -m mdformat README.md $(wildcard docs/src) - python -m codespell_lib --write README.md $(wildcard docs/src) + python -m mdformat README.md docs/*.md + python -m codespell_lib --ignore-words-list fo --write README.md docs/*.md lint: lint-py lint-docs ## run all linters lints: lint diff --git a/README.md b/README.md index f522b6d..7ddee2f 100644 --- a/README.md +++ b/README.md @@ -1,37 +1,23 @@ # fsspec-python -Native python integration for fsspec backends +Import Python source modules from fsspec filesystems. [![Build Status](https://github.com/1kbgz/fsspec-python/actions/workflows/build.yaml/badge.svg?branch=main&event=push)](https://github.com/1kbgz/fsspec-python/actions/workflows/build.yaml) [![codecov](https://codecov.io/gh/1kbgz/fsspec-python/branch/main/graph/badge.svg)](https://codecov.io/gh/1kbgz/fsspec-python) [![License](https://img.shields.io/github/license/1kbgz/fsspec-python)](https://github.com/1kbgz/fsspec-python) [![PyPI](https://img.shields.io/pypi/v/fsspec-python.svg)](https://pypi.python.org/pypi/fsspec-python) -## Overview +`fsspec-python` adds a filesystem-backed finder to Python's import machinery and +provides an optional hook for `builtins.open`. Importers read source files from a +configured root; imported code runs in the current Python process. -This library wraps an [fsspec filesystem](https://github.com/fsspec/filesystem_spec) so that files and folders inside it can be imported in Python. +## Documentation -```python -from fsspec import open +Start with [Import a module from memory](docs/tutorial.md), a complete local lesson +that needs no remote service. -open(f"python::s3://my-python-bucket") - -import my_bucket_lib # s3://my-python-bucket/my_bucket_lib.py -``` - -## Usage - -In addition to the `python::` [chained protocol](https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining) url, this library exposes a handful of functions and monkey patches for connecting Python internal mechanisms to an `fsspec`-based filesystem. - -- install_importer: install an -- `install_importer(fs: Union[str, AbstractFileSystem], **kwargs)`: install an fsspec url/args or filesystem instance as an importer -- `uninstall_importer(fs: Union[str, AbstractFileSystem] = "")`: uninstall an fsspec url or filesystem instance, or if nothing provided remove the last-installed one -- `install_open_hook(fsspec: str, **fsspec_args)`: monkeypatch Python's `open` builtin to run off an `fsspec` filesystem -- `uninstall_open_hook()`: remove the monkeypatched `open` - -Coming soon: - -- Monkey patching for `os` / `os.path` / `pathlib` - -> [!NOTE] -> This library was generated using [copier](https://copier.readthedocs.io/en/stable/) from the [Base Python Project Template repository](https://github.com/python-project-templates/base). +For application work, [install and remove importers](docs/how-to-import.md) or +[redirect file access](docs/how-to-open.md). The [API reference](docs/reference.md) +covers signatures, return values, lookup order, and limits. Read +[How filesystem imports interact with Python](docs/explanation.md) for the design +and process-wide effects. diff --git a/docs/explanation.md b/docs/explanation.md new file mode 100644 index 0000000..8ae39f4 --- /dev/null +++ b/docs/explanation.md @@ -0,0 +1,35 @@ +# How filesystem imports interact with Python + +Python asks finders in `sys.meta_path` to locate modules that are not already in +`sys.modules`. An fsspec finder adds another source location without copying that +location into a local package installation. + +## A filesystem root acts as a source directory + +The finder looks for a source file under its configured root and lets Python's +source loader execute the bytes. Backend authentication and storage access remain +fsspec responsibilities. The importer handles module lookup and lifetime. + +This fits directories of standalone source modules. It does not implement the +package search rules needed for package directories or namespace packages. + +## Registration affects the process + +New finders go before existing finders, so a newly installed root can supply a name +that would otherwise resolve elsewhere. Python's module cache still takes +precedence: installing another finder does not replace an already imported module. + +Removing an importer removes tracked entries from the cache. Code that retained a +reference to a module or function can still call it. Cleanup therefore removes +future lookup paths; it does not undo code execution or its side effects. + +## File access is independent of import lookup + +The open hook redirects calls to `builtins.open`, while the finder redirects module +lookup. Either can be used independently. The open hook's fallback for missing read +paths lets existing local file reads continue, but writes go to the configured +backend. Both mechanisms affect the whole process, which makes explicit cleanup +part of their lifecycle. + +See the [tutorial](tutorial.md) for a complete importer lifecycle and the +[reference](reference.md) for lookup limits. diff --git a/docs/how-to-import.md b/docs/how-to-import.md new file mode 100644 index 0000000..9af405e --- /dev/null +++ b/docs/how-to-import.md @@ -0,0 +1,57 @@ +# How to install and remove importers + +Point an importer at a directory containing `.py` modules. Install the fsspec +backend required by the URL in the same environment. + +## Use a URL + +Pass the root URL to `install_importer`, and retain the returned filesystem for +cleanup: + +```python +from fsspec_python import install_importer, uninstall_importer + +importer = install_importer("s3://my-python-bucket/modules") +try: + import reporting_helpers + + result = reporting_helpers.build_report() +finally: + uninstall_importer(importer) +``` + +Configure backend credentials through fsspec before installing the importer. +Use an explicit filesystem instance when passing backend options directly: + +```python +import fsspec + +from fsspec_python import install_importer, uninstall_importer + +fs = fsspec.filesystem("s3", profile="reports") +importer = install_importer(fs, fo="my-python-bucket/modules") +try: + import reporting_helpers +finally: + uninstall_importer(importer) +``` + +## Use a chained URL + +Resolve `python::` with `url_to_fs`; call `exit()` when finished: + +```python +from fsspec import url_to_fs + +import fsspec_python + +fs, _ = url_to_fs("python::file:///srv/python-modules") +try: + import reporting_helpers +finally: + fs.exit() +``` + +Use source modules from a trusted location: importing executes their code with +this process's permissions. See the [reference](reference.md) for supported module +layouts and cleanup behavior. diff --git a/docs/how-to-open.md b/docs/how-to-open.md new file mode 100644 index 0000000..a779646 --- /dev/null +++ b/docs/how-to-open.md @@ -0,0 +1,26 @@ +# How to redirect file access + +Use the open hook when code calls `builtins.open` and needs a filesystem-backed +root. Install the backend required by the URL first. + +```python +from fsspec_python import install_open_hook, uninstall_open_hook + +install_open_hook("memory://open-demo") +try: + with open("result.txt", "w") as output: + output.write("complete\n") + with open("result.txt") as source: + print(source.read().strip()) +finally: + uninstall_open_hook() +``` + +The output is `complete`. Replace the memory URL with your target root, and pass +backend storage options as keyword arguments to `install_open_hook`. + +Keep the hook's lifetime short and restore it in `finally`. It changes +`builtins.open` for the whole process. Missing read paths fall back to the original +local `open`; write, append, and exclusive-create paths use the configured backend. + +See [API reference](reference.md) for argument handling and unsupported hooks. diff --git a/docs/reference.md b/docs/reference.md new file mode 100644 index 0000000..e9b193d --- /dev/null +++ b/docs/reference.md @@ -0,0 +1,62 @@ +# Python integration reference + +Public names are exported from `fsspec_python` unless stated otherwise. + +## Importers + +### `install_importer(fs, **kwargs)` + +Accepts a URL string or `AbstractFileSystem`. For an instance, `fo` supplies the +module root. A URL already supplies its root; passing `fo` with a string raises +`AssertionError`. Other input types raise `TypeError`. + +Returns the installed `PythonFileSystem`. Repeated installation of the same normalized root reuses its importer. +New finders are inserted at the front of `sys.meta_path`. + +### `uninstall_importer(fs="")` + +Accepts the original URL or returned filesystem. An empty argument removes the +last installed importer. With no installed importers, it does nothing. An unknown +root raises `ValueError` when other importers remain installed. + +Removal deletes the finder's tracked module names from `sys.modules` and removes +the finder from `sys.meta_path`. Existing references to module objects remain valid. + +### `PythonFileSystem(target_protocol=None, target_options=None, fs=None, install=True, **kwargs)` + +Chained filesystem with protocol `python`. Exactly one of `target_protocol` and +`fs` is required. `target_options` configures a newly constructed target; `fo` sets +the source root for an unchained target. `install=False` suppresses automatic +importer registration. `exit()` removes the importer and calls the target's +`exit()` when available. + +### `FSSpecImportFinder` and `FSSpecImportLoader` + +The finder checks the configured root for the final component of a module name +plus a Python source suffix. Matches have `is_package=False`. Package directories, +namespace packages, native extensions, and normal package-relative search are not +implemented. The loader reads source bytes with the filesystem's `open`. + +## Open hook + +### `install_open_hook(fsspec, **fsspec_args)` + +Resolves a root URL with fsspec and replaces `builtins.open`. Read paths present on +the backend use that backend; missing read paths use the original builtin. +Modes containing `w`, `a`, or `x` always use the backend. + +Backend calls receive `mode`, `encoding`, `errors`, and `newline`. `buffering`, +`closefd`, and `opener` are forwarded only on the local fallback path. Nested +installations do not form a restoration stack. + +### `uninstall_open_hook()` + +Restores the builtin saved when `fsspec_python.open` was imported. + +## Other modules + +`fsspec_python.utils.normalize_fsspec(fs=None, target_protocol=None, target_options=None, fo="", **kwargs)` returns a protocol/root string. Exactly +one of `fs` and `target_protocol` is required. The key does not include credentials +or storage options. + +The `path` and `pathlib` modules contain no implemented hooks. diff --git a/docs/tutorial.md b/docs/tutorial.md new file mode 100644 index 0000000..d50d814 --- /dev/null +++ b/docs/tutorial.md @@ -0,0 +1,63 @@ +# Import a module from memory + +We will create a Python module in an in-memory filesystem, import it, and remove +its importer. + +## Install + +Use Python 3.10 or newer: + +```console +python -m pip install fsspec-python +``` + +## Create a module + +Create `import_demo.py`: + +```python +import fsspec + +from fsspec_python import install_importer, uninstall_importer + +fs = fsspec.filesystem("memory") +fs.pipe_file("/import-demo/hello_remote.py", b'def greet():\n return "Hello from memory"\n') +print(fs.ls("/import-demo", detail=False)) +``` + +Run `python import_demo.py`. The listing contains `/import-demo/hello_remote.py`. + +## Import and call the function + +Append: + +```python +importer = install_importer(fs, fo="/import-demo") +try: + import hello_remote + + print(hello_remote.greet()) +finally: + uninstall_importer(importer) +``` + +Run the script again. Its final line is: + +```text +Hello from memory +``` + +## Check cleanup + +Append: + +```python +import sys + +print("hello_remote" in sys.modules) +``` + +Run the script. The final line is `False`: removing the importer also removed its +tracked module from Python's module cache. + +See [How to install and remove importers](how-to-import.md) to use a filesystem URL. diff --git a/pyproject.toml b/pyproject.toml index 197b9f8..b8dbddc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -145,3 +145,15 @@ section-order = [ "F401", "F403", ] + +[tool.yardang] +title = "fsspec-python" +root = "README.md" +pages = [ + "docs/tutorial.md", + "docs/how-to-import.md", + "docs/how-to-open.md", + "docs/reference.md", + "docs/explanation.md", +] +use-autoapi = false