Reorganize documentation with Diataxis

Makefile

@@ -25,16 +25,16 @@ lint-py:  ## lint python with ruff
 	python -m ruff format --check fsspec_python
 
 lint-docs:  ## lint docs with mdformat and codespell
-	python -m mdformat --check README.md $(wildcard docs/src) 
-	python -m codespell_lib README.md $(wildcard docs/src) 
+	python -m mdformat --check README.md docs/*.md
+	python -m codespell_lib --ignore-words-list fo README.md docs/*.md
 
 fix-py:  ## autoformat python code with ruff
 	python -m ruff check --fix fsspec_python
 	python -m ruff format fsspec_python
 
 fix-docs:  ## autoformat docs with mdformat and codespell
-	python -m mdformat README.md $(wildcard docs/src) 
-	python -m codespell_lib --write README.md $(wildcard docs/src) 
+	python -m mdformat README.md docs/*.md
+	python -m codespell_lib --ignore-words-list fo --write README.md docs/*.md
 
 lint: lint-py lint-docs  ## run all linters
 lints: lint

README.md

@@ -1,37 +1,23 @@
 # fsspec-python
 
-Native python integration for fsspec backends
+Import Python source modules from fsspec filesystems.
 
 [![Build Status](https://github.com/1kbgz/fsspec-python/actions/workflows/build.yaml/badge.svg?branch=main&event=push)](https://github.com/1kbgz/fsspec-python/actions/workflows/build.yaml)
 [![codecov](https://codecov.io/gh/1kbgz/fsspec-python/branch/main/graph/badge.svg)](https://codecov.io/gh/1kbgz/fsspec-python)
 [![License](https://img.shields.io/github/license/1kbgz/fsspec-python)](https://github.com/1kbgz/fsspec-python)
 [![PyPI](https://img.shields.io/pypi/v/fsspec-python.svg)](https://pypi.python.org/pypi/fsspec-python)
 
-## Overview
+`fsspec-python` adds a filesystem-backed finder to Python's import machinery and
+provides an optional hook for `builtins.open`. Importers read source files from a
+configured root; imported code runs in the current Python process.
 
-This library wraps an [fsspec filesystem](https://github.com/fsspec/filesystem_spec) so that files and folders inside it can be imported in Python.
+## Documentation
 
-```python
-from fsspec import open
+Start with [Import a module from memory](docs/tutorial.md), a complete local lesson
+that needs no remote service.
 
-open(f"python::s3://my-python-bucket")
-
-import my_bucket_lib  # s3://my-python-bucket/my_bucket_lib.py
-```
-
-## Usage
-
-In addition to the `python::` [chained protocol](https://filesystem-spec.readthedocs.io/en/latest/features.html#url-chaining) url, this library exposes a handful of functions and monkey patches for connecting Python internal mechanisms to an `fsspec`-based filesystem.
-
-- install_importer: install an
-- `install_importer(fs: Union[str, AbstractFileSystem], **kwargs)`: install an fsspec url/args or filesystem instance as an importer
-- `uninstall_importer(fs: Union[str, AbstractFileSystem] = "")`: uninstall an fsspec url or filesystem instance, or if nothing provided remove the last-installed one
-- `install_open_hook(fsspec: str, **fsspec_args)`: monkeypatch Python's `open` builtin to run off an `fsspec` filesystem
-- `uninstall_open_hook()`: remove the monkeypatched `open`
-
-Coming soon:
-
-- Monkey patching for `os` / `os.path` / `pathlib`
-
-> [!NOTE]
-> This library was generated using [copier](https://copier.readthedocs.io/en/stable/) from the [Base Python Project Template repository](https://github.com/python-project-templates/base).
+For application work, [install and remove importers](docs/how-to-import.md) or
+[redirect file access](docs/how-to-open.md). The [API reference](docs/reference.md)
+covers signatures, return values, lookup order, and limits. Read
+[How filesystem imports interact with Python](docs/explanation.md) for the design
+and process-wide effects.

docs/explanation.md

@@ -0,0 +1,35 @@
+# How filesystem imports interact with Python
+
+Python asks finders in `sys.meta_path` to locate modules that are not already in
+`sys.modules`. An fsspec finder adds another source location without copying that
+location into a local package installation.
+
+## A filesystem root acts as a source directory
+
+The finder looks for a source file under its configured root and lets Python's
+source loader execute the bytes. Backend authentication and storage access remain
+fsspec responsibilities. The importer handles module lookup and lifetime.
+
+This fits directories of standalone source modules. It does not implement the
+package search rules needed for package directories or namespace packages.
+
+## Registration affects the process
+
+New finders go before existing finders, so a newly installed root can supply a name
+that would otherwise resolve elsewhere. Python's module cache still takes
+precedence: installing another finder does not replace an already imported module.
+
+Removing an importer removes tracked entries from the cache. Code that retained a
+reference to a module or function can still call it. Cleanup therefore removes
+future lookup paths; it does not undo code execution or its side effects.
+
+## File access is independent of import lookup
+
+The open hook redirects calls to `builtins.open`, while the finder redirects module
+lookup. Either can be used independently. The open hook's fallback for missing read
+paths lets existing local file reads continue, but writes go to the configured
+backend. Both mechanisms affect the whole process, which makes explicit cleanup
+part of their lifecycle.
+
+See the [tutorial](tutorial.md) for a complete importer lifecycle and the
+[reference](reference.md) for lookup limits.

docs/how-to-import.md

@@ -0,0 +1,57 @@
+# How to install and remove importers
+
+Point an importer at a directory containing `.py` modules. Install the fsspec
+backend required by the URL in the same environment.
+
+## Use a URL
+
+Pass the root URL to `install_importer`, and retain the returned filesystem for
+cleanup:
+
+```python
+from fsspec_python import install_importer, uninstall_importer
+
+importer = install_importer("s3://my-python-bucket/modules")
+try:
+    import reporting_helpers
+
+    result = reporting_helpers.build_report()
+finally:
+    uninstall_importer(importer)
+```
+
+Configure backend credentials through fsspec before installing the importer.
+Use an explicit filesystem instance when passing backend options directly:
+
+```python
+import fsspec
+
+from fsspec_python import install_importer, uninstall_importer
+
+fs = fsspec.filesystem("s3", profile="reports")
+importer = install_importer(fs, fo="my-python-bucket/modules")
+try:
+    import reporting_helpers
+finally:
+    uninstall_importer(importer)
+```
+
+## Use a chained URL
+
+Resolve `python::` with `url_to_fs`; call `exit()` when finished:
+
+```python
+from fsspec import url_to_fs
+
+import fsspec_python
+
+fs, _ = url_to_fs("python::file:///srv/python-modules")
+try:
+    import reporting_helpers
+finally:
+    fs.exit()
+```
+
+Use source modules from a trusted location: importing executes their code with
+this process's permissions. See the [reference](reference.md) for supported module
+layouts and cleanup behavior.

docs/how-to-open.md

@@ -0,0 +1,26 @@
+# How to redirect file access
+
+Use the open hook when code calls `builtins.open` and needs a filesystem-backed
+root. Install the backend required by the URL first.
+
+```python
+from fsspec_python import install_open_hook, uninstall_open_hook
+
+install_open_hook("memory://open-demo")
+try:
+    with open("result.txt", "w") as output:
+        output.write("complete\n")
+    with open("result.txt") as source:
+        print(source.read().strip())
+finally:
+    uninstall_open_hook()
+```
+
+The output is `complete`. Replace the memory URL with your target root, and pass
+backend storage options as keyword arguments to `install_open_hook`.
+
+Keep the hook's lifetime short and restore it in `finally`. It changes
+`builtins.open` for the whole process. Missing read paths fall back to the original
+local `open`; write, append, and exclusive-create paths use the configured backend.
+
+See [API reference](reference.md) for argument handling and unsupported hooks.

docs/reference.md

@@ -0,0 +1,62 @@
+# Python integration reference
+
+Public names are exported from `fsspec_python` unless stated otherwise.
+
+## Importers
+
+### `install_importer(fs, **kwargs)`
+
+Accepts a URL string or `AbstractFileSystem`. For an instance, `fo` supplies the
+module root. A URL already supplies its root; passing `fo` with a string raises
+`AssertionError`. Other input types raise `TypeError`.
+
+Returns the installed `PythonFileSystem`. Repeated installation of the same normalized root reuses its importer.
+New finders are inserted at the front of `sys.meta_path`.
+
+### `uninstall_importer(fs="")`
+
+Accepts the original URL or returned filesystem. An empty argument removes the
+last installed importer. With no installed importers, it does nothing. An unknown
+root raises `ValueError` when other importers remain installed.
+
+Removal deletes the finder's tracked module names from `sys.modules` and removes
+the finder from `sys.meta_path`. Existing references to module objects remain valid.
+
+### `PythonFileSystem(target_protocol=None, target_options=None, fs=None, install=True, **kwargs)`
+
+Chained filesystem with protocol `python`. Exactly one of `target_protocol` and
+`fs` is required. `target_options` configures a newly constructed target; `fo` sets
+the source root for an unchained target. `install=False` suppresses automatic
+importer registration. `exit()` removes the importer and calls the target's
+`exit()` when available.
+
+### `FSSpecImportFinder` and `FSSpecImportLoader`
+
+The finder checks the configured root for the final component of a module name
+plus a Python source suffix. Matches have `is_package=False`. Package directories,
+namespace packages, native extensions, and normal package-relative search are not
+implemented. The loader reads source bytes with the filesystem's `open`.
+
+## Open hook
+
+### `install_open_hook(fsspec, **fsspec_args)`
+
+Resolves a root URL with fsspec and replaces `builtins.open`. Read paths present on
+the backend use that backend; missing read paths use the original builtin.
+Modes containing `w`, `a`, or `x` always use the backend.
+
+Backend calls receive `mode`, `encoding`, `errors`, and `newline`. `buffering`,
+`closefd`, and `opener` are forwarded only on the local fallback path. Nested
+installations do not form a restoration stack.
+
+### `uninstall_open_hook()`
+
+Restores the builtin saved when `fsspec_python.open` was imported.
+
+## Other modules
+
+`fsspec_python.utils.normalize_fsspec(fs=None, target_protocol=None, target_options=None, fo="", **kwargs)` returns a protocol/root string. Exactly
+one of `fs` and `target_protocol` is required. The key does not include credentials
+or storage options.
+
+The `path` and `pathlib` modules contain no implemented hooks.

docs/tutorial.md

@@ -0,0 +1,63 @@
+# Import a module from memory
+
+We will create a Python module in an in-memory filesystem, import it, and remove
+its importer.
+
+## Install
+
+Use Python 3.10 or newer:
+
+```console
+python -m pip install fsspec-python
+```
+
+## Create a module
+
+Create `import_demo.py`:
+
+```python
+import fsspec
+
+from fsspec_python import install_importer, uninstall_importer
+
+fs = fsspec.filesystem("memory")
+fs.pipe_file("/import-demo/hello_remote.py", b'def greet():\n    return "Hello from memory"\n')
+print(fs.ls("/import-demo", detail=False))
+```
+
+Run `python import_demo.py`. The listing contains `/import-demo/hello_remote.py`.
+
+## Import and call the function
+
+Append:
+
+```python
+importer = install_importer(fs, fo="/import-demo")
+try:
+    import hello_remote
+
+    print(hello_remote.greet())
+finally:
+    uninstall_importer(importer)
+```
+
+Run the script again. Its final line is:
+
+```text
+Hello from memory
+```
+
+## Check cleanup
+
+Append:
+
+```python
+import sys
+
+print("hello_remote" in sys.modules)
+```
+
+Run the script. The final line is `False`: removing the importer also removed its
+tracked module from Python's module cache.
+
+See [How to install and remove importers](how-to-import.md) to use a filesystem URL.

pyproject.toml

@@ -145,3 +145,15 @@ section-order = [
     "F401",
     "F403",
 ]
+
+[tool.yardang]
+title = "fsspec-python"
+root = "README.md"
+pages = [
+    "docs/tutorial.md",
+    "docs/how-to-import.md",
+    "docs/how-to-open.md",
+    "docs/reference.md",
+    "docs/explanation.md",
+]
+use-autoapi = false