Skip to content

Update documentation - #132

Merged
yhong123 merged 6 commits into
mainfrom
yhong123/131-update_documentation
Sep 4, 2026
Merged

Update documentation#132
yhong123 merged 6 commits into
mainfrom
yhong123/131-update_documentation

Conversation

@yhong123

@yhong123 yhong123 commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Resolved #131

Summary
This PR updates and reorganises the Datafaker documentation to improve clarity, navigation, and onboarding for users. In particular, it adds a new tutorial covering the workflow required by Paul and Shipra for generating synthetic Camino data from Parquet files.

Changes

  • Added "Tutorial: Generate Synthetic Data from CSV and Parquet" (tutorial_parquet.rst), which serves as the primary guide for Paul and Shipra's use case.
    • The existing DuckDB documentation (duckdb.rst) has been retained for the time being and can be removed later if Tim agrees that tutorial_parquet.rst provides sufficient coverage.
  • Restructured the documentation into two main sections: "Getting Started" and "Advanced".
  • Updated index.rst to:
    • Refresh the introduction.
    • Remove outdated notes.
    • Revise the table of contents to reflect the new documentation structure.
  • Renamed "Quick Start" to "Command-line Interface (CLI) Guide" to better represent the scope and content of quickstart.rst.
    • This change can be reverted if the original title is preferred.
  • Renamed "Introductory Tutorial" to "Tutorial: Generate Synthetic Data from PostgreSQL" to improve clarity and maintain consistency with the naming convention used for other tutorials.
  • Corrected a minor formatting issue in installation.rst.
  • Removed "Example: Loan Data" (loan_data.rst) from index.rst because the example describes Turing's SqlSynthGen rather than Datafaker.
    • The file itself remains in the repository and can be removed later if Tim agrees.
  • Added requirements.txt file for building the Sphinx documentation.
    • The file specifies the versions of Sphinx and the Mermaid extension required to build the documentation.
    • This simplifies the process of creating a virtual environment and installing the necessary dependencies before running poetry run make html in the docs directory.
  • Updated /datafaker/README.md to include a link (https://safehr-data.github.io/datafaker/index.html) to the documentation GitHub Pages site.

Future Work
The "Advanced" section would benefit from additional review and reorganisation. Further documentation improvements can be addressed in a subsequent documentation-focused PR.

ucl_arc and others added 3 commits August 20, 2026 17:06
Pin Sphinx and mermaid extension used to build the documentation.\n\nThis file makes it easy to create a virtualenv and install the exact packages tested when running 'make html' in docs. Optional runtime deps that enable autodoc imports are commented out.\n

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@yhong123 yhong123 changed the title Yhong123/131 update documentation Update documentation Aug 27, 2026
@yhong123
yhong123 requested a review from tim-band August 27, 2026 15:08
@yhong123 yhong123 self-assigned this Aug 27, 2026
@yhong123
yhong123 requested a review from myyong August 27, 2026 15:09

1. Configure source and destination DSNs.
2. Generate an ``orm.yaml`` file.
3. Review and refine the schema definition.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good. It feels like it could be better to be clearer that 2 is automatic and that 3 is updating this. Something like:
2. Generate the default schema definition (orm.yaml file)
and leave 3 as it is perhaps?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/source/tutorial_parquet.rst Outdated

.. code-block:: text

orm.yaml

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

orm.yaml is not updated by these operations.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line has been removed now.


datafaker create-tables

Generate synthetic data:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps add that this generates 100 lines in each table?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread docs/source/tutorial_parquet.rst Outdated
Troubleshooting
^^^^^^^^^^^^^^^

* If you see a command not found error when running ``datafaker``, run the command using Poetry instead: ``poetry run datafaker``

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only for developers. Normal users will probably have installed DataFaker with pipx. I suppose "check it's installed"? I don't know.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tim-band tim-band left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is good. Have a look at the suggestions and make improvements if you want. Thanks for doing this!

@yhong123

yhong123 commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

Thank you Tim! I will make the changes based on your comments above and merge the code after you have merged the cross-table-interval branch.

@yhong123
yhong123 merged commit 9b53ff5 into main Sep 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update documentation with a Parquet-to-CSV Synthetic Data Generation Tutorial

2 participants