Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# pygrader

## 2.1.0

- Add `complexity` check, which fails if any function exceeds a cognitive complexity threshold (via complexipy)
- Ship a `py.typed` marker so library users get type information; mypy no longer ignores missing imports globally
- Migrate from `os.path` to `pathlib` internally (public signatures still take `str`)
- Migrate the test suite from `unittest` to `pytest`

## 2.0.0

- Desktop Pygrader can now grade multiple projects via globs
Expand Down
16 changes: 8 additions & 8 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This document provides context and guidelines for Claude Sonnet 4.5 when working

### Key Information
- **Language**: Python 3.10+
- **Version**: 1.7.1
- **Version**: 2.1.0
- **License**: GPL-3.0
- **Architecture**: Modular check-based system with CLI and web interfaces
- **Current Branch**: `show-check-results` (default: `main`)
Expand Down Expand Up @@ -163,8 +163,8 @@ The `Grader` class (`grader/grader.py`) orchestrates:
- **Line Length**: Follow project conventions

### Testing
- **Framework**: unittest with coverage (not pytest)
- **Test Runner**: `uv run -m unittest` or `just test`
- **Framework**: pytest with coverage (plain `assert`, `pytest.raises`, `caplog`/`log_output` fixture; `unittest.mock` for mocking)
- **Test Runner**: `uv run pytest` or `just test`
- **Unit Tests**: `tests/unit/test_*.py` - test individual components (~20 seconds for the whole suite)
- **Functional Tests**: `tests/functional/test_functional.py` - end-to-end tests
- **Coverage Target**: 85% minimum (enforced by `just coverage`)
Expand All @@ -190,10 +190,10 @@ The `Grader` class (`grader/grader.py`) orchestrates:
- **Run Specific Tests**:
```bash
# Run specific test class
uv run python -m unittest tests.unit.test_pylint_check.TestPylintCheck -v
uv run pytest tests/unit/test_pylint_check.py::TestPylintCheck -v

# Run specific test method
uv run python -m unittest tests.unit.test_pylint_check.TestPylintCheck.test_01_pylint_called -v
uv run pytest tests/unit/test_pylint_check.py::TestPylintCheck::test_01_pylint_called -v
```

### Adding New Checks
Expand Down Expand Up @@ -261,16 +261,16 @@ just unit_tests # Unit tests only
just functional_tests # Functional tests only

# Manual test execution
uv run -m unittest discover -s tests/unit -p "test_*.py" -v
uv run -m unittest discover -s tests/functional -p "test_*.py"
uv run pytest tests/unit -v
uv run pytest tests/functional
```

#### Coverage Analysis
```bash
just coverage # Run tests with coverage (85% minimum)

# Manual coverage
uv run coverage run --source=grader,desktop -m unittest discover -s tests/unit -p "test_*.py"
uv run coverage run --source=grader,desktop -m pytest tests/unit
uv run coverage report -m --fail-under 85 --sort=cover
uv run coverage lcov -o lcov.info # Generate lcov report
```
Expand Down
9 changes: 9 additions & 0 deletions config/complexity.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"checks": [
{
"name": "complexity",
"is_venv_required": false,
"max_complexity": 15
}
]
}
3 changes: 1 addition & 2 deletions desktop/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
"""

import glob
import os
import shutil
import zipfile
from pathlib import Path
Expand Down Expand Up @@ -139,7 +138,7 @@ def run_grader() -> int:
# TODO - Add output to a file
print(reporter.to_string())

if os.path.exists(const.WORK_DIR):
if const.WORK_DIR.exists():
shutil.rmtree(const.WORK_DIR)

return 1 if has_failure else 0
86 changes: 65 additions & 21 deletions docs/source/config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,23 @@ The grader supports configuration files in JSON format.
These configuration files define the checks to be performed on student submissions,
along with their parameters and scoring criteria.

The file passed via ``--config`` can be a local path, an ``http``/``https``/``ftp`` URL,
or a ``cove://`` URI pointing to a JSON item (requires the ``COVE_API_KEY`` environment variable).

Configuration Schema
--------------------

Root Structure
~~~~~~~~~~~~~~

The configuration file has two possible root-level structures:
There are two kinds of JSON files:

1. **Check-based configuration** - The most common format with a ``checks`` array
2. **Project structure configuration** - For defining required file patterns
1. **Grading configuration** - The file passed via ``--config``, with a ``checks`` array
2. **Project structure configuration** - A separate file referenced by the ``structure`` check

Check-Based Configuration
~~~~~~~~~~~~~~~~~~~~~~~~~
Grading Configuration
~~~~~~~~~~~~~~~~~~~~~

The root object must contain a key called ``checks`` with an array of check objects.
Optionally, it can also include an ``environment`` object for global environment variables.
Optionally, it can also include ``environment`` and ``venv`` objects.
Unknown root-level keys are silently ignored, so double-check spelling.

.. code-block:: json

Expand Down Expand Up @@ -59,6 +60,9 @@ Root-Level Properties
- ``name``: Custom name for the virtual environment directory, to avoid conflicts with a
student's own ``.venv``. Default: ``.venv-pygrader``.

Any other key inside ``venv`` makes the run fail. The venv is only created if at least one
check has ``is_venv_required`` set to ``true``.

Example:

.. code-block:: json
Expand All @@ -83,7 +87,8 @@ Check Object Properties
Common Properties
^^^^^^^^^^^^^^^^^

All check types support these common properties:
All check types support these common properties. Any key not listed here or under the
check's own section is rejected and makes the run fail.

``name`` (required)
The name/type of the check to perform. Supported values:
Expand All @@ -95,15 +100,17 @@ All check types support these common properties:
- ``tests`` - Runs unit tests
- ``structure`` - Validates project file structure
- ``spec`` - Grades the project against a markdown specification using an LLM
- ``complexity`` - Fails if any function exceeds a cognitive complexity threshold (via complexipy)

``max_points`` (optional)
``max_points`` (required for all checks except ``structure`` and ``complexity``)
The maximum score that can be awarded for this check.
Can be an integer or float value.
Some checks (like ``structure``) may not require this field.
The ``structure`` and ``complexity`` checks are pass/fail and do not accept this field.

``is_venv_required`` (required)
Boolean indicating if the check should be run within a virtual environment.
Can be a boolean value or a string ``"false"`` or ``"true"``.
Must be a JSON boolean (``true``/``false``). Strings are not parsed - ``"false"``
is a non-empty string and is treated as ``true``.

``environment`` (optional)
Check-specific environment configuration that overrides global environment variables.
Expand Down Expand Up @@ -156,9 +163,11 @@ Pylint Check
""""""""""""

``pylintrc_path`` (optional)
Path to a custom pylint configuration file.
Local path to a custom pylint configuration file.
Supports template variables like ``${{config_dir}}``.

URLs and Cove URIs are not supported. Default: the bundled ``config/2024.pylintrc``.
If the file does not exist, pylint runs with its own defaults.

Example:

.. code-block:: json
Expand All @@ -173,12 +182,14 @@ Pylint Check
Tests Check
"""""""""""

``tests_path`` (optional)
Array of paths or URLs to test files to run.
Can be local file paths or URLs to remote test files.
``tests_path`` (required)
Array of test files to run.
Each entry can be a local path, a URL, or a ``cove://`` URI pointing to a Python item
(requires the ``COVE_API_KEY`` environment variable).

``default_test_score`` (optional)
Default score assigned to each test if not specified in ``test_score_mapping``.
Default: ``0``.

``test_score_mapping`` (optional)
Object mapping test class or test function names to their point values.
Expand Down Expand Up @@ -208,10 +219,13 @@ Structure Check
"""""""""""""""

``is_fatal`` (optional)
Boolean indicating whether a failing structure check should abort the rest of the
grading run. Default: ``false``.
Boolean marking the check as fatal. Default: ``false``.

.. note::
The flag is stored on the check but not acted upon yet - a failing structure check
does not currently abort the grading run.

``structure_file`` (optional)
``structure_file`` (required)
Path, URL or Cove URI to a JSON file defining the required project structure.
Supports template variables like ``${{config_dir}}``.
Can be a local path, a URL to a remote structure definition, or a ``cove://`` URI
Expand All @@ -237,6 +251,33 @@ Example with a Cove URI:
"structure_file": "cove://fmi-python/project-structure"
}

Complexity Check
""""""""""""""""

Runs complexipy on the project's source files (tests are excluded). The check passes when no
function is more complex than the threshold, and lists the offending functions otherwise.

``max_complexity`` (optional)
The highest cognitive complexity a single function may have. Default: ``15``.

``is_fatal`` (optional)
Boolean marking the check as fatal. Default: ``false``. Not acted upon yet.

Example:

.. code-block:: json

{
"name": "complexity",
"is_venv_required": false,
"max_complexity": 15
}

Type Hints and Coverage Checks
""""""""""""""""""""""""""""""

The ``type-hints`` and ``coverage`` checks take no options beyond the common properties.

Spec Check
""""""""""

Expand Down Expand Up @@ -350,6 +391,9 @@ Configuration files support template variables for dynamic path resolution:
``${{config_dir}}``
Expands to the directory containing the configuration file.
Useful for referencing other config files relative to the main config.
For remote and Cove configs, this is the temporary directory the config was downloaded to.

Any other ``${{...}}`` placeholder in the file is an error.

Example:

Expand Down
2 changes: 2 additions & 0 deletions grader/checks/checks_factory.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""Factory for creating the checks objects."""

from grader.checks.abstract_check import AbstractCheck
from grader.checks.complexity_check import ComplexityCheck
from grader.checks.coverage_check import CoverageCheck
from grader.checks.pylint_check import PylintCheck
from grader.checks.requirements_check import RequirementsCheck
Expand All @@ -19,6 +20,7 @@
"structure": StructureCheck,
"tests": RunTestsCheck,
"spec": SpecCheck,
"complexity": ComplexityCheck,
}


Expand Down
92 changes: 92 additions & 0 deletions grader/checks/complexity_check.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
"""
Module containing the complexity check.

It uses the complexipy python library directly to measure the cognitive complexity of each function.
"""

import logging

import complexipy

from grader.checks.abstract_check import NonScoredCheck
from grader.exceptions import CheckError
from grader.models.check_result import NonScoredCheckResult
from grader.utils import files

logger = logging.getLogger("grader")

DEFAULT_MAX_COMPLEXITY = 15


class ComplexityCheck(NonScoredCheck):
"""The Complexity check class."""

def __init__(
self,
name: str,
project_root: str,
max_complexity: int = DEFAULT_MAX_COMPLEXITY,
is_fatal: bool = False,
is_venv_required: bool = False,
env_vars: dict[str, str] | None = None,
assets: list[str] | None = None,
):
"""
Initialize the complexity check.

:param name: The name of the check.
:param project_root: The root directory of the project.
:param max_complexity: The highest cognitive complexity a single function may have.
:param is_fatal: Whether the check is fatal.
:param is_venv_required: Whether a virtual environment is required.
:param env_vars: Optional environment variables for the check.
:param assets: Optional list of resource sources (paths, URLs or Cove URIs) for the check.
"""
super().__init__(name, project_root, is_fatal, is_venv_required, env_vars, assets)
self.__max_complexity = max_complexity

def run(self) -> NonScoredCheckResult:
"""
Run complexipy on all source files of the project.

The check passes when no function is more complex than the configured maximum.

Files that cannot be parsed fail the check but do not stop the analysis of the other files.

:raises CheckError: If the source files cannot be found.
:returns: The result of the check, listing the offending functions in the info field.
:rtype: NonScoredCheckResult
"""
self._pre_run()

try:
source_files = files.find_all_source_files(self._project_root)
except OSError as error:
logger.error("Error while finding python files: %s", error)
raise CheckError("Error while finding python files") from error

offenders: list[str] = []
unparsable: list[str] = []
for source_file in source_files:
try:
file_complexity = complexipy.file_complexity(source_file)
except ValueError as error: # complexipy raises ValueError for unreadable or invalid files
logger.warning("Could not analyse %s: %s", source_file, error)
unparsable.append(source_file)
continue

offenders.extend(
f"{source_file}:{function.line_start} {function.name} ({function.complexity})"
for function in file_complexity.functions
if function.complexity > self.__max_complexity
)

if offenders or unparsable:
info = []
if offenders:
info.append(f"Functions above complexity {self.__max_complexity}: " + ", ".join(offenders))
if unparsable:
info.append("Could not analyse: " + ", ".join(unparsable))
return NonScoredCheckResult(self.name, False, ". ".join(info), "")

return NonScoredCheckResult(self.name, True, f"All functions are within complexity {self.__max_complexity}", "")
2 changes: 1 addition & 1 deletion grader/checks/coverage_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ def __init__(
"""
super().__init__(name, max_points, project_root, is_venv_required, env_vars, assets)

self.__coverage_full_path = COVERAGE_PATH
self.__coverage_full_path = str(COVERAGE_PATH)

def run(self) -> ScoredCheckResult:
"""
Expand Down
Loading
Loading