process_failed_first_last_failed → NotImplementedError

[Natim]

I often encounter this exception, especially when I previously ran pytest and a syntax error prevented it to run.

File "/usr/local/lib/python3.7/site-packages/random_order/plugin.py", line 60, in pytest_collection_modifyitems
INTERNALERROR>     process_failed_first_last_failed(session, config, items)
INTERNALERROR>   File "/usr/local/lib/python3.7/site-packages/random_order/cache.py", line 28, in process_failed_first_last_failed
INTERNALERROR>     raise NotImplementedError()
INTERNALERROR> NotImplementedError

[jbasko]

1. Do you have a name of the test that caused this?
2. Is that a method in a test class or a plain test function?

I'm asking because the issue is to do with the parsing of the name of the failed test in the cache plugin's lastfailed.

[Natim]

Oh interesting. Let me dig that information for you.

[Natim]

Is it possible that it is the case when pytest is run on tests that don't include the file in question? I run pytest 3 times in a raw and I have the issue if pytest is ran without the erroneous file.

[jbasko]

I just tried removing the test or test module that has last failed and then running with --last-failed and was not able to reproduce the error. If you could edit locally that raise NotImplementedError() line to include more information that would help. It's my bad to raise such an uninformative exception.

[ssbarnea]

Today I found the same bug on a project but it happened only once, very weird.

[smuuf]

I stumbled upon this several times over the last few days, pretty randomly.

pytest-random-order/random_order/cache.py

Lines 22 to 31 in ac1369a

	# Get the names of last failed tests
	last_failed = []
	for key in last_failed_raw.keys():
	parts = key.split('::')
	if len(parts) == 3:
	last_failed.append(tuple(parts))
	elif len(parts) == 2:
	last_failed.append((parts[0], None, parts[1]))
	else:
	raise NotImplementedError()

I added pprint.pprint(last_failed_raw) before the for key in ... loop and when it crashed I got a dict of entries. I don't know how or why, but this was there:

{'.github/scripts/sentry_check_create_issue.py::PYCODESTYLE': True,
 'backend/myapp-common/tests': True,  # Hello!
 'backend/myapp/tests/api/test_api_version.py::test_deleted_endpoint_multiple': True,
 ...
}

... so apparently there can be a key without any :: in it. And it seems to me that just ignoring it, instead of crashing with NotImplementedError, could be a pretty good-enough fix for this annoying issue. 🤷