Open
Conversation
When a test uses pytest.raises(SomeException) and the student's solution fails to raise that exception, pytest raises _pytest.outcomes.Failed. The ResultCollector hook only recognized AssertionError and pytest.fail.Exception, so the failure was routed to TestOutcome.TEST_ERROR and rendered as a misleading Syntax Error banner instead of a clean Failed row with the 'DID NOT RAISE …' message. Add _pytest.outcomes.Failed to the isinstance check so pytest.raises failures consistently surface as TestOutcome.FAIL across pytest versions. Import Failed explicitly to make the routing self-documenting.
TestCaseResult.to_html() unconditionally rendered every TEST_ERROR outcome as '🚨 Syntax Error', even when the underlying exception was a plain AttributeError or TypeError raised from the student's solution code (e.g. 'NoneType' object has no attribute 'shape' when a stub is left as 'return'). The banner contradicted the actual exception type shown in the error block below. Branch the label on the exception class: real SyntaxError (including its IndentationError subclass) still renders as '🚨 Syntax Error' on the COMPILE_ERROR path; everything else renders as '⚠️ Runtime Error (ExcName)', reflecting what the student actually hit.
run_pytest_for_function downgraded the entire result to IPytestOutcome.PYTEST_ERROR whenever any individual test had a TEST_ERROR outcome, discarding every TestCaseResult including parametrizations that had actually passed. The student saw a single generic error panel instead of per-case feedback. The IPytestOutcome.FINISHED display path already iterates test results and renders a mix of PASS/FAIL/TEST_ERROR correctly per-row, so the downgrade is redundant. Drop it. PYTEST_ERROR remains in use for pytest.ExitCode.INTERNAL_ERROR and for cell compilation failures wrapped by run_cell(). Drop the now-unused TestOutcome import from testsuite.py.
edoardob90
changed the title
%%ipytest error-display mislabeling (partial fix for #345)%%ipytest error-display mislabeling
edoardob90
changed the title
%%ipytest error-display mislabelingipytest error-display mislabeling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Partially addresses #345 – three small display-layer fixes in
tutorial/tests/testsuite/that stop students from seeing "🚨 Syntax Error" for problems that are not syntax errors.#345 requires another refactor: the traceback-walk TODO at
helpers.py:764remains open and is better addressed in the standalone-testsuite rewrite. Full reasoning in this comment on #345.Three changes were made
_pytest.outcomes.FailedtoTestOutcome.FAIL, so failingpytest.raises(X)renders as❌ Failed: DID NOT RAISE X(the originalResultCollectorexception handling: improvement forpytest.raises()failures #345 proposal).TEST_ERRORbanner on the actual exception class: realSyntaxError/IndentationErrorstay🚨 Syntax Error; everything else (AttributeErrorfromNone.shape, etc.) becomes⚠️ Runtime Error (<ExcName>).TESTS_FAILEDruns toPYTEST_ERRORwhen any test errors, so mixed runs render per-row instead of as a single catch-all panel.PYTEST_ERRORstays live for theINTERNAL_ERRORand compile-error paths.Each change is an independent commit, reviewable and revertible.
Test plan
Manual smoke tests in live notebooks (no unit-test exists for the display layer):
02_control_flow.ipynb, setsolution_base_converterbody toreturn 42; run; expect❌ Failed: DID NOT RAISE ValueError, not🚨 Syntax Error.magic_example.ipynb, set the student solution body to barereturn; run; expect⚠️ Runtime Error (<ExcName>). Then break the cell withdef f(: pass; expect🚨 Syntax Errorstill fires (regression guard for theCOMPILE_ERRORpath).✅and⚠️, not a single catch-all panel.Pre-flight (
ruff check,ruff format --check,pre-commit run) clean on the modified files.