[FIXED] Why does a pytest session executed from bazel leave the classname junit attribute empty?

Issue

I discovered some junit xml that we previously generated from a direct pytest session at the commandline no longer had the classname attribute for the testcase tag being populated when the same test was ran through bazel. Why would classname come back void, as an empty string, in the xml generated under bazel?

Here's a reproducible example to demonstrate with...

Project structure

$ tree junit_explore/
junit_explore/
├── BUILD
└── test_explore.py

The Build file

# BUILD
load("@python3_deps//:requirements.bzl", "requirement")

py_test(
    name = "test_explore",
    srcs = ["test_explore.py"],
    args = [
        "--junit-xml=out.xml",
        #"--junit-prefix=THIS", # <----- uncommented in last session
    ],
    deps = [
        requirement("pytest"),
    ],
)

The Test Module

# test_explore.py
import sys
import pytest

def test_pass():
    assert True

def test_fail():
    assert False


if __name__ == "__main__":
    sys.exit(pytest.main([__file__] + sys.argv[1:]))

Direct Pytest Session

pytest test_explore.py --junit-xml=out_pytest.xml

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
    <testsuite errors="0" failures="1" hostname="HOST_X" name="pytest" skipped="0" tests="2" time="0.029" timestamp="2022-03-17T22:16:13.130304">
        <testcase classname="test_explore" file="test_explore.py" line="3" name="test_pass" time="0.000"></testcase>
        <testcase classname="test_explore" file="test_explore.py" line="6" name="test_fail" time="0.000">
            <failure message="assert False">def test_fail():
&gt;       assert False
E       assert False

test_explore.py:8: AssertionError</failure>
        </testcase>
    </testsuite>
</testsuites>

Note the value for classname, which is classname="test_explore" the name of the test module.

Bazel driven pytest session

bazel test //junit_explore:test_explore

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
    <testsuite errors="0" failures="1" hostname="HOST_X" name="pytest" skipped="0" tests="2" time="0.030" timestamp="2022-03-18T05:15:14.118882">
        <testcase classname="" file="../../../../../../../../../../../../workspace_foo/repo_X/junit_explore/test_explore.py" line="3" name="test_pass" time="0.000"></testcase>
        <testcase classname="" file="../../../../../../../../../../../../workspace_foo/repo_X/junit_explore/test_explore.py" line="6" name="test_fail" time="0.000">
            <failure message="assert False">def test_fail():
&gt;       assert False
E       assert False

/home/USERX/workspace_foo/repo_X/junit_explore/test_explore.py:8: AssertionError</failure>
        </testcase>
    </testsuite>
</testsuites>

Note the value for classname, which is classname="" the empty string.

Bazel driven pytest session with JUnit prefix uncommented in the BUILD file

bazel test //junit_explore:test_explore

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
    <testsuite errors="0" failures="1" hostname="HOST_X" name="pytest" skipped="0" tests="2" time="0.032" timestamp="2022-03-18T05:33:10.835593">
        <testcase classname="THIS." file="../../../../../../../../../../../../workspace_foo/repo_X/junit_explore/test_explore.py" line="3" name="test_pass" time="0.000"></testcase>
        <testcase classname="THIS." file="../../../../../../../../../../../../workspace_foo/repo_X/junit_explore/test_explore.py" line="6" name="test_fail" time="0.000">
            <failure message="assert False">def test_fail():
&gt;       assert False
E       assert False

/home/USERX/workspace_foo/repo_X/junit_explore/test_explore.py:8: AssertionError</failure>
        </testcase>
    </testsuite>
</testsuites>

Note the value for classname, which is classname="THIS." simply the dot delimited prefix given at the command line.

This is all occurring on

• bazel 4.2.2-1.4
• pytest 5.4.3

Solution

The root cause for this empty classname attribute is to be found in the following line of source from pytest's nodeid implementation in nodes.py

    nodeid = self.fspath.relto(session.config.rootdir)

The answer to this pytest discussion illuminates with some debug prints the value for nodeid after this operation which establishes that the relative path comes back empty. The reason is because the value for the rootdir places it in the bazel runfiles while the value for fspath places it in the repo clone.

self.fspath = "/home/USERX/repoX/junit_explore/test_explore.py"
session.config.rootdir = "/home/USERX/.cache/bazel/_bazel_USERX/59cb4fc3b17888451f2ad5df027bf8f3/execroot/workspace_foo/bazel-out/bin/junit_explore/test_explore.runfiles/workspace_foo"

As one can see there is no common path between the two paths. There have been changes to pytest since 5.4.3 in the nodes.py source so perhaps this behavior goes away in later releases.

---

(First Answer)

I found out that I could edit the junitxml.py file directly in the bazel cache to add some debug prints. I also did likewise for my site-packages local system python version. It's unclear why exactly but the culprit is in how the nodeid gets generated which from the get go is empty so there's nothing in junitxml.py that could account for this discrepancy, something has already deviated before this logic gets executed.

# junitxml.py
    def record_testreport(self, testreport):
        assert not self.testcase
        print("\n===================================================\nDEBUG START")
        print("testreport.nodeid: %s" % testreport.nodeid)
        names = mangle_test_address(testreport.nodeid)
        print("names: %s" % names)
        existing_attrs = self.attrs
        print("existing_attrs: %s" % existing_attrs)
        classnames = names[:-1]
        print("classnames: %s" % classnames)
        if self.xml.prefix:
            classnames.insert(0, self.xml.prefix)
        attrs = {
            "classname": ".".join(classnames),
            "name": bin_xml_escape(names[-1]),
            "file": testreport.location[0],
        }
        if testreport.location[1] is not None:
            attrs["line"] = testreport.location[1]
        if hasattr(testreport, "url"):
            attrs["url"] = testreport.url
        self.attrs = attrs
        print("attrs: %s" % attrs)
        self.attrs.update(existing_attrs)  # restore any user-defined attributes
        print("DEBUG END")

gives the following for

Bazel execution

===================================================
DEBUG START
testreport.nodeid: ::test_pass
names: ['', 'test_pass']
existing_attrs: {}
classnames: ['']
attrs: {'classname': '', 'name': <py._xmlgen.raw object at 0x7f8114576630>, 'file': '../../../../../../../../../../../../repoX/junit_explore/test_explore.py', 'line': 4}
DEBUG END

===================================================
DEBUG START
testreport.nodeid: ::test_pass
names: ['', 'test_pass']
existing_attrs: {'classname': '', 'name': <py._xmlgen.raw object at 0x7f8114576630>, 'file': '../../../../../../../../../../../../repoX/junit_explore/test_explore.py', 'line': 4}
classnames: ['']
attrs: {'classname': '', 'name': <py._xmlgen.raw object at 0x7f81145764e0>, 'file': '../../../../../../../../../../../../repoX/junit_explore/test_explore.py', 'line': 4}
DEBUG END
F
===================================================
DEBUG START
testreport.nodeid: ::test_fail
names: ['', 'test_fail']
existing_attrs: {}
classnames: ['']
attrs: {'classname': '', 'name': <py._xmlgen.raw object at 0x7f81145761d0>, 'file': '../../../../../../../../../../../../repoX/junit_explore/test_explore.py', 'line': 7}
DEBUG END

===================================================
DEBUG START
testreport.nodeid: ::test_fail
names: ['', 'test_fail']
existing_attrs: {'classname': '', 'name': <py._xmlgen.raw object at 0x7f81145761d0>, 'file': '../../../../../../../../../../../../repoX/junit_explore/test_explore.py', 'line': 7}
classnames: ['']
attrs: {'classname': '', 'name': <py._xmlgen.raw object at 0x7f8114576198>, 'file': '../../../../../../../../../../../../repoX/junit_explore/test_explore.py', 'line': 7}
DEBUG END

Pytest execution

===================================================
DEBUG START
testreport.nodeid: junit_explore/test_explore.py::test_pass
names: ['junit_explore.test_explore', 'test_pass']
existing_attrs: {}
classnames: ['junit_explore.test_explore']
attrs: {'classname': 'junit_explore.test_explore', 'name': <py._xmlgen.raw object at 0x7f9fd3d28630>, 'file': 'junit_explore/test_explore.py', 'line': 4}
DEBUG END

===================================================
DEBUG START
testreport.nodeid: junit_explore/test_explore.py::test_pass
names: ['junit_explore.test_explore', 'test_pass']
existing_attrs: {'classname': 'junit_explore.test_explore', 'name': <py._xmlgen.raw object at 0x7f9fd3d28630>, 'file': 'junit_explore/test_explore.py', 'line': 4}
classnames: ['junit_explore.test_explore']
attrs: {'classname': 'junit_explore.test_explore', 'name': <py._xmlgen.raw object at 0x7f9fd3d59048>, 'file': 'junit_explore/test_explore.py', 'line': 4}
DEBUG END
F
===================================================
DEBUG START
testreport.nodeid: junit_explore/test_explore.py::test_fail
names: ['junit_explore.test_explore', 'test_fail']
existing_attrs: {}
classnames: ['junit_explore.test_explore']
attrs: {'classname': 'junit_explore.test_explore', 'name': <py._xmlgen.raw object at 0x7f9fd3d51438>, 'file': 'junit_explore/test_explore.py', 'line': 7}
DEBUG END

===================================================
DEBUG START
testreport.nodeid: junit_explore/test_explore.py::test_fail
names: ['junit_explore.test_explore', 'test_fail']
existing_attrs: {'classname': 'junit_explore.test_explore', 'name': <py._xmlgen.raw object at 0x7f9fd3d51438>, 'file': 'junit_explore/test_explore.py', 'line': 7}
classnames: ['junit_explore.test_explore']
attrs: {'classname': 'junit_explore.test_explore', 'name': <py._xmlgen.raw object at 0x7f9fd3d59278>, 'file': 'junit_explore/test_explore.py', 'line': 7}
DEBUG END

This nodeid traces all the way up to item.nodeid

# runner.py
def call_and_report(
    item, when: "Literal['setup', 'call', 'teardown']", log=True, **kwds
):
    call = call_runtest_hook(item, when, **kwds)
    hook = item.ihook
    report = hook.pytest_runtest_makereport(item=item, call=call)

...

def pytest_runtest_makereport(item, call):
    return TestReport.from_item_and_call(item, call) # <--- this item's nodeid

I suspect that bazel's prolific reliance on symlinks throughout the bazel cache may be interfering with these filesystem/path operations that pytest somehow relies on.

Answered By - jxramos
Answer Checked By - Katrina (JavaFixing Volunteer)