Analyzer: python-inspector ignores LICENSE file unless setup.py exists

[bennati]

Python projects are ignored if a setup.py file is not present, see

ort/analyzer/src/main/kotlin/managers/utils/PythonInspector.kt

Line 197 in 218c55f

val setupProject = projects.find { it.path.endsWith("/setup.py") }

Is this an desired behavior?
Before switching to python-inspector, projects defined by a requirements.txt or a setup.cfg were supported.

[sschuberth]

@bennati you're quoting a line of code that's irrelevant in this context. The relevant lines are at

ort/analyzer/src/main/kotlin/managers/utils/PythonInspector.kt

Lines 220 to 239 in 218c55f

	// First try to get identifier components from "setup.py" in any case, even for "requirements.txt" projects.
	val (setupName, setupVersion) = projects.find { it.path.endsWith("/setup.py") }
	?.let { project ->
	listOf(
	project.packageData.single().name.orEmpty(),
	project.packageData.single().version.orEmpty()
	)
	} ?: listOf("", "")
	val (requirementsName, requirementsVersion, requirementsSuffix) = projects.find { !it.path.endsWith("/setup.py") }
	?.let {
	// In case of "requirements*.txt" there is no metadata at all available, so use the parent directory name
	// plus what "*" expands to as the project name and the VCS revision, if any, as the project version.
	val suffix = definitionFile.name.removePrefix("requirements").removeSuffix(".txt")
	val name = definitionFile.parentFile.name + suffix
	val version = VersionControlSystem.getCloneInfo(definitionFile.parentFile).revision
	listOf(name, version, suffix)
	} ?: listOf("", "", "")

requirements.txt-projects are still supported and our according tests pass:

ort/analyzer/src/funTest/kotlin/managers/PipFunTest.kt

Lines 72 to 87 in 218c55f

	"resolve requirements.txt dependencies correctly for example-python-flask" {
	val definitionFile = projectsDir.resolve("external/example-python-flask/requirements.txt")
	val result = createPip().resolveSingleProject(definitionFile)
	// Note: The expected results were generated with Python 3.8 and are incorrect for versions < 3.8.
	val expectedResultsFile = buildString {
	append("external/example-python-flask-expected-output")
	if (Os.isWindows) append("-windows")
	append(".yml")
	}
	val expectedResult = projectsDir.resolve(expectedResultsFile).readText()
	result.toYaml() shouldBe expectedResult
	}

[bennati]

Hi @sschuberth I clarified the context by editing the title.
The line I mention is what causes the LICENSE file to be ignored:

for a requirements.txt-project, setupProject would be null:

ort/analyzer/src/main/kotlin/managers/utils/PythonInspector.kt

Line 197 in 218c55f

val setupProject = projects.find { it.path.endsWith("/setup.py") }

Hence, projectData would also be null:

ort/analyzer/src/main/kotlin/managers/utils/PythonInspector.kt

Line 198 in 218c55f

val projectData = setupProject?.packageData?.singleOrNull()

and so declaredLicense:

ort/analyzer/src/main/kotlin/managers/utils/PythonInspector.kt

Line 207 in 218c55f

declaredLicenses = projectData?.declaredLicense?.getDeclaredLicenses() ?: emptySet(),

[sschuberth]

I agree that we'd have no declared license without a setup.py, and that's expected. But I don't get how that's related to a LICENSE file, which would contain a detected license. So, are you maybe confusing declared vs. detected license sources here, @bennati?

[bennati]

@sschuberth yes I thought the license in LICENSE file would count as declared.
Maybe then this is a scanner bug, or where else are licenses detected?
It would be great if someone could point me to a function where I can start looking.

[sschuberth]

Maybe then this is a scanner bug, or where else are licenses detected?

Could be. To start with, run plain scancode from the CLI on the project directory and look at its raw JSON results whether they contain your expected license finding.

[bennati]

Good idea! Yes, it's detected

{
      "path": "LICENSE",
      "type": "file",
      "licenses": [
        {
          "key": "here-proprietary",
          "score": 100.0,
          "name": "HERE Proprietary License",
          "short_name": "HERE Proprietary",
          "category": "Commercial",
       ```

[sschuberth]

Yes, it's detected

Ok. Next step, debug getLicenseFindings() in ScanCodeResultParser.kt to see if it's picked up.

[bennati]

Ok never mind, the issue is caused by scancode v32.x.x having a different output format, which ORT does not support yet.
We ran a scan using V31 and the issue is not there. We can close this ticket!

[sschuberth]

Ok never mind, the issue is caused by scancode v32.x.x having a different output format, which ORT does not support yet.

Ping @maxhbr who I believe was experiencing something similar with the latest ScanCode pre-release. I'll create a new issue.