May be a bit confused and multiple questions.

[ididitmyway]

Hi there,
I have a installed and running scancode server from docker-compose on a Ubuntu LTS 20.04 Server.

I successfully scaned a open repository in github, but I still have some questions left.

1 How can I scan a repository where authorisation is required.

2 How can I rescan a repository?

3 How can I automatically create a new Scanpipeline with a new repository?

4 How can I trigger / retrigger the Scan from a CI/CD like, Jenkins, TeamCity or Azure Devops?

5 How can I compare the results? / Can I get a report, from only the things that changed by the last scan?

6 How can I generate a readable documentation for my customers? (only containing the nessesary informations)

Kind Regards
Ben

[tdruez]

Hi Ben

I successfully scaned a open repository in github

Well done!

1 How can I scan a repository where authorisation is required.

What kind of authorization? We do not have specific support for auth yet but we are planning to implement in the future. It'll be good to know a bit more about your specific requirements.
For now, you can download the file locally on your machine through your usual auth system and upload that file on project creation.

2 How can I rescan a repository?

Do you mean rescan the same exact code or rather scan another version of that code?
You can create a new Project if you want to keep the results from the previous scan.

3 How can I automatically create a new Scanpipeline with a new repository?

You can use the REST API to automate your project/pipeline managements, see https://scancodeio.readthedocs.io/en/latest/scanpipe-api.html (we definitely need to improve that part of the documentation)
Everything available through the web UI is also available in the API.

4 How can I trigger / retrigger the Scan from a CI/CD like, Jenkins, TeamCity or Azure Devops?

Same as above, the API can be used for such automation.

5 How can I compare the results? / Can I get a report, from only the things that changed by the last scan?

You can only download full reports (json, xlsx) at the moment. Could you enter a new issue in this repository explaining a bit more in details your needs, we can discuss implementation there. See also our tool DeltaCode https://github.com/nexB/deltacode that we use to compare scans.

6 How can I generate a readable documentation for my customers? (only containing the nessesary informations)

Could you provide an example of the content you'd like to generate?

[ididitmyway]

What kind of authorization? We do not have specific support for auth yet but we are planning to implement in the future. It'll be good to know a bit more about your specific requirements.
For now, you can download the file locally on your machine through your usual auth system and upload that file on project creation.

We have several repositories, and we want automatically scan them and check automatically if there was added a new licence in a period of two weeks.
We want also scan new repositories automatically.
We use azure devops (b.t.w. it is free for open source, you can use it as ci-cd

Here are some Open Source Project´s that use AzureDevops as Pipeline for ci-cd

	Atom 			https://dev.azure.com/github/Atom/_build?definitionId=1
	CPython 		https://dev.azure.com/python/cpython/_build?definitionId=9
	Pipenv 			https://dev.azure.com/pypa/pipenv/_build?definitionId=12
	Tox 			https://dev.azure.com/toxdev/tox/_build?definitionId=9
	Visual Studio Code 	https://dev.azure.com/vscode/VSCode/_build?definitionId=1
	TypeScript 		https://dev.azure.com/typescript/TypeScript/_build?definitionId=14

5 How can I compare the results? / Can I get a report, from only the things that changed by the last scan?

You can only download full reports (json, xlsx) at the moment. Could you enter a new issue in this repository explaining a bit more in details your needs, we can discuss implementation there. See also our tool DeltaCode https://github.com/nexB/deltacode that we use to compare scans.

thanks, I will take a look to this tool

anything else

thanks for your help and your questions, we decided to take the docker container instead of the scancode.io server.
We will scan each Repository and will check in the the scan results in another repository to compare it.

I created some shell script´s to do so, here you are ->

Clone Repos

#this script request a username and password and clone all repos in the repos.tx
#!/bin/bash
# Run Command
echo  "start cloning"
server="<your git host>";
project="<your project>";

git login server

while read p;
do
  echo "git clone ${server}/${project}/_git/${p} user -> ${username}"
  mkdir "${p}"
  git clone "${server}/${project}/_git/${p}" "${p}" 
done <repos.txt

Scan Repos

#this script starts a docker container to scan  all repos in the repos.tx
#!/bin/bash
while read p;
do
  echo "start scan ${p}"
  sudo docker run -v $PWD/$p/:/mytemp -l repo1 scancode-toolkit:21.3.31 \
  -clpeui -n 1 --json-pp /mytemp/myresult$p.json /mytemp \
  --license-text --csv /mytemp/myresult$p.csv \
  --html /mytemp/myresult$p.html \
done <repos.txt

[ddmesh]

Hi tdruez,

you mentioned https://scancodeio.readthedocs.io/en/latest/scanpipe-api.html. I really have problems to get anything working.
After starting scancode.io with docker-compose I can access http://localhost/api/projects. But I always get
"Authentication credentials were not provided."

I then found the url http://localhost/admin/. After more try and error I figured out that I can create a superuser account with
./manage.py createsuperuser
From there I then could login into the admin page, create users and tokens. But when I specify users on curl or http-tool, I still get the same error.

Unfortunately I could not find any documentation about the usage of REST API or the defined URLs. When searching with google it seems that no-one is using the REST API of the scancode.io at all.

I then searched the source code and found some url-files defining urlpatterns.

./scancodeio/urls.py
api_router = DefaultRouter()
api_router.register(r"scans", ScanViewSet)
api_router.register(r"projects", ProjectViewSet)
api_router.register(r"runs", RunViewSet)

urlpatterns = [
    path("admin/", admin.site.urls),
    path("api/", include(api_router.urls)),
    path("license/", include(licenses.urls)),
    path("", include("scanpipe.urls")),
    path("", RedirectView.as_view(url="project/")),
]

By trying some combinations I got following:

• http://localhost/license/ -> working
  
• http://localhost/admin/ -> working
  
• http://localhost/api/projects/ -> auth error
  
• http://localhost/api/scans/ -> auth error
  
• http://localhost/api/runs/ -> not found (note, that the normal interface to acces scancode via browser is working, have a lot of test projects create via docker command line or via www-interface)
  

I then found another urlpattern file: ./scanpipe/urls.py

urlpatterns = [
    path(
        "project//resources//raw/",
        views.CodebaseResourceRawView.as_view(),
        name="resource_raw",
    ),
    path(
        "project//resources//",
        views.CodebaseResourceDetailsView.as_view(),
        name="resource_detail",
    ),
 ....

I tried urls like http://localhost/api/project/... but any request tells me that url does not exisit.
I also could not figure out with what data I should replace the place holder.

I then looked ad https://www.django-rest-framework.org/tutorial/4-authentication-and-permissions/#authenticating-with-the-api
to get any hint how to use authentication, pass token or user credentials. But scancode.io always says "Authentication credentials were not provided."

Questions:

• how are credentials/tokens passed to access api (I have added all permissions to all created users in http://localhost/admin/)
• what are the complete URL format for all supported commands of REST API
• Are there specific http headers, that must be provided?
• What commands should use GET/POST requests?

Can someone please provide some documentation and examples? Please!
I do not know how to interpret the little information at https://scancodeio.readthedocs.io/en/latest/scanpipe-api.html.

Thanks a lot.
Stephan

[tdruez]

Hi @ddmesh I've updated the default settings to not enable the Authentication system in the API.
You should be able to access http://localhost/api/ without the credentials using the latest code from the main branch.

Create a new project using the REST API

• Using cURL:

curl -X POST  http://localhost/api/projects/ \
    -H "Content-Type: application/json" \
    --data '{"name": "project_name", "input_urls": "https://download.url/package.archive", "pipeline": "scan_package", "execute_now": true}'

• Using Python and the requests library:

import requests

api_url = 'http://localhost/api/projects/'
headers = {
    'Accept': 'application/json; indent=4',
}
data = {
    "name": "project_name1",
    "input_urls": "https://download.url/package.archive",
    "pipeline": "scan_package",
    "execute_now": True,
}
response = requests.post(api_url, headers=headers, data=data)
print(response.json())

[ddmesh]

Hi,

Thanks. I didn't know that I have to pass the parameters as json data.

About Authentication: I got a little further. The created token (http://localhost/admin/) could be used at command line as followed:
http --debug GET http://localhost/api/projects/ 'Authorization:Token 4dc5aed25414bedbce603dea5b5e0ac33d474c25'
or
curl -X GET http://localhost/api/projects/ -H 'Authorization:Token 4dc5aed25414bedbce603dea5b5e0ac33d474c25' | jq

Can you tell me where I can enable the authentication again, because I intend to use it later in an environment where I need authentication.

Can you give me also the commands/format (json data) that I need to use to run each api command separately?
How can I download the output json files for a project or request the current scan status (if still busy)?

How do I upload a project.tgz instead of providing an imput URL?

Thanks a lot
Stephan

[tdruez]

@ddmesh

Can you tell me where I can enable the authentication again, because I intend to use it later in an environment where I need authentication.

Sure, see ebd9fe3
You can set "DEFAULT_PERMISSION_CLASSES": ("rest_framework.permissions.IsAuthenticated",),

Can you give me also the commands/format (json data) that I need to use to run each api command separately?

We need to raise the priority on improving the API docs, but I the mean time, you can check the following:
From the browsable API view at /api/projects/, click on the OPTIONS button and you'll get details about each "actions". You can also browse the project list and project details to get and idea about the available fields.

When creating a project the response will provide a url value in the returned data. You can make a GET request on this URL to get all the information you need about a project, including the status of the pipeline run:

{
    "name":"project_name",
    "url":"/api/projects/b0c92a72-6c01-461d-993a-5360c30d7937/",
    "uuid":"b0c92a72-6c01-461d-993a-5360c30d7937",
    "created_date":"2021-07-21T16:06:29.132795+02:00"
    [...]
}

How can I download the output json files for a project or request the current scan status (if still busy)?

Use the results action: /api/projects/b0c92a72-6c01-461d-993a-5360c30d7937/results/

How do I upload a project.tgz instead of providing an imput URL?

curl -X POST  http://localhost/api/projects/ \
    -F name=project_name \
    -F upload_file=@file.ext \
    -F pipeline=scan_codebase \
    -F execute_now=true

import requests

api_url = 'http://localhost/api/projects/'
headers = {
    'Accept': 'application/json; indent=4',
}
data = {
    "name": "project_na21me1",
    "pipeline": "scan_package",
    "execute_now": True,
}
with open('your_file_location.ext', 'rb') as f:
    response = requests.post(api_url, headers=headers, data=data, files={"upload_file": f})

print(response.json())

[ddmesh]

Hi and thanks.

My next steps were:

- git pull
- docker-compose build
- docker-compuse up

Because you have disabled the authentication, I now see the "Option" button and "Extra Actions" drop-down.
Before this, I couldn't pass the Token via browser, only via command line tools like curl.

via http://localhost/api/projects/b0c92a72-6c01-461d-993a-5360c30d7937/ I get the selected project and can now try other commands/urls by playing with the "Extra Actions" and Buttons.

About DEFAULT_PERMISSION_CLASSES, should I revert this settings or is there a "user-config" file that I have to use
to overwrite this settings, when I need to enable authorization again?

Thanks a lot

[tdruez]

About DEFAULT_PERMISSION_CLASSES, should I revert this settings or is there a "user-config" file that I have to use
to overwrite this settings, when I need to enable authorization again?

With this last change @ 97779b0 you can now easily enable the Authentication by adding the following line in your local .env file:

REST_FRAMEWORK_DEFAULT_PERMISSION_CLASSES=rest_framework.permissions.IsAuthenticated

Thanks for all your feedback, I'll move all this content in the API documentation.

[ddmesh]

super, thanks a lot

[ddmesh]

Hi,

when I created a project via REST API but did not specify input_url nor upload and keep execute_now:false, the project is created.

How can I then add for instance two pipelines via REST API and later how do I start executing?

Can you provide me with those two curl examples?
Thanks a lot.
Stephan

[tdruez]

How can I then add for instance two pipelines via REST API and later how do I start executing?

api_url="http://localhost/api/projects/9cdaa9be-e48c-4265-b94e-5457784df60f/add_pipeline/"
headers="Content-Type: application/json"

curl -X POST "$api_url" -H "$headers" --data '{"pipeline": "scan_package", "execute_now": false}'

curl -X POST "$api_url" -H "$headers" --data '{"pipeline": "a_pipeline", "execute_now": true}'

To start and existing pipeline pre-added to a project, find it's API URL on the "runs": [ entry of the project details, and POST on its start_pipeline action:

api_url="http://localhost/api/runs/6ded4e0e-cf65-40c4-883c-3cd038262327/start_pipeline/"
headers="Content-Type: application/json"

curl -X POST "$api_url" -H "$headers"

[ddmesh]

Thanks, this sounds good, thanks also for the fast immediate support. :-)