[dpdk-dev] Coverity scan

[dpdk-dev] Coverity scan

[Thomas Monjalon]

We have a public Coverity scan triggered by John for the community:
	https://scan.coverity.com/projects/dpdk-data-plane-development-kit
Note there is a tool to help with this task:
	http://thyrsus.com/gitweb/?p=coverity-submit.git;a=shortlog;h=refs/tags/1.13

I see two issues with this scan:
	- it is run manually
	- not all code is scanned currently

Note that we should be able to run one scan per day for free:
	https://scan.coverity.com/faq#frequency

With David, we looked at automating the Coverity scan,
with the help of Travis automation:
	https://scan.coverity.com/travis_ci
Such automation cannot be configured on the existing Coverity project.
I tried to open a new Coverity project connected to our GitHub.
I have a very poor confidence in Coverity/Travis/GitHub integration.
I will explain below why.

1/ The instructions were wrong. In this command, there are two mistakes:
	openssl s_client -connect https://scan.coverity.com:443 |
	sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' |
	sudo tee -a /etc/ssl/certs/ca-
For the record, a proper a simpler command is:
	true | openssl s_client -connect scan.coverity.com:443 |
	openssl x509 |
	sudo tee -a /etc/ssl/certs/ca-certificates.crt

2/ The coverity scan is triggered as a job addon.
The rest of the job must be cancelled with this tricky patch:

-script: ./.ci/${TRAVIS_OS_NAME}-build.sh
+script: if [ "${COVERITY_SCAN_BRANCH}" != 1 ] ; then ./.ci/${TRAVIS_OS_NAME}-build.sh ; fi

3/ We need only to prepare the source code once per day.
But our .travis.yml has many jobs which must be dropped or ignored.

4/ A big encrypted token must be added in the configuration:
	# encrypted COVERITY_SCAN_TOKEN
	- secure: "VgRYG9N5adKkM9/QpPgswn1c+VXS1mFVN0vgdjuC/bDv2x4u...etc..."

5/ The addon is triggered when pushing to a specific branch
(adding config for the record):
	coverity_scan:
		project:
			name: "DPDK/dpdk"
		notification_email: test-report@dpdk.org
		build_command_prepend: "meson build -Dexamples=all"
		build_command: "ninja -C build"
		branch_pattern: coverity_scan

6/ This attempt failed with this log (no more information):
	$ export PROJECT_NAME=DPDK/dpdk
	Coverity Scan analysis selected for branch coverity_scan.
	Coverity Scan API access denied. Check $PROJECT_NAME and $COVERITY_SCAN_TOKEN.


So I am giving up with Travis+Coverity.
The only benefit of Travis is to have a central build configuration.
So when a driver is enabled in Travis, it would be scanned in Coverity.
Note: Coverity does a build step to prepare the sources.


Now the question: how can we better configure the community Coverity scan?
I propose to set it up in our community lab.
Comments? Suggestions?

Re: [dpdk-dev] Coverity scan

[Aaron Conole]

Thomas Monjalon <thomas@monjalon.net> writes:

> We have a public Coverity scan triggered by John for the community:
> 	https://scan.coverity.com/projects/dpdk-data-plane-development-kit
> Note there is a tool to help with this task:
> 	http://thyrsus.com/gitweb/?p=coverity-submit.git;a=shortlog;h=refs/tags/1.13
>
> I see two issues with this scan:
> 	- it is run manually
> 	- not all code is scanned currently
>
> Note that we should be able to run one scan per day for free:
> 	https://scan.coverity.com/faq#frequency
>
> With David, we looked at automating the Coverity scan,
> with the help of Travis automation:
> 	https://scan.coverity.com/travis_ci
> Such automation cannot be configured on the existing Coverity project.

Why not?

> I tried to open a new Coverity project connected to our GitHub.

I don't know that it will work.  Either you'll need a separate GitHub,
or you'll need to use a special branch.

> I have a very poor confidence in Coverity/Travis/GitHub integration.
> I will explain below why.

Hrrm.. lots of projects use it.  And they do just what you prescribe
below (skipping jobs/builds when on the coverity branch).

> 1/ The instructions were wrong. In this command, there are two mistakes:
> 	openssl s_client -connect https://scan.coverity.com:443 |
> 	sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' |
> 	sudo tee -a /etc/ssl/certs/ca-
> For the record, a proper a simpler command is:
> 	true | openssl s_client -connect scan.coverity.com:443 |
> 	openssl x509 |
> 	sudo tee -a /etc/ssl/certs/ca-certificates.crt

Okay, that's fixable.

> 2/ The coverity scan is triggered as a job addon.
> The rest of the job must be cancelled with this tricky patch:
>
> -script: ./.ci/${TRAVIS_OS_NAME}-build.sh
> +script: if [ "${COVERITY_SCAN_BRANCH}" != 1 ] ; then ./.ci/${TRAVIS_OS_NAME}-build.sh ; fi

More than that, because we probably also want:

if ([[ "${TRAVIS_JOB_NUMBER##*.}" == "1" ]] && [[ "${TRAVIS_BRANCH}" == "coverity_scan" ]]); then ./.ci/${TRAVIS_OS_NAME}-build.sh ; fi

That will only do one job (which solves 3/ below)

> 3/ We need only to prepare the source code once per day.
> But our .travis.yml has many jobs which must be dropped or ignored.
>
> 4/ A big encrypted token must be added in the configuration:
> 	# encrypted COVERITY_SCAN_TOKEN
> 	- secure: "VgRYG9N5adKkM9/QpPgswn1c+VXS1mFVN0vgdjuC/bDv2x4u...etc..."

Why it's a problem?

> 5/ The addon is triggered when pushing to a specific branch
> (adding config for the record):
> 	coverity_scan:
> 		project:
> 			name: "DPDK/dpdk"
> 		notification_email: test-report@dpdk.org
> 		build_command_prepend: "meson build -Dexamples=all"
> 		build_command: "ninja -C build"
> 		branch_pattern: coverity_scan
>
> 6/ This attempt failed with this log (no more information):
> 	$ export PROJECT_NAME=DPDK/dpdk
> 	Coverity Scan analysis selected for branch coverity_scan.
> 	Coverity Scan API access denied. Check $PROJECT_NAME and $COVERITY_SCAN_TOKEN.

Probably there is an issue with the token + PROJECT_NAME.

>
> So I am giving up with Travis+Coverity.
> The only benefit of Travis is to have a central build configuration.
> So when a driver is enabled in Travis, it would be scanned in Coverity.
> Note: Coverity does a build step to prepare the sources.

I can try to assist with this if you've not completely abandoned the idea.

>
> Now the question: how can we better configure the community Coverity scan?
> I propose to set it up in our community lab.
> Comments? Suggestions?

Since we do have something working, but it's manual, is there a way to
at least make it happen automatically?  Maybe some cron job?

Re: [dpdk-dev] Coverity scan

[Thomas Monjalon]

11/03/2020 18:34, Aaron Conole:
> Thomas Monjalon <thomas@monjalon.net> writes:
> 
> > We have a public Coverity scan triggered by John for the community:
> > 	https://scan.coverity.com/projects/dpdk-data-plane-development-kit
> > Note there is a tool to help with this task:
> > 	http://thyrsus.com/gitweb/?p=coverity-submit.git;a=shortlog;h=refs/tags/1.13
> >
> > I see two issues with this scan:
> > 	- it is run manually
> > 	- not all code is scanned currently
> >
> > Note that we should be able to run one scan per day for free:
> > 	https://scan.coverity.com/faq#frequency
> >
> > With David, we looked at automating the Coverity scan,
> > with the help of Travis automation:
> > 	https://scan.coverity.com/travis_ci
> > Such automation cannot be configured on the existing Coverity project.
> 
> Why not?

Because Coverity does not allow it.
Travis integration is possible only if the project was created with GitHub credentials.


> > I tried to open a new Coverity project connected to our GitHub.
> 
> I don't know that it will work.  Either you'll need a separate GitHub,
> or you'll need to use a special branch.
> 
> > I have a very poor confidence in Coverity/Travis/GitHub integration.
> > I will explain below why.
> 
> Hrrm.. lots of projects use it.  And they do just what you prescribe
> below (skipping jobs/builds when on the coverity branch).

Which project is using Travis integration of Coverity?
How do they automatically update the specific branch without conflict?


> > 1/ The instructions were wrong. In this command, there are two mistakes:
> > 	openssl s_client -connect https://scan.coverity.com:443 |
> > 	sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' |
> > 	sudo tee -a /etc/ssl/certs/ca-
> > For the record, a proper a simpler command is:
> > 	true | openssl s_client -connect scan.coverity.com:443 |
> > 	openssl x509 |
> > 	sudo tee -a /etc/ssl/certs/ca-certificates.crt
> 
> Okay, that's fixable.
> 
> > 2/ The coverity scan is triggered as a job addon.
> > The rest of the job must be cancelled with this tricky patch:
> >
> > -script: ./.ci/${TRAVIS_OS_NAME}-build.sh
> > +script: if [ "${COVERITY_SCAN_BRANCH}" != 1 ] ; then ./.ci/${TRAVIS_OS_NAME}-build.sh ; fi
> 
> More than that, because we probably also want:
> 
> if ([[ "${TRAVIS_JOB_NUMBER##*.}" == "1" ]] && [[ "${TRAVIS_BRANCH}" == "coverity_scan" ]]); then ./.ci/${TRAVIS_OS_NAME}-build.sh ; fi
> 
> That will only do one job (which solves 3/ below)

OK good

> > 3/ We need only to prepare the source code once per day.
> > But our .travis.yml has many jobs which must be dropped or ignored.
> >
> > 4/ A big encrypted token must be added in the configuration:
> > 	# encrypted COVERITY_SCAN_TOKEN
> > 	- secure: "VgRYG9N5adKkM9/QpPgswn1c+VXS1mFVN0vgdjuC/bDv2x4u...etc..."
> 
> Why it's a problem?

It's not a problem. I explained all steps in this email, that's why it's here.


> > 5/ The addon is triggered when pushing to a specific branch
> > (adding config for the record):
> > 	coverity_scan:
> > 		project:
> > 			name: "DPDK/dpdk"
> > 		notification_email: test-report@dpdk.org
> > 		build_command_prepend: "meson build -Dexamples=all"
> > 		build_command: "ninja -C build"
> > 		branch_pattern: coverity_scan
> >
> > 6/ This attempt failed with this log (no more information):
> > 	$ export PROJECT_NAME=DPDK/dpdk
> > 	Coverity Scan analysis selected for branch coverity_scan.
> > 	Coverity Scan API access denied. Check $PROJECT_NAME and $COVERITY_SCAN_TOKEN.
> 
> Probably there is an issue with the token + PROJECT_NAME.

Probably. How can I debug it?


> > So I am giving up with Travis+Coverity.
> > The only benefit of Travis is to have a central build configuration.
> > So when a driver is enabled in Travis, it would be scanned in Coverity.
> > Note: Coverity does a build step to prepare the sources.
> 
> I can try to assist with this if you've not completely abandoned the idea.

OK, feel free to ping me for troubleshooting.


> > Now the question: how can we better configure the community Coverity scan?
> > I propose to set it up in our community lab.
> > Comments? Suggestions?
> 
> Since we do have something working, but it's manual, is there a way to
> at least make it happen automatically?  Maybe some cron job?

Yes a cron job, but where? I proposed a server of the community lab.

Re: [dpdk-dev] Coverity scan

[Aaron Conole]

Thomas Monjalon <thomas@monjalon.net> writes:

> 11/03/2020 18:34, Aaron Conole:
>> Thomas Monjalon <thomas@monjalon.net> writes:
>> 
>> > We have a public Coverity scan triggered by John for the community:
>> > 	https://scan.coverity.com/projects/dpdk-data-plane-development-kit
>> > Note there is a tool to help with this task:
>> > 	http://thyrsus.com/gitweb/?p=coverity-submit.git;a=shortlog;h=refs/tags/1.13
>> >
>> > I see two issues with this scan:
>> > 	- it is run manually
>> > 	- not all code is scanned currently
>> >
>> > Note that we should be able to run one scan per day for free:
>> > 	https://scan.coverity.com/faq#frequency
>> >
>> > With David, we looked at automating the Coverity scan,
>> > with the help of Travis automation:
>> > 	https://scan.coverity.com/travis_ci
>> > Such automation cannot be configured on the existing Coverity project.
>> 
>> Why not?
>
> Because Coverity does not allow it.
> Travis integration is possible only if the project was created with GitHub credentials.
>
>
>> > I tried to open a new Coverity project connected to our GitHub.
>> 
>> I don't know that it will work.  Either you'll need a separate GitHub,
>> or you'll need to use a special branch.
>> 
>> > I have a very poor confidence in Coverity/Travis/GitHub integration.
>> > I will explain below why.
>> 
>> Hrrm.. lots of projects use it.  And they do just what you prescribe
>> below (skipping jobs/builds when on the coverity branch).
>
> Which project is using Travis integration of Coverity?
> How do they automatically update the specific branch without conflict?

RabbitMQ-c looked to be doing it:

https://github.com/rabbitmq/rabbitmq-c redirects to:
https://github.com/alanxz/rabbitmq-c/blob/master/.travis.yml

Looks like they haven't done a push in a while, though. :-/

That project also uses coveralls (something I want to integrate the DPDK
project with).

>
>> > 1/ The instructions were wrong. In this command, there are two mistakes:
>> > 	openssl s_client -connect https://scan.coverity.com:443 |
>> > 	sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' |
>> > 	sudo tee -a /etc/ssl/certs/ca-
>> > For the record, a proper a simpler command is:
>> > 	true | openssl s_client -connect scan.coverity.com:443 |
>> > 	openssl x509 |
>> > 	sudo tee -a /etc/ssl/certs/ca-certificates.crt
>> 
>> Okay, that's fixable.
>> 
>> > 2/ The coverity scan is triggered as a job addon.
>> > The rest of the job must be cancelled with this tricky patch:
>> >
>> > -script: ./.ci/${TRAVIS_OS_NAME}-build.sh
>> > +script: if [ "${COVERITY_SCAN_BRANCH}" != 1 ] ; then ./.ci/${TRAVIS_OS_NAME}-build.sh ; fi
>> 
>> More than that, because we probably also want:
>> 
>> if ([[ "${TRAVIS_JOB_NUMBER##*.}" == "1" ]] && [[ "${TRAVIS_BRANCH}"
>> == "coverity_scan" ]]); then ./.ci/${TRAVIS_OS_NAME}-build.sh ; fi
>> 
>> That will only do one job (which solves 3/ below)
>
> OK good
>
>> > 3/ We need only to prepare the source code once per day.
>> > But our .travis.yml has many jobs which must be dropped or ignored.
>> >
>> > 4/ A big encrypted token must be added in the configuration:
>> > 	# encrypted COVERITY_SCAN_TOKEN
>> > 	- secure: "VgRYG9N5adKkM9/QpPgswn1c+VXS1mFVN0vgdjuC/bDv2x4u...etc..."
>> 
>> Why it's a problem?
>
> It's not a problem. I explained all steps in this email, that's why it's here.

Ahh, okay.

>
>> > 5/ The addon is triggered when pushing to a specific branch
>> > (adding config for the record):
>> > 	coverity_scan:
>> > 		project:
>> > 			name: "DPDK/dpdk"
>> > 		notification_email: test-report@dpdk.org
>> > 		build_command_prepend: "meson build -Dexamples=all"
>> > 		build_command: "ninja -C build"
>> > 		branch_pattern: coverity_scan
>> >
>> > 6/ This attempt failed with this log (no more information):
>> > 	$ export PROJECT_NAME=DPDK/dpdk
>> > 	Coverity Scan analysis selected for branch coverity_scan.
>> > 	Coverity Scan API access denied. Check $PROJECT_NAME and $COVERITY_SCAN_TOKEN.
>> 
>> Probably there is an issue with the token + PROJECT_NAME.
>
> Probably. How can I debug it?

Need to go through the steps at (which you probably already did):

https://docs.travis-ci.com/user/coverity-scan/

And make sure the account that has access to the travis repo also has
access to the coverity scan project.

The nice thing on that page is step 8 will have the blocks needed for
the security token (that needs to go in env: global: - secure=...) and
the project settings.

I will ping you offline, though.

>
>> > So I am giving up with Travis+Coverity.
>> > The only benefit of Travis is to have a central build configuration.
>> > So when a driver is enabled in Travis, it would be scanned in Coverity.
>> > Note: Coverity does a build step to prepare the sources.
>> 
>> I can try to assist with this if you've not completely abandoned the idea.
>
> OK, feel free to ping me for troubleshooting.

Will do.

>
>> > Now the question: how can we better configure the community Coverity scan?
>> > I propose to set it up in our community lab.
>> > Comments? Suggestions?
>> 
>> Since we do have something working, but it's manual, is there a way to
>> at least make it happen automatically?  Maybe some cron job?
>
> Yes a cron job, but where? I proposed a server of the community lab.

If it's just a matter of pushing something (or even a few small steps),
we can add it to the robot's sunday "master" rebuild.  Just need whatever
credentials.