Questions about JaCoCo vs Cobertura exclusion coverage capabilities

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Questions about JaCoCo vs Cobertura exclusion coverage capabilities

Doug Beattie
I noticed a new JIRA http://jira.codehaus.org/browse/SONAR-3136 with the
summary of "Jacoco excludes are ignored in Sonar" has been filed.
In it Freddy also referred to http://jira.codehaus.org/browse/SONAR-766
which is "Allow definition of exclusion patterns for code coverage".
I have some questions and concerns in this area.
When I updated to 2.12 I tried to switch projects over to use JaCoCo
by default. This switch from using Cobertura to JaCoCo met with a lot
of resistance as most of the projects coverage numbers dropped. Engineers
were not please at all with the numbers being shown. They had worked hard
to put exclusions in place in their pom.xml files so code they brought in
from outside resources would not be shown in coverage results that they
were responsible for and had the ability to make corrections on.
They do not want to have to go through more effort to add exclusions to
Sonar to handle this. They want to control the exclusions at both a parent
as well as sub-project level in appropriate pom files so when the refactor
their code the exclusions will easily go with the code and not have to
be managed directly via Sonar.
I personally would have a problem as an administrator trying to keep up
with managing 198, (the current number of projects we now are analyzing),
from the prospective of keeping exclusions up to date.
I voted for SONAR-3136 thinking if this person has a patch available to
allow for exclusions to be dealt with under JaCoCo the same way we were
able to do this with Cobertura that this would be a good thing.
Freddy appears to disagree with this approach.
Freddy could you explain more why you and the project would be against
this approach in general so I can understand better?
Others, what are your feelings?
I just don't want to cause our engineers or myself to have to jump through
more hoops to get the same exclusion capability under JaCoCo we had with
Cobertura.
Thanks for your discussion on this topic.
Doug
--

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Questions about JaCoCo vs Cobertura exclusion coverage capabilities

Michael Kimber
I'm considering moving to JaCoCo as Sonar seems to favour it and it seems to better support multi module project and split unit/integration testing than coberture, but to be honest if it cant handle exclusions then its a bit of a non starter.

Mike

On 4 January 2012 21:11, Doug Beattie <[hidden email]> wrote:
I noticed a new JIRA http://jira.codehaus.org/browse/SONAR-3136 with the
summary of "Jacoco excludes are ignored in Sonar" has been filed.
In it Freddy also referred to http://jira.codehaus.org/browse/SONAR-766
which is "Allow definition of exclusion patterns for code coverage".
I have some questions and concerns in this area.
When I updated to 2.12 I tried to switch projects over to use JaCoCo
by default. This switch from using Cobertura to JaCoCo met with a lot
of resistance as most of the projects coverage numbers dropped. Engineers
were not please at all with the numbers being shown. They had worked hard
to put exclusions in place in their pom.xml files so code they brought in
from outside resources would not be shown in coverage results that they
were responsible for and had the ability to make corrections on.
They do not want to have to go through more effort to add exclusions to
Sonar to handle this. They want to control the exclusions at both a parent
as well as sub-project level in appropriate pom files so when the refactor
their code the exclusions will easily go with the code and not have to
be managed directly via Sonar.
I personally would have a problem as an administrator trying to keep up
with managing 198, (the current number of projects we now are analyzing),
from the prospective of keeping exclusions up to date.
I voted for SONAR-3136 thinking if this person has a patch available to
allow for exclusions to be dealt with under JaCoCo the same way we were
able to do this with Cobertura that this would be a good thing.
Freddy appears to disagree with this approach.
Freddy could you explain more why you and the project would be against
this approach in general so I can understand better?
Others, what are your feelings?
I just don't want to cause our engineers or myself to have to jump through
more hoops to get the same exclusion capability under JaCoCo we had with
Cobertura.
Thanks for your discussion on this topic.
Doug
--

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: Questions about JaCoCo vs Cobertura exclusion coverage capabilities

Freddy Mallet
In reply to this post by Doug Beattie
Hi Doug,

See below my comments :

I noticed a new JIRA http://jira.codehaus.org/browse/SONAR-3136 with the
summary of "Jacoco excludes are ignored in Sonar" has been filed.
In it Freddy also referred to http://jira.codehaus.org/browse/SONAR-766
which is "Allow definition of exclusion patterns for code coverage".
I have some questions and concerns in this area.
When I updated to 2.12 I tried to switch projects over to use JaCoCo
by default. This switch from using Cobertura to JaCoCo met with a lot
of resistance as most of the projects coverage numbers dropped. Engineers
were not please at all with the numbers being shown. They had worked hard
to put exclusions in place in their pom.xml files so code they brought in
from outside resources would not be shown in coverage results that they
were responsible for and had the ability to make corrections on.
They do not want to have to go through more effort to add exclusions to
Sonar to handle this. They want to control the exclusions at both a parent
as well as sub-project level in appropriate pom files so when the refactor
their code the exclusions will easily go with the code and not have to
be managed directly via Sonar.
I personally would have a problem as an administrator trying to keep up
with managing 198, (the current number of projects we now are analyzing),
from the prospective of keeping exclusions up to date.

And I can fully understand that this can be a blocker issue for you Doug to migrate from Cobertura to Jacoco. That's why we haven't stopped supporting Cobertura. We know that the integration of Jacoco is still young and must be improved even if the stability of this integration is already really strong.
 
I voted for SONAR-3136 thinking if this person has a patch available to
allow for exclusions to be dealt with under JaCoCo the same way we were
able to do this with Cobertura that this would be a good thing.

Most of the time, integrating a patch is not an issue, the only question is "how maintainable this patch is ?". We've decided to not integrate this patch for one reason :
  1. Jacoco is natively integrated into Sonar without any dependency on the Jacoco Maven plugin. This is something that has also been done for the Checkstyle, PMD, Findbugs, ... plugins to remove as many dependencies as possible on intermediate libraries/tools and on the Maven bootstrapper. The patch provides a quick-win to reuse the Jacoco Maven plugin configuration in order to manage those exclusions. That's not a viable solution for us as it add a dependency on the Maven bootstrapper and another one on the Jacoco Maven plugin. 
Freddy appears to disagree with this approach.
Freddy could you explain more why you and the project would be against
this approach in general so I can understand better?

That's why I would prefer fixing ticket SONAR-766 that would provide a more viable solution by working with a global "sonar.coverage.exclusions" property. 
 
Others, what are your feelings?

Moreover, when such exclusions will be supported I'd like to report in Sonar both code coverage measures : the one without exclusions and the one with exclusions. Indeed, otherwise the code coverage measure can be error prone : '90% of my code is covered by unit tests ... but this applies only to 5% of my whole code which means that only 4% of my whole code is covered by unit tests'. Nevertheless, providing such "advanced" measures should not be a blocker issue to support the new "sonar.coverage.exclusions" property

Does it make sens for you Doug ?

Thanks
Freddy
 
I just don't want to cause our engineers or myself to have to jump through
more hoops to get the same exclusion capability under JaCoCo we had with
Cobertura.
Thanks for your discussion on this topic.
Doug
--

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: Questions about JaCoCo vs Cobertura exclusion coverage capabilities

Doug Beattie
Freddy, thanks for your reply and explanation. I understand the desire to abstract
and stay independent from Maven, Ant, or others.

We have found work-around that seems to do what we need although it means we duplicate
items to be excluded in two sections in our pom.xml files. Doing this of course could
lead to typos and other problems. Maintaining two areas in the configuration is prone
to have problems. That said here is what we put into our pom file to allow exclusions
with JaCoCo to work and report coverage as was done in Cobertura. I have cut and pasted
this information from a section of a Confluence document I maintain for our development
group on the use of Sonar as shown between the dashed lines following.

---------------------------------------------------------------------------------
Set up the <properties> and <jacoco-maven-plugin> sections in either the
project parent pom.xml file or in a sub-project pom.xml file which
tell Sonar to exclude specified code from being analyzed for problems
and code coverage. This can be done by providing the following sections
in the pom file.

At the main level set up the properties section as follows using the two
excluded examples shown.
Note that in the properties section the excludes are comma separated.
Notice that the same exclusions must also be spelled out in the
build/plugins/plugin/jacoco-maven-plugin section.

<properties>
    <!- Be sure to update the jacoco plugin section excludes to match these items too ->
    <sonar.exclusions>*/MavenArtifactInfo,*/XMLClientProperties</sonar.exclusions>
</properties>
...
...
...
<build>
    ...
    <plugins>
      ...
      <plugin>
      <!- Code Coverage ->
      <!- Be sure to update sonar.exclusions in the properties section to match these items too ->
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <configuration>
            <excludes>
              <!- Generated Files ->
              <exclude>*/MavenArtifactInfo</exclude>
              <exclude>*/XMLClientProperties</exclude>
            </excludes>
        </configuration>
      </plugin>
      ...
    </plugins>
    ...
</build>

The examples shown use ** and * notation per what Maven allows. You can
specify name patterns per the Maven documented notations.
You could just as easily specify exact names like MavenArtifactInfo.class
if that is all you want to exclude.
---------------------------------------------------------------------------------

Doug

On Sun, Jan 08, 2012 at 05:07:06PM +0100, Freddy Mallet wrote:

> Hi Doug,
>
> See below my comments :
>
> I noticed a new JIRA http://jira.codehaus.org/browse/SONAR-3136 with the
> > summary of "Jacoco excludes are ignored in Sonar" has been filed.
> > In it Freddy also referred to http://jira.codehaus.org/browse/SONAR-766
> > which is "Allow definition of exclusion patterns for code coverage".
> > I have some questions and concerns in this area.
> > When I updated to 2.12 I tried to switch projects over to use JaCoCo
> > by default. This switch from using Cobertura to JaCoCo met with a lot
> > of resistance as most of the projects coverage numbers dropped. Engineers
> > were not please at all with the numbers being shown. They had worked hard
> > to put exclusions in place in their pom.xml files so code they brought in
> > from outside resources would not be shown in coverage results that they
> > were responsible for and had the ability to make corrections on.
> > They do not want to have to go through more effort to add exclusions to
> > Sonar to handle this. They want to control the exclusions at both a parent
> > as well as sub-project level in appropriate pom files so when the refactor
> > their code the exclusions will easily go with the code and not have to
> > be managed directly via Sonar.
> > I personally would have a problem as an administrator trying to keep up
> > with managing 198, (the current number of projects we now are analyzing),
> > from the prospective of keeping exclusions up to date.
> >
>
> And I can fully understand that this can be a blocker issue for you Doug to
> migrate from Cobertura to Jacoco. That's why we haven't stopped supporting
> Cobertura. We know that the integration of Jacoco is still young and must
> be improved even if the stability of this integration is already really
> strong.
>
>
> > I voted for SONAR-3136 thinking if this person has a patch available to
> > allow for exclusions to be dealt with under JaCoCo the same way we were
> > able to do this with Cobertura that this would be a good thing.
> >
>
> Most of the time, integrating a patch is not an issue, the only question is
> "how maintainable this patch is ?". We've decided to not integrate this
> patch for one reason :
>
>    1. Jacoco is natively integrated into Sonar without any dependency on
>    the Jacoco Maven plugin. This is something that has also been done for the
>    Checkstyle, PMD, Findbugs, ... plugins to remove as many dependencies as
>    possible on intermediate libraries/tools and on the Maven bootstrapper. The
>    patch provides a quick-win to reuse the Jacoco Maven plugin configuration
>    in order to manage those exclusions. That's not a viable solution for us as
>    it add a dependency on the Maven bootstrapper and another one on the Jacoco
>    Maven plugin.
>
> Freddy appears to disagree with this approach.
> > Freddy could you explain more why you and the project would be against
> > this approach in general so I can understand better?
> >
>
> That's why I would prefer fixing ticket SONAR-766 that would provide a more
> viable solution by working with a global "sonar.coverage.exclusions"
> property.
>
>
> > Others, what are your feelings?
> >
>
> Moreover, when such exclusions will be supported I'd like to report in
> Sonar both code coverage measures : the one without exclusions and the one
> with exclusions. Indeed, otherwise the code coverage measure can be error
> prone : '90% of my code is covered by unit tests ... but this applies only
> to 5% of my whole code which means that only 4% of my whole code is covered
> by unit tests'. Nevertheless, providing such "advanced" measures should not
> be a blocker issue to support the new "sonar.coverage.exclusions" property
>
> Does it make sens for you Doug ?
>
> Thanks
> Freddy
>
>
> > I just don't want to cause our engineers or myself to have to jump through
> > more hoops to get the same exclusion capability under JaCoCo we had with
> > Cobertura.
> > Thanks for your discussion on this topic.
> > Doug
> > --
> >
> > ---------------------------------------------------------------------
> > To unsubscribe from this list, please visit:
> >
> >    http://xircles.codehaus.org/manage_email
> >
> >
> >

--

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Questions about JaCoCo vs Cobertura exclusion coverage capabilities

Freddy Mallet
Hi Doug,

Perhaps there is a misunderstanding because of course the Sonar Jacoco plugin perfectly supports the Sonar exclusion mechanism. So in your case, you can remove from your pom.xml file the configuration of the Jacoco Maven plugin. 

What is not supported is the ability to not exclude from the Sonar analysis a source code but to not compute code coverage on this source code. For instance when you want to analyse the source code of some Struts actions but also want to exclude this source code from the code coverage computation as writing some unit tests to test those Struts actions is not valuable.
-----
Sonar for Continuous Inspection



On Mon, Jan 9, 2012 at 11:10 PM, Doug Beattie <[hidden email]> wrote:
Freddy, thanks for your reply and explanation. I understand the desire to abstract
and stay independent from Maven, Ant, or others.

We have found work-around that seems to do what we need although it means we duplicate
items to be excluded in two sections in our pom.xml files. Doing this of course could
lead to typos and other problems. Maintaining two areas in the configuration is prone
to have problems. That said here is what we put into our pom file to allow exclusions
with JaCoCo to work and report coverage as was done in Cobertura. I have cut and pasted
this information from a section of a Confluence document I maintain for our development
group on the use of Sonar as shown between the dashed lines following.

---------------------------------------------------------------------------------
Set up the <properties> and <jacoco-maven-plugin> sections in either the
project parent pom.xml file or in a sub-project pom.xml file which
tell Sonar to exclude specified code from being analyzed for problems
and code coverage. This can be done by providing the following sections
in the pom file.

At the main level set up the properties section as follows using the two
excluded examples shown.
Note that in the properties section the excludes are comma separated.
Notice that the same exclusions must also be spelled out in the
build/plugins/plugin/jacoco-maven-plugin section.

<properties>
   <!- Be sure to update the jacoco plugin section excludes to match these items too ->
   <sonar.exclusions>*/MavenArtifactInfo,*/XMLClientProperties</sonar.exclusions>
</properties>
...
...
...
<build>
   ...
   <plugins>
     ...
     <plugin>
     <!- Code Coverage ->
     <!- Be sure to update sonar.exclusions in the properties section to match these items too ->
       <groupId>org.jacoco</groupId>
       <artifactId>jacoco-maven-plugin</artifactId>
       <configuration>
           <excludes>
             <!- Generated Files ->
             <exclude>*/MavenArtifactInfo</exclude>
             <exclude>*/XMLClientProperties</exclude>
           </excludes>
       </configuration>
     </plugin>
     ...
   </plugins>
   ...
</build>

The examples shown use ** and * notation per what Maven allows. You can
specify name patterns per the Maven documented notations.
You could just as easily specify exact names like MavenArtifactInfo.class
if that is all you want to exclude.
---------------------------------------------------------------------------------

Doug

On Sun, Jan 08, 2012 at 05:07:06PM +0100, Freddy Mallet wrote:
> Hi Doug,
>
> See below my comments :
>
> I noticed a new JIRA http://jira.codehaus.org/browse/SONAR-3136 with the
> > summary of "Jacoco excludes are ignored in Sonar" has been filed.
> > In it Freddy also referred to http://jira.codehaus.org/browse/SONAR-766
> > which is "Allow definition of exclusion patterns for code coverage".
> > I have some questions and concerns in this area.
> > When I updated to 2.12 I tried to switch projects over to use JaCoCo
> > by default. This switch from using Cobertura to JaCoCo met with a lot
> > of resistance as most of the projects coverage numbers dropped. Engineers
> > were not please at all with the numbers being shown. They had worked hard
> > to put exclusions in place in their pom.xml files so code they brought in
> > from outside resources would not be shown in coverage results that they
> > were responsible for and had the ability to make corrections on.
> > They do not want to have to go through more effort to add exclusions to
> > Sonar to handle this. They want to control the exclusions at both a parent
> > as well as sub-project level in appropriate pom files so when the refactor
> > their code the exclusions will easily go with the code and not have to
> > be managed directly via Sonar.
> > I personally would have a problem as an administrator trying to keep up
> > with managing 198, (the current number of projects we now are analyzing),
> > from the prospective of keeping exclusions up to date.
> >
>
> And I can fully understand that this can be a blocker issue for you Doug to
> migrate from Cobertura to Jacoco. That's why we haven't stopped supporting
> Cobertura. We know that the integration of Jacoco is still young and must
> be improved even if the stability of this integration is already really
> strong.
>
>
> > I voted for SONAR-3136 thinking if this person has a patch available to
> > allow for exclusions to be dealt with under JaCoCo the same way we were
> > able to do this with Cobertura that this would be a good thing.
> >
>
> Most of the time, integrating a patch is not an issue, the only question is
> "how maintainable this patch is ?". We've decided to not integrate this
> patch for one reason :
>
>    1. Jacoco is natively integrated into Sonar without any dependency on
>    the Jacoco Maven plugin. This is something that has also been done for the
>    Checkstyle, PMD, Findbugs, ... plugins to remove as many dependencies as
>    possible on intermediate libraries/tools and on the Maven bootstrapper. The
>    patch provides a quick-win to reuse the Jacoco Maven plugin configuration
>    in order to manage those exclusions. That's not a viable solution for us as
>    it add a dependency on the Maven bootstrapper and another one on the Jacoco
>    Maven plugin.
>
> Freddy appears to disagree with this approach.
> > Freddy could you explain more why you and the project would be against
> > this approach in general so I can understand better?
> >
>
> That's why I would prefer fixing ticket SONAR-766 that would provide a more
> viable solution by working with a global "sonar.coverage.exclusions"
> property.
>
>
> > Others, what are your feelings?
> >
>
> Moreover, when such exclusions will be supported I'd like to report in
> Sonar both code coverage measures : the one without exclusions and the one
> with exclusions. Indeed, otherwise the code coverage measure can be error
> prone : '90% of my code is covered by unit tests ... but this applies only
> to 5% of my whole code which means that only 4% of my whole code is covered
> by unit tests'. Nevertheless, providing such "advanced" measures should not
> be a blocker issue to support the new "sonar.coverage.exclusions" property
>
> Does it make sens for you Doug ?
>
> Thanks
> Freddy
>
>
> > I just don't want to cause our engineers or myself to have to jump through
> > more hoops to get the same exclusion capability under JaCoCo we had with
> > Cobertura.
> > Thanks for your discussion on this topic.
> > Doug
> > --
> >
> > ---------------------------------------------------------------------
> > To unsubscribe from this list, please visit:
> >
> >    http://xircles.codehaus.org/manage_email
> >
> >
> >

--

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

   http://xircles.codehaus.org/manage_email