Quantcast

[sonar-dev] NUnit reuseReport support

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[sonar-dev] NUnit reuseReport support

John M. Wright-2
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

mkoertgen
This post has NOT been accepted by the mailing list yet.
This post was updated on .
As Mr. Wright i also would like to offer help in realizing the "reuse NUnit reports"-feature.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

Fabrice Bellingard-4
In reply to this post by John M. Wright-2
Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [sonar-dev] NUnit reuseReport support

John M. Wright-2
That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: [sonar-dev] NUnit reuseReport support

John M. Wright-2
Fabrice and other interested devs:

I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.

If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository: https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.

Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.

Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.

Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :

#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip

#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml

#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True

As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.

Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support

That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

Fabrice Bellingard-4
Hi John,

thanks for the notification! No problem if you work on side projects, this will give us time to get back on the .NET ecosystem. 

Thanks for your contribution and your efforts!


Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Fri, Jan 24, 2014 at 2:34 PM, John M. Wright <[hidden email]> wrote:
Fabrice and other interested devs:

I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.

If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository: https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.

Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.

Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.

Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :

#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip

#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml

#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True

As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.

Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support


That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

Dinesh Bolkensteyn-2
Hi John,

Thank you for your work on the .NET plugin.

I am afraid that we slightly changed our minds, and that the ticket you started to work on is no longer going to get fixed:

Without a doubt, code coverage is a good indicator of code quality, and we need to be able to access and visualise it within SonarQube.

However, the story regarding test results is a bit different: A project is expected to always have 0 test failures, else continuous integration will fail.
Failing and skipped tests can also highly impact code coverage.
We would rather not see such jumps in the timeline within SonarQube, simply because of a bad commit at the bad time (right before the SQ analysis).

If there are failing or skipped unit tests in a project, the top most priority should be to fix them: it does not functionally work as expected.
It does not make much sense to "manage the technical debt" of a functionally broken project.

To sum up:
 1) the SonarQube analysis is expected to be triggered only if all tests are passing, hence the results are always identical: all green.
 2) Moreover unit tests are also expected to always be fast, and if that is no longer the case, continuous integration should be able to spot it.

Also, the support of Gallio will be fully dropped in this sprint, as it is simply not practical to most users of the .NET plugin.

So the goal for the current .NET 2.1 sprint is definitely to provide native support for the most popular code coverage tools, as a replacement to Gallio.

Of course, feel free to challenge this if you think we're mistaken!

Sorry about SONARDOTNT-372.

Kind regards,





On Fri, Jan 24, 2014 at 2:52 PM, Fabrice Bellingard <[hidden email]> wrote:
Hi John,

thanks for the notification! No problem if you work on side projects, this will give us time to get back on the .NET ecosystem. 

Thanks for your contribution and your efforts!


Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Fri, Jan 24, 2014 at 2:34 PM, John M. Wright <[hidden email]> wrote:
Fabrice and other interested devs:

I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.

If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository: https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.

Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.

Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.

Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :

#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip

#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml

#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True

As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.

Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support


That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

John M. Wright-2
As I've mentioned before on this list, I very strongly disagree with the direction your going.

That said, I will likely release the plugin as a standalone for people who do want the features. Is this something that would be allowed into the Forge?

Thanks
~john


On Jan 29, 2014, at 8:21 AM, "Dinesh Bolkensteyn" <[hidden email]> wrote:

Hi John,

Thank you for your work on the .NET plugin.

I am afraid that we slightly changed our minds, and that the ticket you started to work on is no longer going to get fixed:

Without a doubt, code coverage is a good indicator of code quality, and we need to be able to access and visualise it within SonarQube.

However, the story regarding test results is a bit different: A project is expected to always have 0 test failures, else continuous integration will fail.
Failing and skipped tests can also highly impact code coverage.
We would rather not see such jumps in the timeline within SonarQube, simply because of a bad commit at the bad time (right before the SQ analysis).

If there are failing or skipped unit tests in a project, the top most priority should be to fix them: it does not functionally work as expected.
It does not make much sense to "manage the technical debt" of a functionally broken project.

To sum up:
 1) the SonarQube analysis is expected to be triggered only if all tests are passing, hence the results are always identical: all green.
 2) Moreover unit tests are also expected to always be fast, and if that is no longer the case, continuous integration should be able to spot it.

Also, the support of Gallio will be fully dropped in this sprint, as it is simply not practical to most users of the .NET plugin.

So the goal for the current .NET 2.1 sprint is definitely to provide native support for the most popular code coverage tools, as a replacement to Gallio.

Of course, feel free to challenge this if you think we're mistaken!

Sorry about SONARDOTNT-372.

Kind regards,





On Fri, Jan 24, 2014 at 2:52 PM, Fabrice Bellingard <[hidden email]> wrote:
Hi John,

thanks for the notification! No problem if you work on side projects, this will give us time to get back on the .NET ecosystem. 

Thanks for your contribution and your efforts!


Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Fri, Jan 24, 2014 at 2:34 PM, John M. Wright <[hidden email]> wrote:
Fabrice and other interested devs:

I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.

If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository: https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.

Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.

Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.

Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :

#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip

#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml

#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True

As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.

Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support


That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

Jorge Costa

Hi John,

I can give you a hand with this since at my work we also value this feature. And we like to keep Sonar the place that tracks all metrics in our projects.

So I guess the new plugin should be called Test Report Plugin?

On Jan 29, 2014 4:52 PM, "John M. Wright" <[hidden email]> wrote:
As I've mentioned before on this list, I very strongly disagree with the direction your going.

That said, I will likely release the plugin as a standalone for people who do want the features. Is this something that would be allowed into the Forge?

Thanks
~john


On Jan 29, 2014, at 8:21 AM, "Dinesh Bolkensteyn" <[hidden email]> wrote:

Hi John,

Thank you for your work on the .NET plugin.

I am afraid that we slightly changed our minds, and that the ticket you started to work on is no longer going to get fixed:

Without a doubt, code coverage is a good indicator of code quality, and we need to be able to access and visualise it within SonarQube.

However, the story regarding test results is a bit different: A project is expected to always have 0 test failures, else continuous integration will fail.
Failing and skipped tests can also highly impact code coverage.
We would rather not see such jumps in the timeline within SonarQube, simply because of a bad commit at the bad time (right before the SQ analysis).

If there are failing or skipped unit tests in a project, the top most priority should be to fix them: it does not functionally work as expected.
It does not make much sense to "manage the technical debt" of a functionally broken project.

To sum up:
 1) the SonarQube analysis is expected to be triggered only if all tests are passing, hence the results are always identical: all green.
 2) Moreover unit tests are also expected to always be fast, and if that is no longer the case, continuous integration should be able to spot it.

Also, the support of Gallio will be fully dropped in this sprint, as it is simply not practical to most users of the .NET plugin.

So the goal for the current .NET 2.1 sprint is definitely to provide native support for the most popular code coverage tools, as a replacement to Gallio.

Of course, feel free to challenge this if you think we're mistaken!

Sorry about SONARDOTNT-372.

Kind regards,





On Fri, Jan 24, 2014 at 2:52 PM, Fabrice Bellingard <[hidden email]> wrote:
Hi John,

thanks for the notification! No problem if you work on side projects, this will give us time to get back on the .NET ecosystem. 

Thanks for your contribution and your efforts!


Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Fri, Jan 24, 2014 at 2:34 PM, John M. Wright <[hidden email]> wrote:
Fabrice and other interested devs:

I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.

If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository: https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.

Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.

Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.

Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :

#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip

#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml

#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True

As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.

Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support


That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john




Best Regards
Jorge Costa
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

Peter Stevens
In reply to this post by John M. Wright-2
What is the alternative for Gallio?
 
We're starting to use sonarQube in a C# environment, and we have not gotten to the stage where it is fully integrated in our tfs builds. Gallio works nice for us
 
Peter Stevens
 
Op 29 januari 2014 om 15:52 schreef "John M. Wright" <[hidden email]>:

As I've mentioned before on this list, I very strongly disagree with the direction your going.
 
That said, I will likely release the plugin as a standalone for people who do want the features. Is this something that would be allowed into the Forge?

Thanks
~john
 

On Jan 29, 2014, at 8:21 AM, "Dinesh Bolkensteyn" < [hidden email]> wrote:

Hi John,
 
Thank you for your work on the .NET plugin.
 
I am afraid that we slightly changed our minds, and that the ticket you started to work on is no longer going to get fixed:
 
Without a doubt, code coverage is a good indicator of code quality, and we need to be able to access and visualise it within SonarQube.
 
However, the story regarding test results is a bit different: A project is expected to always have 0 test failures, else continuous integration will fail.
Failing and skipped tests can also highly impact code coverage.
We would rather not see such jumps in the timeline within SonarQube, simply because of a bad commit at the bad time (right before the SQ analysis).
 
If there are failing or skipped unit tests in a project, the top most priority should be to fix them: it does not functionally work as expected.
It does not make much sense to "manage the technical debt" of a functionally broken project.
 
To sum up:
 1) the SonarQube analysis is expected to be triggered only if all tests are passing, hence the results are always identical: all green.
 2) Moreover unit tests are also expected to always be fast, and if that is no longer the case, continuous integration should be able to spot it.
 
Also, the support of Gallio will be fully dropped in this sprint, as it is simply not practical to most users of the .NET plugin.
 
So the goal for the current .NET 2.1 sprint is definitely to provide native support for the most popular code coverage tools, as a replacement to Gallio.
 
Of course, feel free to challenge this if you think we're mistaken!
 
Sorry about SONARDOTNT-372.
 
Kind regards,
 
 



On Fri, Jan 24, 2014 at 2:52 PM, Fabrice Bellingard <[hidden email]> wrote:
Hi John,
 
thanks for the notification! No problem if you work on side projects, this will give us time to get back on the .NET ecosystem. 
 
Thanks for your contribution and your efforts!

 
Best regards,
 
 
Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Fri, Jan 24, 2014 at 2:34 PM, John M. Wright <[hidden email]> wrote:
Fabrice and other interested devs:
 
I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.
 
If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository:  https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.
 
Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.
 
Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.
 
Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :
 
#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip
 
#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml
 
#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True
 
As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.
 
Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support


That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,
 
starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.
 
 
So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance " org.sonar.plugins.dotnet.tests.mstest", " org.sonar.plugins.dotnet.tests.mscoverage", " org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?
 
 

 
Best regards,
 
 
Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:
 
Looking at the v2.2 release backlog, you have this tagged for the release:
 
SONARDOTNT-372  Make it possible to reuse NUnit reports
 
I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?
 
Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.
 
Tha nks
~john
 

 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

Jorge Costa

I think you can still use Gallio to produce coverage results. It's just that the plugin will no longer be responsible for running any tests. You just need to produce your results in your TFS build and then use the reuse reports approach. Correct me if I'm wrong.

On Jan 30, 2014 4:05 AM, "Peter Stevens" <[hidden email]> wrote:
What is the alternative for Gallio?
 
We're starting to use sonarQube in a C# environment, and we have not gotten to the stage where it is fully integrated in our tfs builds. Gallio works nice for us
 
Peter Stevens
 
Op 29 januari 2014 om 15:52 schreef "John M. Wright" <[hidden email]>:

As I've mentioned before on this list, I very strongly disagree with the direction your going.
 
That said, I will likely release the plugin as a standalone for people who do want the features. Is this something that would be allowed into the Forge?

Thanks
~john
 

On Jan 29, 2014, at 8:21 AM, "Dinesh Bolkensteyn" < [hidden email]> wrote:

Hi John,
 
Thank you for your work on the .NET plugin.
 
I am afraid that we slightly changed our minds, and that the ticket you started to work on is no longer going to get fixed:
 
Without a doubt, code coverage is a good indicator of code quality, and we need to be able to access and visualise it within SonarQube.
 
However, the story regarding test results is a bit different: A project is expected to always have 0 test failures, else continuous integration will fail.
Failing and skipped tests can also highly impact code coverage.
We would rather not see such jumps in the timeline within SonarQube, simply because of a bad commit at the bad time (right before the SQ analysis).
 
If there are failing or skipped unit tests in a project, the top most priority should be to fix them: it does not functionally work as expected.
It does not make much sense to "manage the technical debt" of a functionally broken project.
 
To sum up:
 1) the SonarQube analysis is expected to be triggered only if all tests are passing, hence the results are always identical: all green.
 2) Moreover unit tests are also expected to always be fast, and if that is no longer the case, continuous integration should be able to spot it.
 
Also, the support of Gallio will be fully dropped in this sprint, as it is simply not practical to most users of the .NET plugin.
 
So the goal for the current .NET 2.1 sprint is definitely to provide native support for the most popular code coverage tools, as a replacement to Gallio.
 
Of course, feel free to challenge this if you think we're mistaken!
 
Sorry about SONARDOTNT-372.
 
Kind regards,
 
 



On Fri, Jan 24, 2014 at 2:52 PM, Fabrice Bellingard <[hidden email]> wrote:
Hi John,
 
thanks for the notification! No problem if you work on side projects, this will give us time to get back on the .NET ecosystem. 
 
Thanks for your contribution and your efforts!

 
Best regards,
 
 
Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Fri, Jan 24, 2014 at 2:34 PM, John M. Wright <[hidden email]> wrote:
Fabrice and other interested devs:
 
I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.
 
If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository:  https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.
 
Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.
 
Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.
 
Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :
 
#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip
 
#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml
 
#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True
 
As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.
 
Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support


That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,
 
starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.
 
 
So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance " org.sonar.plugins.dotnet.tests.mstest", " org.sonar.plugins.dotnet.tests.mscoverage", " org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?
 
 

 
Best regards,
 
 
Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:
 
Looking at the v2.2 release backlog, you have this tagged for the release:
 
SONARDOTNT-372  Make it possible to reuse NUnit reports
 
I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?
 
Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.
 
Tha nks
~john
 

 
Best Regards
Jorge Costa
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

John M. Wright-2
In reply to this post by John M. Wright-2
Dinesh,
As part of my changes, I modified the logic in the existing .net plugins in order to support the AST changes required by my plugin. Would you still consider those changes if I submitted a pull request?  If so, would you respond to this snippet regarding changing the resource bridge index from my original email so I can submit the appropriate implementation:

  • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
    • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.

Thanks
~john


On Jan 29, 2014, at 8:52 AM, "John M. Wright" <[hidden email]> wrote:

As I've mentioned before on this list, I very strongly disagree with the direction your going.

That said, I will likely release the plugin as a standalone for people who do want the features. Is this something that would be allowed into the Forge?

Thanks
~john


On Jan 29, 2014, at 8:21 AM, "Dinesh Bolkensteyn" <[hidden email]> wrote:

Hi John,

Thank you for your work on the .NET plugin.

I am afraid that we slightly changed our minds, and that the ticket you started to work on is no longer going to get fixed:

Without a doubt, code coverage is a good indicator of code quality, and we need to be able to access and visualise it within SonarQube.

However, the story regarding test results is a bit different: A project is expected to always have 0 test failures, else continuous integration will fail.
Failing and skipped tests can also highly impact code coverage.
We would rather not see such jumps in the timeline within SonarQube, simply because of a bad commit at the bad time (right before the SQ analysis).

If there are failing or skipped unit tests in a project, the top most priority should be to fix them: it does not functionally work as expected.
It does not make much sense to "manage the technical debt" of a functionally broken project.

To sum up:
 1) the SonarQube analysis is expected to be triggered only if all tests are passing, hence the results are always identical: all green.
 2) Moreover unit tests are also expected to always be fast, and if that is no longer the case, continuous integration should be able to spot it.

Also, the support of Gallio will be fully dropped in this sprint, as it is simply not practical to most users of the .NET plugin.

So the goal for the current .NET 2.1 sprint is definitely to provide native support for the most popular code coverage tools, as a replacement to Gallio.

Of course, feel free to challenge this if you think we're mistaken!

Sorry about SONARDOTNT-372.

Kind regards,





On Fri, Jan 24, 2014 at 2:52 PM, Fabrice Bellingard <[hidden email]> wrote:
Hi John,

thanks for the notification! No problem if you work on side projects, this will give us time to get back on the .NET ecosystem. 

Thanks for your contribution and your efforts!


Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Fri, Jan 24, 2014 at 2:34 PM, John M. Wright <[hidden email]> wrote:
Fabrice and other interested devs:

I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.

If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository: https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.

Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.

Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.

Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :

#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip

#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml

#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True

As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.

Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support


That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

Dinesh Bolkensteyn-2
John,

I need some time to understand what exactly you are referring to, as I am currently in the process of taking over the development of this .NET plugin.

This is in my TODO, and I'll get back to you as soon as I can, sometime next week.



On Thu, Jan 30, 2014 at 1:23 PM, John M. Wright <[hidden email]> wrote:
Dinesh,
As part of my changes, I modified the logic in the existing .net plugins in order to support the AST changes required by my plugin. Would you still consider those changes if I submitted a pull request?  If so, would you respond to this snippet regarding changing the resource bridge index from my original email so I can submit the appropriate implementation:

  • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
    • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.

Thanks
~john


On Jan 29, 2014, at 8:52 AM, "John M. Wright" <[hidden email]> wrote:

As I've mentioned before on this list, I very strongly disagree with the direction your going.

That said, I will likely release the plugin as a standalone for people who do want the features. Is this something that would be allowed into the Forge?

Thanks
~john


On Jan 29, 2014, at 8:21 AM, "Dinesh Bolkensteyn" <[hidden email]> wrote:

Hi John,

Thank you for your work on the .NET plugin.

I am afraid that we slightly changed our minds, and that the ticket you started to work on is no longer going to get fixed:

Without a doubt, code coverage is a good indicator of code quality, and we need to be able to access and visualise it within SonarQube.

However, the story regarding test results is a bit different: A project is expected to always have 0 test failures, else continuous integration will fail.
Failing and skipped tests can also highly impact code coverage.
We would rather not see such jumps in the timeline within SonarQube, simply because of a bad commit at the bad time (right before the SQ analysis).

If there are failing or skipped unit tests in a project, the top most priority should be to fix them: it does not functionally work as expected.
It does not make much sense to "manage the technical debt" of a functionally broken project.

To sum up:
 1) the SonarQube analysis is expected to be triggered only if all tests are passing, hence the results are always identical: all green.
 2) Moreover unit tests are also expected to always be fast, and if that is no longer the case, continuous integration should be able to spot it.

Also, the support of Gallio will be fully dropped in this sprint, as it is simply not practical to most users of the .NET plugin.

So the goal for the current .NET 2.1 sprint is definitely to provide native support for the most popular code coverage tools, as a replacement to Gallio.

Of course, feel free to challenge this if you think we're mistaken!

Sorry about SONARDOTNT-372.

Kind regards,





On Fri, Jan 24, 2014 at 2:52 PM, Fabrice Bellingard <[hidden email]> wrote:
Hi John,

thanks for the notification! No problem if you work on side projects, this will give us time to get back on the .NET ecosystem. 

Thanks for your contribution and your efforts!


Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Fri, Jan 24, 2014 at 2:34 PM, John M. Wright <[hidden email]> wrote:
Fabrice and other interested devs:

I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.

If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository: https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.

Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.

Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.

Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :

#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip

#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml

#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True

As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.

Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support


That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john





Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [sonar-dev] NUnit reuseReport support

Dinesh Bolkensteyn-2
Hi again John,

So I managed to do some progress on the C# resource bridge.
There were 2 bugs reported on it, one in SONARDOTNT-382 and one in SONARDOTNT-375.
See my final comment on those 2 tickets for an example of how the keys are now computed.

Now, regarding your request on method keys:

-> You're right, the key should be a logical one, not one based on the line number where the method is declared
-> You're right, we want types in the method signature
-> We don't care about the compatibility of some external plugins -> they should not prevent us from going forward with this plugin
-> Hence we don't want a secondary index, let's stick with just 1 (which already causes us enough troubles :p)

Now I see one corner case which is going to be hard to cover properly:

File1.cs:

using Foo;

public partial MyClass
{
  public void MyMethod(T foo) {} // "T" here comes from the namespace "Foo"
}

File2.cs:

using Bar;

public partial MyClass
{
  public void MyMethod(T foo); // "T" now comes from the namespace "Bar"
}

Ideally, the method keys should contain fully qualified data types, i.e. "MyClass#MyMethod(Foo.T)" for the first case, and "MyClass#MyMethod(Bar.T)" for the second.

However, this requires a complete symbol table to be done.

In the meantime, I think that going for "MyClass#MyMethod(T)"
is going to be sufficient.

That means that we won't properly cover the corner case, but hey, we're doing our best...

Do you think that all this would work for you? (including the changes of SONARDOTNT-382 and SONARDOTNT-375) 

I will take care of the method keys, I've created the following ticket for that: http://jira.codehaus.org/browse/SONARDOTNT-391




On Fri, Jan 31, 2014 at 12:06 PM, Dinesh Bolkensteyn <[hidden email]> wrote:
John,

I need some time to understand what exactly you are referring to, as I am currently in the process of taking over the development of this .NET plugin.

This is in my TODO, and I'll get back to you as soon as I can, sometime next week.
On Thu, Jan 30, 2014 at 1:23 PM, John M. Wright <[hidden email]> wrote:
Dinesh,
As part of my changes, I modified the logic in the existing .net plugins in order to support the AST changes required by my plugin. Would you still consider those changes if I submitted a pull request?  If so, would you respond to this snippet regarding changing the resource bridge index from my original email so I can submit the appropriate implementation:

  • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
    • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.

Thanks
~john


On Jan 29, 2014, at 8:52 AM, "John M. Wright" <[hidden email]> wrote:

As I've mentioned before on this list, I very strongly disagree with the direction your going.

That said, I will likely release the plugin as a standalone for people who do want the features. Is this something that would be allowed into the Forge?

Thanks
~john


On Jan 29, 2014, at 8:21 AM, "Dinesh Bolkensteyn" <[hidden email]> wrote:

Hi John,

Thank you for your work on the .NET plugin.

I am afraid that we slightly changed our minds, and that the ticket you started to work on is no longer going to get fixed:

Without a doubt, code coverage is a good indicator of code quality, and we need to be able to access and visualise it within SonarQube.

However, the story regarding test results is a bit different: A project is expected to always have 0 test failures, else continuous integration will fail.
Failing and skipped tests can also highly impact code coverage.
We would rather not see such jumps in the timeline within SonarQube, simply because of a bad commit at the bad time (right before the SQ analysis).

If there are failing or skipped unit tests in a project, the top most priority should be to fix them: it does not functionally work as expected.
It does not make much sense to "manage the technical debt" of a functionally broken project.

To sum up:
 1) the SonarQube analysis is expected to be triggered only if all tests are passing, hence the results are always identical: all green.
 2) Moreover unit tests are also expected to always be fast, and if that is no longer the case, continuous integration should be able to spot it.

Also, the support of Gallio will be fully dropped in this sprint, as it is simply not practical to most users of the .NET plugin.

So the goal for the current .NET 2.1 sprint is definitely to provide native support for the most popular code coverage tools, as a replacement to Gallio.

Of course, feel free to challenge this if you think we're mistaken!

Sorry about SONARDOTNT-372.

Kind regards,





On Fri, Jan 24, 2014 at 2:52 PM, Fabrice Bellingard <[hidden email]> wrote:
Hi John,

thanks for the notification! No problem if you work on side projects, this will give us time to get back on the .NET ecosystem. 

Thanks for your contribution and your efforts!


Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Fri, Jan 24, 2014 at 2:34 PM, John M. Wright <[hidden email]> wrote:
Fabrice and other interested devs:

I've made some decent progress with the "sonar.dotnet.tests" plugin, but need to set it aside for a few months while I work on other side projects.

If you're interested in taking a look, you can find my code in the "dotnet-tests-plugin" branch of my GitHub clone of the dotnet repository: https://github.com/johnmwright/sonar-dotnet/tree/dotnet-tests-plugin  Note: I have multiple branches in my repo, so make sure you're looking at the "dotnet-tests-plugin" branch.

Some general notes:
  • There is one known open issue: It currently is unable to associate test results to the appropriate classes when the tests are defined in a base class of the test fixture.
  • The plugin utilizes the default SonarQube unit test widget (no need for the separate widget used in the gallio plugin)
  • The plugin is using the org.sonar.api.test classes (TestCase, TestPlan, etc), which required changing the sonar API version to v3.5 (from 3.0). Since 3.7 is the LTS version, I think this should not create too many complaints.
  • I have only implemented the nunit support so far, but it should be pretty straight forward to migrate the mbunit (gallio) parsing logic from the existing gallio plugin, as well as almost all of the coverage report parsing code.
  • The plugin supports only "reuseReport" and "skip" modes.

Implementation details:
  • This branch also includes Peter Steven's Pull request for SONARDOTNT-382: https://github.com/SonarCommunity/sonar-dotnet/pull/8, which is required for the NUnit support to work.
  • As part of this implementation, I made some changes to the base dotnet plugins:
    • Added a property ("sonar.csharp.squid.includeTestSources") to include Test projects when performing squid parsing for c# projects (building out the AST). (Default is 'false' to mirror prior behavior).  This allows consumers to lookup file resources for types/members in test classes. This also required changes to the CSharpSquidSensor so that it will run for test projects and be able to lookup files in the test sources tree but not to include the Test files when saving measures, complexity, issues, etc.
    • CSharpResourcesBridge now also indexes methods using their parameter signatures (ex: My.Namespace.MyClass#MethodName(bool, int)) as a second index. This allows callers to lookup methods based on their usages without having to know the line number they are defined at.
      • The current indexing implementation includes the line number in the index key and the doc for DotNetResourceBridge currently says this:
* /!\ Do not use for the moment! <br>
* <br>
* For the moment, method key ends with ':XXXX', where 'XXXX' is the line
number, so this API does not work. <br>
* <br>
* TODO: Need to work on that.
*
      • As such, the changes I made to include a secondary index for methods could be removed and made the primary index instead. This would remove the need for the custom SourceMethod class as well, but would break anyone who has implemented against the current line number based signature.  I would appreciate feedback on whether this should be a second index or the primary index.
    • Similar changes would likely be required in the VB.NET plugin, which I don't have access to, so haven't tested/reviewed.
  • New unit test result parsers (such as MSTest and MBUnit/gallio) would need to extend o.s.p.dotnet.tests.parser.DotNetTestResultParser and be added to DotNetTestsSensor.getParsers() at the line that currently says "//TODO: add other parsers here".  I would eventually like to refactor this to use dependency injection to discover the parsers, but haven't gotten there yet.

Usage:
For the NUnit parsing, the following properties need to be defined in the project.properties file (or equivalent) :

#
# global flags to enable/disable all parsing. Integration tests have a separate flag.
#
sonar.dotnet.tests.mode=reuseReport
sonar.dotnet.tests.it.mode=skip

#
# each parser will have it's own pair of report path keys (unit and integration):
#
sonar.dotnet.tests.nunit.reports.path=$(SolutionDir)/TestResult*.xml
#sonar.dotnet.tests.nunit.it.reports.path=$(SolutionDir)/AggregateResults.xml

#
# The NUnit parser requires the new c# squid property to be set to true
#
sonar.csharp.squid.includeTestSources=True

As I mentioned, I will be setting this aside for several months as I will working on another side project.  I would appreciate anyone who could review things as they currently stand and provide any feedback, particularly from the current dotnet plugin maintainers and SourceSource crew.

Thanks
~john


From: [hidden email]
To: [hidden email]
Date: Thu, 5 Dec 2013 11:23:36 -0600
Subject: RE: [sonar-dev] NUnit reuseReport support


That sounds great!
thanks
~john


From: [hidden email]
Date: Thu, 5 Dec 2013 18:05:55 +0100
To: [hidden email]
Subject: Re: [sonar-dev] NUnit reuseReport support

Hi John,

starting with next version (2.2):
  • we want to deprecate the use of the Gallio plugin to warn users that its support will be removed in a "near" future
  • we'll work on the support of reuseReports for some unit test and code coverage tools - ideally the most widely used ones
And when we support reuseReports for most unit test and code coverage tools (in some months? in one year?), we'll definitely drop automatic execution of Gallio.


So for now (because that's what you're interesting in ;-)), we could create a new sonar-dotnet-tests-plugin (under the "dotnet" folder) that would host all the parsers & associated sensors for unit test and code coverage tools (1 per package - for instance "org.sonar.plugins.dotnet.tests.mstest", "org.sonar.plugins.dotnet.tests.mscoverage", "org.sonar.plugins.dotnet.tests.nunit",....). We may split it later into independent plugins - but we'll see that later.
So if you want to contribute to the .NET ecosystem by SONARDOTNT-372 for instance, here's how we could proceed for the first steps together:
  • you clone the .NET ecosystem on your GitHub account
  • you add this new sonar-dotnet-tests-plugin with a first support for NUnit
  • we review all this together
  • and based on this, we'll see the next steps (probably merge your modifications and then give you the possibility to directly work on the original GitHub repository)
How does that sound to you?




Best regards,

Fabrice BELLINGARD | SonarSource
http://sonarsource.com


On Mon, Dec 2, 2013 at 2:19 PM, John M. Wright <[hidden email]> wrote:
dotNet plugin devs:

Looking at the v2.2 release backlog, you have this tagged for the release:

SONARDOTNT-372 Make it possible to reuse NUnit reports

I may be able to provide some assistance in writing this support. Can you provide some detail as to how you were planning to achieve this?  Do you envision a new plugin, or adding it to the existing gallio plugin?  If a new plugin, did you see this as an NUnit-specific plugin, or a shared plugin with MSTest and MSCoverage (which are also slated for the 2.2 release)?

Along those lines, after NUnit support, my next target will be NCover report support, which would follow the same patterns, so any guidance you have for the coverage reports would be helpful as well.

Thanks
~john






Loading...