Java heap space error when migrating from 5.0.1 -> 5.1-RC2

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Java heap space error when migrating from 5.0.1 -> 5.1-RC2

Laurent TOURREAU-2

Hi guys

 

I tried to migrate from 5.0.1 to 5.1-RC2.

 

Unfortunately I have the following memory issue:

 


2015.03.26 16:41:20 INFO  web[DbMigration] ==  AddFileSourcesBinaryData: migrated (0.0410s) ==============================

2015.03.26 16:41:20 INFO  web[DbMigration]

2015.03.26 16:41:21 INFO  web[DbMigration] ==  FeedFileSourcesBinaryData: migrating ======================================

java.lang.OutOfMemoryError: Java heap space

Dumping heap to java_pid4424.hprof ...

Heap dump file created [119038342 bytes in 1.246 secs]

2015.03.26 16:41:49 ERROR web[o.s.s.ui.JRubyFacade] Fail to upgrade database

An error has occurred, all later migrations canceled:

 

Java heap space

        java.lang.reflect.Array.newArray(Native Method)

        java.lang.reflect.Array.newInstance(Array.java:70)

        oracle.jdbc.driver.BufferCache.get(BufferCache.java:226)

        oracle.jdbc.driver.PhysicalConnection.getCharBuffer(PhysicalConnection.java:7698)

        oracle.jdbc.driver.PhysicalConnection.getCharBufferSync(PhysicalConnection.java:7704)

        oracle.sql.CLOB.getSubString(CLOB.java:317)

        oracle.jdbc.driver.T4CClobAccessor.getString(T4CClobAccessor.java:474)

        oracle.jdbc.driver.OracleResultSetImpl.getString(OracleResultSetImpl.java:1297)

        org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)

        org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:213)

        org.sonar.server.db.migrations.Select$Row.getNullableString(Select.java:82)

        org.sonar.server.db.migrations.v51.FeedFileSourcesBinaryData$1.handle(FeedFileSourcesBinaryData.java:60)

        org.sonar.server.db.migrations.MassUpdate$1.handle(MassUpdate.java:79)

        org.sonar.server.db.migrations.SelectImpl.scroll(SelectImpl.java:79)

        org.sonar.server.db.migrations.MassUpdate.execute(MassUpdate.java:76)

        org.sonar.server.db.migrations.v51.FeedFileSourcesBinaryData.execute(FeedFileSourcesBinaryData.java:56)

        org.sonar.server.db.migrations.BaseDataChange.execute(BaseDataChange.java:45)

        org.sonar.server.db.migrations.DatabaseMigrator.executeMigration(DatabaseMigrator.java:94)

        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        java.lang.reflect.Method.invoke(Method.java:601)

        org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:455)

        org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:316)

       org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:61)

        org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)

        org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)

        org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)

        org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)

        org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)

        org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)

        org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:198)

 

Cordialement / Best regards

 

Laurent TOURREAU

Fixed Income IT F2B BO CMQA Team

 

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Reply | Threaded
Open this post in threaded view
|

FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

Laurent TOURREAU-2

Hi guys

 

Finally i changed the Xmx value on web process :

-Xmx1024 Mo -> heap space error

-Xmx2048Mo -> heap space error

-Xmx4096Mo -> heap space error
-Xmx8192Mo -> Ok but now the migration failed at this step:

2015.03.26 17:04:24 INFO  web[DbMigration] ==  FeedFileSourcesBinaryData: migrating ======================================

2015.03.26 17:05:24 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (3 items/sec)

2015.03.26 17:06:24 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:07:33 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:08:35 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:09:42 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:10:51 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:12:02 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:13:11 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:13:30 WARN  web[o.e.transport] [sonar-1427389402053] Transport response handler not found of id [72]

2015.03.26 17:14:18 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:15:24 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:16:43 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:18:00 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:19:06 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:26:49 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:26:49 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:26:49 INFO  web[o.e.client.transport] [sonar-1427389402053] failed to get node info for [#transport#-1][myhost.net.intra][inet[/127.0.0.1:9001]], disconnecting...

org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/127.0.0.1:9001]][cluster:monitor/nodes/info] request_id [72] timed out after [20214ms]

        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) ~[elasticsearch-1.4.4.jar:na]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_21]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_21]

        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21]

2015.03.26 17:26:56 INFO  app[o.s.p.m.TerminatorThread] Process[search] is stopping

2015.03.26 17:26:57 INFO   es[o.s.p.StopWatcher]  Stopping process

 

What’s wrong?

 

Regards

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Reply | Threaded
Open this post in threaded view
|

Re: FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

Simon Brandhof
Hi Laurent,

Which version of Oracle driver do you use ? And could you please execute the following SQL request ?
select max(length(data)) from file_sources

Thanks


Simon BRANDHOF | SonarSource
Tech Lead & Co-Founder
http://twitter.com/SimonBrandhof

On 27 March 2015 at 09:51, Laurent TOURREAU <[hidden email]> wrote:

Hi guys

 

Finally i changed the Xmx value on web process :

-Xmx1024 Mo -> heap space error

-Xmx2048Mo -> heap space error

-Xmx4096Mo -> heap space error
-Xmx8192Mo -> Ok but now the migration failed at this step:

2015.03.26 17:04:24 INFO  web[DbMigration] ==  FeedFileSourcesBinaryData: migrating ======================================

2015.03.26 17:05:24 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (3 items/sec)

2015.03.26 17:06:24 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:07:33 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:08:35 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:09:42 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:10:51 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:12:02 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:13:11 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:13:30 WARN  web[o.e.transport] [sonar-1427389402053] Transport response handler not found of id [72]

2015.03.26 17:14:18 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:15:24 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:16:43 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:18:00 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:19:06 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:26:49 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:26:49 INFO  web[o.s.s.d.m.MassUpdate] 225 issues processed (0 items/sec)

2015.03.26 17:26:49 INFO  web[o.e.client.transport] [sonar-1427389402053] failed to get node info for [#transport#-1][myhost.net.intra][inet[/127.0.0.1:9001]], disconnecting...

org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/127.0.0.1:9001]][cluster:monitor/nodes/info] request_id [72] timed out after [20214ms]

        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) ~[elasticsearch-1.4.4.jar:na]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_21]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_21]

        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_21]

2015.03.26 17:26:56 INFO  app[o.s.p.m.TerminatorThread] Process[search] is stopping

2015.03.26 17:26:57 INFO   es[o.s.p.StopWatcher]  Stopping process

 

What’s wrong?

 

Regards

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.


Reply | Threaded
Open this post in threaded view
|

RE: FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

Laurent TOURREAU-2

Hi Simon

 

I use ojdbc6-11.2.0.4.0.jar.
My database version is :

 

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

PL/SQL Release 11.2.0.4.0 - Production

"CORE   11.2.0.4.0            Production"

TNS for Linux: Version 11.2.0.4.0 - Production

NLSRTL Version 11.2.0.4.0 - Production

 

The result select max(length(data)) from file_sources:

 

919192479

 

Regards

 

Laurent TOURREAU

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Reply | Threaded
Open this post in threaded view
|

RE: FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

Laurent TOURREAU-2
In reply to this post by Simon Brandhof

Simon

 

Eventually we have rollbacked in 5.0.1 removed the project where there was a huge file into it.

 

We have restarted the migration to version 5.1.

 

Now the migration is successful.

Regards

 

Laurent

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Reply | Threaded
Open this post in threaded view
|

Re: FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

Simon Brandhof
So this row is more than 900M characters. That's huge. It means that the source file is at least some dozens of millions characters. Do you know what kind of source file is it ?
We will try to better handle such corner-case in future version. 


Simon BRANDHOF | SonarSource
Tech Lead & Co-Founder
http://twitter.com/SimonBrandhof

On 27 March 2015 at 14:42, Laurent TOURREAU <[hidden email]> wrote:

Simon

 

Eventually we have rollbacked in 5.0.1 removed the project where there was a huge file into it.

 

We have restarted the migration to version 5.1.

 

Now the migration is successful.

Regards

 

Laurent

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.


Reply | Threaded
Open this post in threaded view
|

RE: FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

Laurent TOURREAU-2

Hi Simon

 

The file in question was a .osql containing SQL code.
In fact this file contained a lot of INSERT clauses to initialize all tables in a database (I suppose mainly for testing purpose).

We have deleted the project containing this file because it was for PL/SQL plugin evaluation purpose.

 

How do you think to deal with such case?

Only analyzing files which don’t oversize 10 Mbytes for instance, is a good solution according  me.

 

Regards

 

Laurent TOURREAU

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Reply | Threaded
Open this post in threaded view
|

RE: FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

dgrierso

It's arguable that a 10Mb source file is going to be difficult to maintain and should be refactored into constituent source files which perform specific functions.

 

Perhaps this could/should be a particular code quality rule?

 

The Atlassian FishEye code indexer has a configuration option which prohibits the indexing of files larger than a configured size.

 

My suggestion would be:

·         Introduce a new rule for _source_ file size

·         Analyse all files using the appropriate code quality rules

·         Only store the code for source files under an appropriate size (e.g. 5Mb)

 

Dg.


--
David Grierson - SDLC Tools Specialist

Sky Broadcasting - Customer Business Systems - SDLC Tools

Email: [hidden email]

Watermark Building, Alba Campus, Livingston, EH54 7HH

 

 

From: Laurent TOURREAU [mailto:[hidden email]]
Sent: 30 March 2015 08:55
To: [hidden email]
Subject: RE: [sonar-user] FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

 

Hi Simon

 

The file in question was a .osql containing SQL code.
In fact this file contained a lot of INSERT clauses to initialize all tables in a database (I suppose mainly for testing purpose).

We have deleted the project containing this file because it was for PL/SQL plugin evaluation purpose.

 

How do you think to deal with such case?

Only analyzing files which don’t oversize 10 Mbytes for instance, is a good solution according  me.

 

Regards

 

Laurent TOURREAU

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky plc and Sky International AG and are used under licence. Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.
Reply | Threaded
Open this post in threaded view
|

Re: FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

Simon Brandhof
I agree. https://jira.codehaus.org/browse/SONAR-6178 is about defining break values on the data to be persisted.
Regards


Simon BRANDHOF | SonarSource
Tech Lead & Co-Founder
http://twitter.com/SimonBrandhof

On 30 March 2015 at 11:52, Grierson, David <[hidden email]> wrote:

It's arguable that a 10Mb source file is going to be difficult to maintain and should be refactored into constituent source files which perform specific functions.

 

Perhaps this could/should be a particular code quality rule?

 

The Atlassian FishEye code indexer has a configuration option which prohibits the indexing of files larger than a configured size.

 

My suggestion would be:

·         Introduce a new rule for _source_ file size

·         Analyse all files using the appropriate code quality rules

·         Only store the code for source files under an appropriate size (e.g. 5Mb)

 

Dg.


--
David Grierson - SDLC Tools Specialist

Sky Broadcasting - Customer Business Systems - SDLC Tools

Email: [hidden email]

Watermark Building, Alba Campus, Livingston, EH54 7HH

 

 

From: Laurent TOURREAU [mailto:[hidden email]]
Sent: 30 March 2015 08:55
To: [hidden email]
Subject: RE: [sonar-user] FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

 

Hi Simon

 

The file in question was a .osql containing SQL code.
In fact this file contained a lot of INSERT clauses to initialize all tables in a database (I suppose mainly for testing purpose).

We have deleted the project containing this file because it was for PL/SQL plugin evaluation purpose.

 

How do you think to deal with such case?

Only analyzing files which don’t oversize 10 Mbytes for instance, is a good solution according  me.

 

Regards

 

Laurent TOURREAU

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky plc and Sky International AG and are used under licence. Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.

Reply | Threaded
Open this post in threaded view
|

RE: FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

dgrierso

Thanks – I've added a comment to that issue to this effect.

 

Dg.


--
David Grierson - SDLC Tools Specialist

Sky Broadcasting - Customer Business Systems - SDLC Tools

Email: [hidden email]

Watermark Building, Alba Campus, Livingston, EH54 7HH

 

 

From: Simon Brandhof [mailto:[hidden email]]
Sent: 30 March 2015 11:03
To: [hidden email]
Subject: Re: [sonar-user] FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

 

I agree. https://jira.codehaus.org/browse/SONAR-6178 is about defining break values on the data to be persisted.

Regards


 

Simon BRANDHOF | SonarSource

Tech Lead & Co-Founder
http://twitter.com/SimonBrandhof

 

On 30 March 2015 at 11:52, Grierson, David <[hidden email]> wrote:

It's arguable that a 10Mb source file is going to be difficult to maintain and should be refactored into constituent source files which perform specific functions.

 

Perhaps this could/should be a particular code quality rule?

 

The Atlassian FishEye code indexer has a configuration option which prohibits the indexing of files larger than a configured size.

 

My suggestion would be:

·         Introduce a new rule for _source_ file size

·         Analyse all files using the appropriate code quality rules

·         Only store the code for source files under an appropriate size (e.g. 5Mb)

 

Dg.


--
David Grierson - SDLC Tools Specialist

Sky Broadcasting - Customer Business Systems - SDLC Tools

Email: [hidden email]

Watermark Building, Alba Campus, Livingston, EH54 7HH

 

 

From: Laurent TOURREAU [mailto:[hidden email]]
Sent: 30 March 2015 08:55
To: [hidden email]
Subject: RE: [sonar-user] FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

 

Hi Simon

 

The file in question was a .osql containing SQL code.
In fact this file contained a lot of INSERT clauses to initialize all tables in a database (I suppose mainly for testing purpose).

We have deleted the project containing this file because it was for PL/SQL plugin evaluation purpose.

 

How do you think to deal with such case?

Only analyzing files which don’t oversize 10 Mbytes for instance, is a good solution according  me.

 

Regards

 

Laurent TOURREAU

 

This message and any attachments (the "message") is
intended solely for the intended addressees and is confidential.
If you receive this message in error,or are not the intended recipient(s),
please delete it and any copies from your systems and immediately notify
the sender. Any unauthorized view, use that does not comply with its purpose,
dissemination or disclosure, either whole or partial, is prohibited. Since the internet
cannot guarantee the integrity of this message which may not be reliable, BNP PARIBAS
(and its subsidiaries) shall not be liable for the message if modified, changed or falsified.
Do not print this message unless it is necessary,consider the environment.

----------------------------------------------------------------------------------------------------------------------------------

Ce message et toutes les pieces jointes (ci-apres le "message")
sont etablis a l'intention exclusive de ses destinataires et sont confidentiels.
Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
immediatement l'expediteur. Toute lecture non autorisee, toute utilisation de
ce message qui n'est pas conforme a sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer
l'integrite de ce message electronique susceptible d'alteration, BNP Paribas
(et ses filiales) decline(nt) toute responsabilite au titre de ce message dans l'hypothese
ou il aurait ete modifie, deforme ou falsifie.
N'imprimez ce message que si necessaire, pensez a l'environnement.

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky plc and Sky International AG and are used under licence. Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.

 

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky plc and Sky International AG and are used under licence. Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of Sky plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.
Reply | Threaded
Open this post in threaded view
|

RE: FW: Java heap space error when migrating from 5.0.1 -> 5.1-RC2

sberthiaume
I had the same issue as Laurent did but with the release version of 5.1, but my file was "only" 73M characters; setting the -Xmx to 8g was, however, enough to have the migration work.

I understand that those are fringe cases, but, at the same time, people that are only starting to track quality might have some very ugly code files from time to time.