Discussion:
Frequent FULL gc's and then heap crash, why?
(too old to reply)
k***@gmail.com
2013-12-23 13:41:32 UTC
Permalink
Hello,

I am running weblogic with jdk 1.7_0_25. We had been running an application for YEARS on jdk 1.6_0_xx without issue. When we migrated to jdk 1.7 we started having issues with GC.... Server works fine for several days, I see normal gc's, but then all of a sudden it goes in a continuous FULL GC cycle and will not stop... ideas??

416355.858: [GC [PSYoungGen: 620307K->57440K(620736K)] 1991153K->1428285K(2018880K), 0.0900130 secs] [Times: user=0.36 sys=0.00, real=0.11 secs]
416392.757: [GC [PSYoungGen: 605024K->62587K(610176K)] 1975869K->1433432K(2008320K), 0.0565270 secs] [Times: user=0.41 sys=0.00, real=0.05 secs]
416463.418: [GC [PSYoungGen: 610171K->51415K(620672K)] 1981016K->1430601K(2018816K), 0.0516100 secs] [Times: user=0.37 sys=0.00, real=0.05 secs]
416736.929: [GC [PSYoungGen: 592535K->50944K(619456K)] 1971721K->1432341K(2017600K), 0.0512740 secs] [Times: user=0.36 sys=0.00, real=0.05 secs]
416865.440: [GC [PSYoungGen: 592064K->51736K(623232K)] 1973461K->1435101K(2021376K), 0.0505060 secs] [Times: user=0.33 sys=0.01, real=0.05 secs]
417129.884: [GC [PSYoungGen: 598104K->51949K(622144K)] 1981469K->1436658K(2020288K), 0.0504380 secs] [Times: user=0.33 sys=0.00, real=0.05 secs]
417322.683: [GC [PSYoungGen: 598317K->54181K(624576K)] 1983026K->1440818K(2022720K), 0.0506910 secs] [Times: user=0.36 sys=0.00, real=0.05 secs]
417597.562: [GC [PSYoungGen: 604069K->52297K(624320K)] 1990706K->1444571K(2022464K), 0.0752950 secs] [Times: user=0.35 sys=0.00, real=0.07 secs]
417597.638: [Full GC [PSYoungGen: 52297K->1022K(624320K)] [ParOldGen: 1392273K->1398143K(1398144K)] 1444571K->1399166K(2022464K) [PSPermGen: 162096K->161853K(199808K)], 2.3327670 secs] [Times: user=15.54 sys=0.03, real=2.33 secs]
417743.769: [Full GC [PSYoungGen: 550910K->11213K(624320K)] [ParOldGen: 1398143K->1396146K(1398144K)] 1949054K->1407360K(2022464K) [PSPermGen: 161853K->161853K(194496K)], 2.7090850 secs] [Times: user=18.77 sys=0.06, real=2.71 secs]
418023.140: [Full GC [PSYoungGen: 549888K->9270K(624320K)] [ParOldGen: 1396146K->1397934K(1398144K)] 1946034K->1407205K(2022464K) [PSPermGen: 161853K->161853K(189248K)], 2.8707310 secs] [Times: user=19.94 sys=0.08, real=2.87 secs]
418342.476: [Full GC [PSYoungGen: 549888K->2929K(624320K)] [ParOldGen: 1397934K->1398143K(1398144K)] 1947822K->1401072K(2022464K) [PSPermGen: 161894K->161835K(184960K)], 2.7683850 secs] [Times: user=18.73 sys=0.05, real=2.80 secs]
.
.
.

Exception in thread "DoSManager"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DoSManager"
Exception in thread "Thread-13" java.lang.OutOfMemoryError: Java heap space

<Dec 22, 2013 5:42:07 AM EST> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
<Dec 22, 2013 5:42:57 AM EST> <Error> <Server> <BEA-002608> <The ListenThread failed because of an error: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at weblogic.utils.io.Chunk.<init>(Chunk.java:348)
at weblogic.utils.io.Chunk.getChunk(Chunk.java:183)
at weblogic.socket.AbstractMuxableSocket.<init>(AbstractMuxableSocket.java:117)
at weblogic.socket.MuxableSocketDiscriminator.<init>(MuxableSocketDiscriminator.java:51)
at weblogic.server.channels.DynamicListenThread.registerSocket(DynamicListenThread.java:389)
Truncated. see log file for complete stacktrace
k***@gmail.com
2013-12-23 13:48:21 UTC
Permalink
Here is the last GC log entries:

606667.298: [Full GC [PSYoungGen: 549888K->540087K(624320K)] [ParOldGen: 1398140K->1398140K(1398144K)] 1948028K->1938228K(2022464K) [PSPermGen: 160980K->160980K(161280K)], 1.8969770 secs] [Times: user=12.32 sys=0.00, real=1.90 secs]
606710.698: [Full GC [PSYoungGen: 549888K->540087K(624320K)] [ParOldGen: 1398140K->1398140K(1398144K)] 1948028K->1938228K(2022464K) [PSPermGen: 160980K->160980K(161280K)], 1.8543470 secs] [Times: user=12.22 sys=0.01, real=1.86 secs]
606754.098: [Full GC [PSYoungGen: 549888K->540087K(624320K)] [ParOldGen: 1398140K->1398140K(1398144K)] 1948028K->1938228K(2022464K) [PSPermGen: 160980K->160980K(161280K)], 1.8512890 secs] [Times: user=12.26 sys=0.00, real=1.86 secs]
Post by k***@gmail.com
Hello,
I am running weblogic with jdk 1.7_0_25. We had been running an application for YEARS on jdk 1.6_0_xx without issue. When we migrated to jdk 1.7 we started having issues with GC.... Server works fine for several days, I see normal gc's, but then all of a sudden it goes in a continuous FULL GC cycle and will not stop... ideas??
416355.858: [GC [PSYoungGen: 620307K->57440K(620736K)] 1991153K->1428285K(2018880K), 0.0900130 secs] [Times: user=0.36 sys=0.00, real=0.11 secs]
416392.757: [GC [PSYoungGen: 605024K->62587K(610176K)] 1975869K->1433432K(2008320K), 0.0565270 secs] [Times: user=0.41 sys=0.00, real=0.05 secs]
416463.418: [GC [PSYoungGen: 610171K->51415K(620672K)] 1981016K->1430601K(2018816K), 0.0516100 secs] [Times: user=0.37 sys=0.00, real=0.05 secs]
416736.929: [GC [PSYoungGen: 592535K->50944K(619456K)] 1971721K->1432341K(2017600K), 0.0512740 secs] [Times: user=0.36 sys=0.00, real=0.05 secs]
416865.440: [GC [PSYoungGen: 592064K->51736K(623232K)] 1973461K->1435101K(2021376K), 0.0505060 secs] [Times: user=0.33 sys=0.01, real=0.05 secs]
417129.884: [GC [PSYoungGen: 598104K->51949K(622144K)] 1981469K->1436658K(2020288K), 0.0504380 secs] [Times: user=0.33 sys=0.00, real=0.05 secs]
417322.683: [GC [PSYoungGen: 598317K->54181K(624576K)] 1983026K->1440818K(2022720K), 0.0506910 secs] [Times: user=0.36 sys=0.00, real=0.05 secs]
417597.562: [GC [PSYoungGen: 604069K->52297K(624320K)] 1990706K->1444571K(2022464K), 0.0752950 secs] [Times: user=0.35 sys=0.00, real=0.07 secs]
417597.638: [Full GC [PSYoungGen: 52297K->1022K(624320K)] [ParOldGen: 1392273K->1398143K(1398144K)] 1444571K->1399166K(2022464K) [PSPermGen: 162096K->161853K(199808K)], 2.3327670 secs] [Times: user=15.54 sys=0.03, real=2.33 secs]
417743.769: [Full GC [PSYoungGen: 550910K->11213K(624320K)] [ParOldGen: 1398143K->1396146K(1398144K)] 1949054K->1407360K(2022464K) [PSPermGen: 161853K->161853K(194496K)], 2.7090850 secs] [Times: user=18.77 sys=0.06, real=2.71 secs]
418023.140: [Full GC [PSYoungGen: 549888K->9270K(624320K)] [ParOldGen: 1396146K->1397934K(1398144K)] 1946034K->1407205K(2022464K) [PSPermGen: 161853K->161853K(189248K)], 2.8707310 secs] [Times: user=19.94 sys=0.08, real=2.87 secs]
418342.476: [Full GC [PSYoungGen: 549888K->2929K(624320K)] [ParOldGen: 1397934K->1398143K(1398144K)] 1947822K->1401072K(2022464K) [PSPermGen: 161894K->161835K(184960K)], 2.7683850 secs] [Times: user=18.73 sys=0.05, real=2.80 secs]
.
.
.
Exception in thread "DoSManager"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "DoSManager"
Exception in thread "Thread-13" java.lang.OutOfMemoryError: Java heap space
<Dec 22, 2013 5:42:07 AM EST> <Error> <Socket> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
<Dec 22, 2013 5:42:57 AM EST> <Error> <Server> <BEA-002608> <The ListenThread failed because of an error: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
at weblogic.utils.io.Chunk.<init>(Chunk.java:348)
at weblogic.utils.io.Chunk.getChunk(Chunk.java:183)
at weblogic.socket.AbstractMuxableSocket.<init>(AbstractMuxableSocket.java:117)
at weblogic.socket.MuxableSocketDiscriminator.<init>(MuxableSocketDiscriminator.java:51)
at weblogic.server.channels.DynamicListenThread.registerSocket(DynamicListenThread.java:389)
Truncated. see log file for complete stacktrace
markspace
2013-12-23 22:37:25 UTC
Permalink
Post by k***@gmail.com
Hello,
I am running weblogic with jdk 1.7_0_25. We had been running an
application for YEARS on jdk 1.6_0_xx without issue. When we migrated
to jdk 1.7 we started having issues with GC.... Server works fine for
several days, I see normal gc's, but then all of a sudden it goes in
a continuous FULL GC cycle and will not stop... ideas??
[Full GC [PSYoungGen: 52297K->1022K(624320K)]
609 MB total
Post by k***@gmail.com
[ParOldGen: 1392273K->1398143K(1398144K)] 1444571K->1399166K(2022464K)
1975 MB total
Post by k***@gmail.com
[PSPermGen: 162096K->161853K(199808K)], 2.3327670 secs]
195 MB total
Post by k***@gmail.com
[Times: user=15.54 sys=0.03, real=2.33 secs]
I think these number look a bit low for a "real" app. Are you running
64 or 32 bit Java? Can you verify that the new JVM is getting the
correct memory parameters when your app starts? Can you spit those
parameters out here on this list so we can see them? Are there any
other GC or memory parameters the VM starts with that we should know about?

Otherwise it looks normal. That is, you should expect to see the GC
thrash itself just before you run out of memory, and that's what's
happening. The problem is the OOM, not the thrashing.
Roedy Green
2013-12-28 10:27:34 UTC
Permalink
I am running weblogic with jdk 1.7_0_25. We had been running an application=
for YEARS on jdk 1.6_0_xx without issue. When we migrated to jdk 1.7 we st=
arted having issues with GC.... Server works fine for several days, I see n=
ormal gc's, but then all of a sudden it goes in a continuous FULL GC cycle =
and will not stop... ideas??
There are java.exe command line settings for the size of the virtual
address space. If it is too small, you fill it almost to the brim, and
it will go nuts running out of space over and over having to redo the
GC.

See http://mindprod.com/jgloss/javaexe.html
--
Roedy Green Canadian Mind Products http://mindprod.com
Unlike many machines, computers require no water once they are
manufactured.
John B. Matthews
2013-12-29 11:44:39 UTC
Permalink
Post by Roedy Green
I am running weblogic with jdk 1.7_0_25. We had been running an application=
for YEARS on jdk 1.6_0_xx without issue. When we migrated to jdk 1.7 we st=
arted having issues with GC.... Server works fine for several days, I see n=
ormal gc's, but then all of a sudden it goes in a continuous FULL GC cycle =
and will not stop... ideas??
There are java.exe command line settings for the size of the virtual
address space. If it is too small, you fill it almost to the brim, and
it will go nuts running out of space over and over having to redo the
GC.
See http://mindprod.com/jgloss/javaexe.html
Some related examples are examined here:

<http://stackoverflow.com/a/15307253/230513>
--
John B. Matthews
trashgod at gmail dot com
<http://sites.google.com/site/drjohnbmatthews>
Loading...