Subscribe in a reader« The Economics of War | Main | Parents: Use Your Head »
August 3, 2006
java.net.SocketException Broken Pipe Exception
For the last week, I've been chasing down a randomly occuring broken pipe exception in the banking app we've written. It was like looking for a needle in a haystack.
The Problem
Database connections were being tied up to the length of wait_timeout on the MySQL server. We're using Hibernate for our persistence layer and C3P0 for our connection pool. During initial pilot testing, we only saw issues after a long period of time. My assumption was that the pool was not doing its job and MySQL was killing the connections. However, after ramping up the testing to a larger scale, the exception began occuring much more often.
Here is the dump I was seeing:
10:01:57,812 WARN NewPooledConnection:356 - [c3p0] A PooledConnection that has already signalled a Connection error is still in use!
10:01:57,813 WARN NewPooledConnection:357 - [c3p0] Another error has occurred [ com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:** BEGIN NESTED EXCEPTION **
java.net.SocketException
MESSAGE: Broken pipeSTACKTRACE:
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java(Compiled Code))
at java.net.SocketOutputStream.write(SocketOutputStream.java(Compiled Code))
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java(Inlined Compiled Code))
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java(Inlined Compiled Code))
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java(Compiled Code))
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java(Inlined Compiled Code))
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java(Compiled Code))
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java(Compiled Code))
at com.mysql.jdbc.Connection.execSQL(Connection.java(Compiled Code))
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:936)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1030)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1669)
at org.hibernate.loader.Loader.doQuery(Loader.java:662)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
at org.hibernate.loader.Loader.doList(Loader.java:2150)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2029)
at org.hibernate.loader.Loader.list(Loader.java:2024)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:369)
at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:300)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:153)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1127)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:749)
at com.bbcs.consumercard.dao.AccountDAO.getAccount(AccountDAO.java:72)
at com.bbcs.consumercard.actions.account.WebPayRequestAction.doExecute(WebPayRequestAction.java:53)
at com.bbcs.consumercard.actions.SecureBaseAction.execute(SecureBaseAction.java:38)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:570)
** END NESTED EXCEPTION **
Possible Solutions
Initially I thought the problem may lie somewhere in our code, but since this was the first time I used C3P0 for my connection pool I decided to Google the problem first. Many of the postings I found related to failures on the part of the connection pool. Based on this post, I ripped out C3P0 and implemented DBCP. That did not fix my problem, though. Some posts say it is a problem with the driver (I do not believe that it is), others blame the c3p0 connection pool (I do not).
Our Solution
In fact, our problem was sitting in our own code and was implemented by someone who was not seasoned in transactions. When you demarcate transactions, you have to explicity commit or rollback every time. Some newbies believe that queries do not have to be committed. They do. If not, the connection is left assigned in case you wanted to manipulate the resultset or do additional reads (there are other reasons, but these are the easiest to wrap your mind around).
So in our case, a transaction was begun and a query was executed. Unfortunately, the transaction was not committed in every circumstance, leaving dangling connections that later got killed by MySQL. C3P0 would later attempt to reattach to the now defunct dangling connection and kaboom - java.net.SocketException Broken PIpe would appear. Finding the spot in the code (it was in some complicated if/else logic) and correcting it solved our problem and we have not had a problem with Hibernate or C3P0 since!
Update: 7 August 2006
In the 3 days since I've posted this, it has already garnered a lot of page views. I didn't realize it would be such a hit. For most of you, it should just be a matter of setting up periodic connection testing. Here are some more pointers to information if you are getting this error:
C3P0: Configuring Connection Testing
Hibernate specific properties for C3P0
More Hibernate specific configuration info
Updated 15 February 2007 - fixed link to "Hibernate Specific Properties for C3P0"
Posted by Mike on August 3, 2006 3:30 PM
Comments
1. java.net.SocketException:Connection reset by peer:
JVM_recv in socket input stream read.
2. IOException:Connection reset by peer:socket write error.
3. IOExcepion:Broken pipe.
All this exceptions are not thrown at the same time and might not always been thrown.
These Exceptions are mainly caused due to the resubmission of jsp page twice. By using Input type = 'submit' itself submits the page not need to call from java script as form.submit. If we want to use form.submit from java script for Input type as 'submit' then you need to disable the submit button after one click.
The threads running under the server may not be executing synchronously and inappropriate page submissions can cause this exceptions.
Posted by: Naveen at October 11, 2006 3:31 PM
cool info
Thanks!
Posted by: Ovidiu at December 15, 2006 3:29 AM
Thanks for the info, it's very valuable.
The "Hibernate specific properties for C3P0" link is broken.
I think the correct link is : http://www.mchange.com/projects/c3p0/index.html#hibernate-specific
Posted by: Sébastien Christy at February 15, 2007 5:43 AM
Thank you for the info. It is very helpful.
Posted by: Yassin SIbai at May 17, 2007 11:30 AM
Thank you very much! Very useful!
Posted by: Klaus Bechtold at September 29, 2007 4:37 AM
Good One!
Thanks
Dina.
Posted by: dinakar at October 7, 2007 4:41 PM
Thank you! Your report helped me very much!!
Posted by: Jens Wienstroer at November 15, 2007 11:09 AM
Short cut approach could be use
<property name="hibernate.connection.autocommit">true</property>
to make the auto commit happen.
Regards,
Raja Nagendra Kumar
Java Code Audit and Refactoring Offshore company
Posted by: Raja Nagendra Kumar at April 13, 2008 2:48 AM
Good post. Thanks!
Posted by: Arman at April 29, 2008 1:47 PM
thanks for this Guidence.
Posted by: prashant at May 7, 2008 9:21 AM
Thanks a lot
Posted by: nyjava at June 25, 2008 5:15 AM
Thank !
Posted by: Julien at August 7, 2008 8:34 PM