Mike Schubert

A strange error started occuring in our deployment of Lotus Connections a month or so back. The business card functionality started showing the error "A profile does not exist for this person". It was interesting because you could be sitting on one of the affected users' profiles, initiate the business card popup and still receive this error.

Turns out that if any part of the JSON processing fails, this is the error you get. It may be that a profile was truly not found, or it may be that something invalid was returned. In our case, a field that was included on the business card contained an apostrophe - and the apostrophe was not escaped or URL encoded, so it was terminating a string prematurely. Once I put code in to begin encoding the apostrophe all was fine.

For the last week, I've been chasing down a randomly occuring broken pipe exception in the banking app we've written. It was like looking for a needle in a haystack.

The Problem

Database connections were being tied up to the length of wait_timeout on the MySQL server. We're using Hibernate for our persistence layer and C3P0 for our connection pool. During initial pilot testing, we only saw issues after a long period of time. My assumption was that the pool was not doing its job and MySQL was killing the connections. However, after ramping up the testing to a larger scale, the exception began occuring much more often.

Here is the dump I was seeing:

10:01:57,812 WARN NewPooledConnection:356 - [c3p0] A PooledConnection that has already signalled a Connection error is still in use!
10:01:57,813 WARN NewPooledConnection:357 - [c3p0] Another error has occurred [ com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:

 

** BEGIN NESTED EXCEPTION **

java.net.SocketException
MESSAGE: Broken pipe

STACKTRACE:

java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java(Compiled Code))
at java.net.SocketOutputStream.write(SocketOutputStream.java(Compiled Code))
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java(Inlined Compiled Code))
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java(Inlined Compiled Code))
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java(Compiled Code))
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java(Inlined Compiled Code))
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java(Compiled Code))
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java(Compiled Code))
at com.mysql.jdbc.Connection.execSQL(Connection.java(Compiled Code))
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:936)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1030)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1669)
at org.hibernate.loader.Loader.doQuery(Loader.java:662)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
at org.hibernate.loader.Loader.doList(Loader.java:2150)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2029)
at org.hibernate.loader.Loader.list(Loader.java:2024)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:369)
at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:300)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:153)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1127)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:749)
at com.bbcs.consumercard.dao.AccountDAO.getAccount(AccountDAO.java:72)
at com.bbcs.consumercard.actions.account.WebPayRequestAction.doExecute(WebPayRequestAction.java:53)
at com.bbcs.consumercard.actions.SecureBaseAction.execute(SecureBaseAction.java:38)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:570)

** END NESTED EXCEPTION **

Possible Solutions

Initially I thought the problem may lie somewhere in our code, but since this was the first time I used C3P0 for my connection pool I decided to Google the problem first. Many of the postings I found related to failures on the part of the connection pool. Based on this post, I ripped out C3P0 and implemented DBCP. That did not fix my problem, though. Some posts say it is a problem with the driver (I do not believe that it is), others blame the c3p0 connection pool (I do not).

Our Solution

In fact, our problem was sitting in our own code and was implemented by someone who was not seasoned in transactions. When you demarcate transactions, you have to explicity commit or rollback every time. Some newbies believe that queries do not have to be committed. They do. If not, the connection is left assigned in case you wanted to manipulate the resultset or do additional reads (there are other reasons, but these are the easiest to wrap your mind around).

So in our case, a transaction was begun and a query was executed. Unfortunately, the transaction was not committed in every circumstance, leaving dangling connections that later got killed by MySQL. C3P0 would later attempt to reattach to the now defunct dangling connection and kaboom - java.net.SocketException Broken PIpe would appear. Finding the spot in the code (it was in some complicated if/else logic) and correcting it solved our problem and we have not had a problem with Hibernate or C3P0 since!

Update: 7 August 2006

In the 3 days since I've posted this, it has already garnered a lot of page views. I didn't realize it would be such a hit. For most of you, it should just be a matter of setting up periodic connection testing. Here are some more pointers to information if you are getting this error:

C3P0: Configuring Connection Testing

Hibernate specific properties for C3P0

More Hibernate specific configuration info

Updated 15 February 2007 - fixed link to "Hibernate Specific Properties for C3P0"