Subscribe in a readerFebruary 19, 2008
Thought For the Day
Unfortunately this thought is not swimming, cycling nor running related. But it struck a chord with me as I spend 40 hours a week dealing in the information technology world architecting and building solutions that solve business problems.
Great employees routinely overcome bad or non-existent process, ineffective leadership and governance, and messy technical architecture. Bad employees just as routinely cause even the best process designs to fail while turning elegant architecture into a tangle of spaghetti, and efficient governance into meaningless committee meetings as projects become eternal.
Read more thoughts on the connection between IT process and leadership. Ignore his rant on the CMM. The paper he cites is from 2005 using data from 2001 and takes a very United States centric view. My experience has been that overseas companies, particularly those in emerging economies have widely adopted CMM and the more recent CMMI-DEV. The reason? They don't have 50 years of legacy methodologies / code / people that are all resistant to change. But that's a topic for a different day.
Posted by Mike at 8:00 AM | Comments (0)
January 9, 2008
Questioning CAPTCHA Usability
Since the first interactive form was put on the Internet, bot writers have been writing programs to automatically exploit them. To prevent this, programmers began putting those screwy images of warped letters and numbers on the screen and asking you to key them in. This is known as a turing test, a sort of intelligence test to tell a computer and a human apart.
This works fairly well for those of us who have our eyesight, but what about the visually impaired? Yesterday I sent you to get a $40 coupon toward a digital converter box. If you filled out the coupon, you were challenged with a CAPTCHA code. For the visually impaired, you would click on the speaker icon and hear a code for you to type in like this one garbledCaptcha.mp3.
I listen to that sequence and it just sounds terrible. Perhaps the problem is that I have all my senses, so my hearing is not heightened to make up for vision. I guess that the same exploits that attempt to decode CAPTCHA images are also used to parse these audio files, thus the poor quality and background noise. Still, I don't think I would've gotten this one right.
In the case of the digital tv site, they provided a link to download a paper application to fill out. The visually impaired person would still no doubt need help in filling it out, but at least there were other options presented. Many sites don't offer that capability (think of places like Ticketmaster where the only other option is slow service over the telephone).
Defeating the automated bots is a necessary task, but I think that the evolution needs to continue. Perhaps this would be a good route for a PhD thesis...
Posted by Mike at 8:00 AM | Comments (0)
January 3, 2008
Java is becoming the new Cobol
Wow. Something I've been feeling for the past year or so is showing up in trends amongst other developers. I guess it's like going to these new mega churches that are non-denominational and everyone saying "hey, I used to be <fill in your religion> too!"
What is most interesting to me is that the .Net platform that for 6+ years was not ready for prime time is showing up in the 8pm Thursday night time slot (for you non-US readers, the best tv shows in America generally get slotted to air on Thursday nights between 8pm and 10pm). I have been building my fluency in .Net Web Services, Windows Workflow Foundation and Windows Communication Foundation for some time now and am optimistic about the direction these technologies are headed. They kind of remind me of how I felt when Lotus Domino 4.6 came out.
No, I haven't become a Microsoft fanboy. I'm just an enterprise software architect interested in bringing the widest expanse of knowledge and experience to the table when technology problems are needing to be solved.
Read More ->Java is becoming the new Cobol | InfoWorld | Analysis | 2007-12-28 | By Bill Snyder
Posted by Mike at 4:00 PM | Comments (0)
January 2, 2008
Cannot Find File Created By Java IE Plug-in on Vista
The virtualized directories provided by Vista have been both good and bad to me. One thing that keeps biting me is that the privilege level that Internet Explorer has is less than what I have as a user. Thus, when I download files from the Internet or interact with certain plug-ins, I am actually dealing with a sandbox version of my file system.
Today I was generating some XML files using an IE plug-in that allowed me to connect with various systems around the public and private network. When I saved the file, I was saving it to my default Documents folder (they dropped the redundant "My" in all the folder names for Vista). I even saw the other files that were already in my Documents folder. Alas, when I opened Documents, the file I just saved wasn't there.
What was wierder was when I regenerated the file, it showed up as being there, right next to all the normal files I could see. I was prompted "Overwrite Y/N?". I overwrote it, when to find it and it still wasn't there. I executed an "attrib" command on the folder thinking maybe it was hidden, but it wasn't listed. Then it hit me - the Documents folder I saw in the save dialog was virtualized because I was running the Java plug-in for Internet Explorer. To find the file, I had to navigate to
%userprofile%\AppData\Local\Microsoft\Windows\Temporary Internet Files\Virtualized\C\Users\mschubert\Documents
Maybe now that I've written about it I won't be caught so off guard in 3 weeks when this happens to me yet again.
Posted by Mike at 1:30 PM | Comments (0)
November 3, 2006
A Funny Thing Happened While Debugging Code Today
When testing new code, we often use names of baseball players, football players, etc in order to make it a point that the information we are entering is bogus data. It also gives us an easy way to differentiate test cases and say something like "Pujols can't change his email address." Until yesterday, this had never been an issue for me.
So I get the phone call that "The last three mornings I have had to the clear the cache for [big name star]." No big deal, I think. We've been having an issue with cached transactions ever since we upgraded this application a few months ago. Out of 40,000+ people, it only affects about 9 per month. I thought this was just reporting a new twist on the problem.
I logged into the database to start troubleshooting, found the header record, found the child record causing the problem and was finally able to figure out what has been plaguing us for the past 2 months. If it was the first time a user accessed the system, the flawed child record was not written. However, if a cached header record existed previously, it was purged and a new header and new flawed child record was created. So I put in a patch that would check if cache was pre-existing, check to see if the child record was flawed (the flaw was created by a closed system I have no access to, so I had to work around it), and write things out accordingly.
After putting the patch in, I checked the header for my test user. Wait - this isn't the [big name star] test user I created. [Big name star] was actually a customer -- and he actually was having a problem (or at least his financial manager was). What are the odds on that?
Posted by Mike at 8:09 AM
August 3, 2006
java.net.SocketException Broken Pipe Exception
For the last week, I've been chasing down a randomly occuring broken pipe exception in the banking app we've written. It was like looking for a needle in a haystack.
The Problem
Database connections were being tied up to the length of wait_timeout on the MySQL server. We're using Hibernate for our persistence layer and C3P0 for our connection pool. During initial pilot testing, we only saw issues after a long period of time. My assumption was that the pool was not doing its job and MySQL was killing the connections. However, after ramping up the testing to a larger scale, the exception began occuring much more often.
Here is the dump I was seeing:
10:01:57,812 WARN NewPooledConnection:356 - [c3p0] A PooledConnection that has already signalled a Connection error is still in use!
10:01:57,813 WARN NewPooledConnection:357 - [c3p0] Another error has occurred [ com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:** BEGIN NESTED EXCEPTION **
java.net.SocketException
MESSAGE: Broken pipeSTACKTRACE:
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java(Compiled Code))
at java.net.SocketOutputStream.write(SocketOutputStream.java(Compiled Code))
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java(Inlined Compiled Code))
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java(Inlined Compiled Code))
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java(Compiled Code))
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java(Inlined Compiled Code))
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java(Compiled Code))
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java(Compiled Code))
at com.mysql.jdbc.Connection.execSQL(Connection.java(Compiled Code))
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:936)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1030)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1669)
at org.hibernate.loader.Loader.doQuery(Loader.java:662)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:224)
at org.hibernate.loader.Loader.doList(Loader.java:2150)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2029)
at org.hibernate.loader.Loader.list(Loader.java:2024)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:369)
at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:300)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:153)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1127)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
at org.hibernate.impl.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:749)
at com.bbcs.consumercard.dao.AccountDAO.getAccount(AccountDAO.java:72)
at com.bbcs.consumercard.actions.account.WebPayRequestAction.doExecute(WebPayRequestAction.java:53)
at com.bbcs.consumercard.actions.SecureBaseAction.execute(SecureBaseAction.java:38)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:570)
** END NESTED EXCEPTION **
Possible Solutions
Initially I thought the problem may lie somewhere in our code, but since this was the first time I used C3P0 for my connection pool I decided to Google the problem first. Many of the postings I found related to failures on the part of the connection pool. Based on this post, I ripped out C3P0 and implemented DBCP. That did not fix my problem, though. Some posts say it is a problem with the driver (I do not believe that it is), others blame the c3p0 connection pool (I do not).
Our Solution
In fact, our problem was sitting in our own code and was implemented by someone who was not seasoned in transactions. When you demarcate transactions, you have to explicity commit or rollback every time. Some newbies believe that queries do not have to be committed. They do. If not, the connection is left assigned in case you wanted to manipulate the resultset or do additional reads (there are other reasons, but these are the easiest to wrap your mind around).
So in our case, a transaction was begun and a query was executed. Unfortunately, the transaction was not committed in every circumstance, leaving dangling connections that later got killed by MySQL. C3P0 would later attempt to reattach to the now defunct dangling connection and kaboom - java.net.SocketException Broken PIpe would appear. Finding the spot in the code (it was in some complicated if/else logic) and correcting it solved our problem and we have not had a problem with Hibernate or C3P0 since!
Update: 7 August 2006
In the 3 days since I've posted this, it has already garnered a lot of page views. I didn't realize it would be such a hit. For most of you, it should just be a matter of setting up periodic connection testing. Here are some more pointers to information if you are getting this error:
C3P0: Configuring Connection Testing
Hibernate specific properties for C3P0
More Hibernate specific configuration info
Updated 15 February 2007 - fixed link to "Hibernate Specific Properties for C3P0"
Posted by Mike at 3:30 PM | Comments (12)
December 16, 2005
Lotus Notes Tip of the Day - Scheduled Agents Don't Run in .ntf Files on R6+
"Based on feedback from customers, agents will no longer run in templates with the NTF extension. (They will, however, continue to run in templates with NSF extensions.)"
I'm sure I was one who complained about this. The problem as a consultant is that you are often asked to fix problems that you did not create. In the glory days, if a scheduled agent wasn't running either the person who signed it did not have rights to run scheduled agents on the server, or they were trying to do something exotic in the code that would break a security rule. When those two didn't hold true yesterday, I had to reach into the inner recesses of my brain to remember to check the file type.
So now I am writing it down here in hopes of remembering this in the future. Here is the article that details all of the changes that came about in R6. Haven't seen it for R7 yet. Here is a good article on troubleshooting Notes 5 and Notes 6 agents.
Posted by Mike at 8:30 AM
September 20, 2005
It Needs To Do What It Does Today
Mark Cuban had a post recently about doing things simply because that's the way things have always been done. I've been thinking a lot about some of the project kickoff and requirements meetings I have had for system upgrades over the years and one of the phrases that has always stuck out at me is "It needs to do what it does today."
Exactly what is that? In some cases, companies are able to produce documentation that shows the intentionality of a system from its conception all the way through every feature change and bug fix report. That is really the exception, though. Even as a system expert for many systems, I'm not entirely sure what some of these things "do today." My exposure to them as a software architect is generally in terms of what they don't do today or, more precisely, what they don't do well today.
Companies of all sizes would be best advised to assign software owners within their business units. The knee-jerk reaction to that is to say that IT needs to own it. IT should only own the plumbing, though. Things like the hardware (PCs, servers, network, switches, etc) and commodity software (productivity suite
I see a future where the IT department isn't a silo sitting off on the horizon. In particular, I see developers within departments other than IT. They undoubtedly still report to Software Development Managers and up the chain within the IT department, but the organization chart will probably start to show dotted lines across to another department as well. Bringing a sense of system ownership within other departments is the first step towards bridging the gap between the geeks and those that run the business. And isn't system ownership what the American Dream is all about?
As for my response to people who say "well, it needs to do what it does today" ... "Great - we're already finished."
Posted by Mike at 7:45 AM
June 22, 2005
Fewer Roundtrips to the Server
Gary Potter points us to an open source project Sabre has opened up called Rico. Rico currently consists of a couple of javascript libraries that enable drag&drop functionality, some cool effects, and some interesting routines that access server data without requiring a round-trip (post to server then reload of the same page with updated data). The technology is all Ajax - asynchronous javascript and XML, so all my friends, whether in java, .net, or domino, can take advantage of it.
Check out the demos and see what inspires you.
And remember:
Round-trips to Hawaii - Good
Round-trips to the server - Bad
Posted by Mike at 8:21 AM