Software has two ingredients: opinions and logic (=programming). The second ingredient is rare and is typically replaced by the first.
I blog about code correctness, maintainability, testability, and functional programming.
This blog does not represent views or opinions of my employer.

Friday, September 12, 2014

I don't like Hibernate/Grails part 7: working on more complex project

Grails big claim to fame is productivity.  A simple CRUD application can be up and running in just a few hours of development work.  Grails makes writing small CRUD application unbelievably easy.
But small simple apps sometimes need to grow up to become bigger and complex.  How easy is to do that with Grails?  Here is my experience with how that works.

In my previous posts I tried to rigorously document framework issues that I believed are not well known by Grails community, some not know at all.  You may find some surprising things in this post too, but instead of providing exact code examples I simply 'ramble' about my experience.  I have grouped this post into sections so you can pick and choose what is of interest to you.
If you find that boring, I am going to have next installment about Unit Testing soon.

My current project:
I am not going to bore you about the details, just a very high level bits and pieces.   I need to put 'more complex' in some context.  As it often happens, my project started relatively simple and the requirements grew more and more complex.

Now, the project enjoys complex domain model with nontrivial business logic (even CRUD is complex).  Some domain objects can be shared between users from one office, some across several offices, some across regions, some across everywhere (add model relationships and this becomes very interesting).  Objects are time dependent, maintain historical information and some relationships can go across points in time (object A at time T1 is in relationship with B at time T2).  Complex uniqueness constraints are in place, with overrides for users with certain roles and certain object types.  Users can view 'deleted' objects... We do some fun algorithmic work too, like programming on graphs and trees.

All of this may sound crazy complicated but it mostly boils down to more complex business logic, need for more complex queries and more complex testing requirements. I keep asking myself:  is Hibernate simply a wrong choice for more complex apps like this?

High level design:  All logic is kept in or behind services.  Services form deep class inheritance trees and use mixins.  Controllers are thin, but use inheritance too.  Domain objects are very anemic (almost no logic).  We had used domain objects with mixins at one point but these have been refactored to use JPA style and class inheritance.  Level 2 cache is used on certain domain objects which are never updated.  GORM criteria queries are used extensively (and almost exclusively) for querying. Some direct Sql is used for performance. Processing of large imports is often split into small hibernate sessions for performance.

Groovy Magic:
I hear that commonly as a complaint,  but I do not really agree.  With improving tool support (such as IntelliJ) introspecting framework code becomes easier and easier.   I think Grails team has done phenomenal job in providing a terse interface to Spring/Hibernate stack.  My problems are typically not with the Groovy visible top but with the whole stack.  Terseness makes things look nice, but then 's..t' is a very terse word too.

Complexity of Grails/Spring/Hibernate stack:
Beast sitting on a beast sitting on a beast.  This stuff is complex.  I did not work with Spring much nor with Hibernate (lucky 'previous' me).  A lot of stuff can be configured but how safe is it? How much configuration can I change in Hibernate to not upset Spring?  How much in Spring to not upset GORM?
For example,  some time ago Spring/Hibernate used to hold on to database connection for the duration of whole HTTP request before returning it to the pool. Our app needs this behavior.  There is a legacy setting for it and which accepts the on_close value (hibernate.connection.release_mode). But it does not work, other settings need to be modified too ... up to the point where we ended up implementing Hibernate interceptors to handle some of the problems. Seem like this configuration ('on_close') is no longer supported by the stack.  Not that we found much documentation about it.  (Please comment on this thread started by Tim and Chris if you have more knowledge about this:!topic/grails-dev-discuss/w3MnxautR1c)

There are some problems reported in JIRA that many developers have experienced but Grails/Spring project developers cannot reproduce.  I imagine there are many that do not get reported at all. One example is the notorious problem with hot reloading of code changes to services.  ( We are seeing this problem.  Other developers that I know are seeing this too.  Reproducing this behavior on a freshly created Grails app seems impossible.  Tim was determined to figure out how to reproduce it 'from scratch' but so far, no luck.

Framework Bugs:
The prize for the most interesting bug ever goes to Chris:  If controller tries to render JSON on detached criteria query (instead of results of that query) the source code is whipped out. Yes, you are reading this correctly, invoking controller method ends up with removing your code. Thank GOD for version control!    Chris has entered bug for it into Grails JIRA, sadly it was never assigned or acknowledged in any way   I have not tried to reproduce it so I do not know if it is still a problem. 
Side Note: Groovy/Grails is in a very good company here!  Once upon a time, a similarly embarrassing bug has happened with GHC (Glasgow Haskell Compiler) which would have erased the source if it did not like some code.
As entertaining (and embarrassing) as this problem was/is, I consider it not as big a deal as, say, some Hibernate 'features'.  The scariest to me is code where adding a not relevant functionality (such as an isolated database query) can cause the code to break or behave differently.  With Hibernate this is not even considered a bug, it is how Hibernate works.

None of the problems that I have discussed in my previous posts have been exclusive to complex apps. I guess, there must be some law which makes the relationship between project complexity and exposure to framework bugs go exponential.  We are seeing a lot of problems now, much more than when the app was 'young and naive'.

Keeping up with Grails releases:
Historically, upgrades had been painful.  Here are some examples of things that got us in trouble: mixins, sending JSON from the client,  ivy, having both a unit and an integration tests.  The project started on  2.0.1.  Release 2.1 had some bugs which where a no-go for us so we waited.  Eventually we decided to bite the bullet and upgrade to 2.3.2.   The cost of that upgrade was somewhere in the 2 developer week range!   Updating to 2.3.3 was easy, but 2.3.4 broke our code again and we decided to wait ...
We are currently in process of upgrading to 2.4.3... and this time it looks like a much easier process, done in about 2-3 developer days.  I hope this means that Grails project is stating to stabilize and next upgrades will get easier and easier.

Update process is important,  just the improvements in Groovy compiler are really making it worth the high cost... but the cost is high.

For reasons that should be apparent from my previous posts, we have reversed the typical testing pyramid:  We have a lot of functional tests,  many integration tests and no unit tests.  Out 'atomic'/unit testing is implemented in Grails Integration Tests. This differentiates my current project form other Grails projects and may explain, in part, why we are seeing problems that others are not.

Can popularity of Hibernate be explained with how developers approach the testing process?  Can Grails bugs be explained by how Grails project tests itself?  Can unit tests be blamed for all of that?

This approach has a drawback:  out tests are slow to run, which impacts the overall productivity.
I will write about testing in Grails in a separate post (or posts).

Fortunately, my project has no high scalability requirements. My problems are more related to the need for our custom graph and tree algorithms to work fast.   Grails and Groovy are working on improving performance, which is great. The biggest headache for me is the cost of the call stack.  Basically the cost of method or closure call in Groovy is much higher than in Java.  
My take on this is that some performance critical parts of the code need to be implemented directly in Java.  Hopefully there will be very few of these.

Concurrency is not something exclusive to bigger projects.  Any web app needs to consider it.
My previous posts have identified some concurrency issues in GORM.  I consider 'repeatable finder' a serious Hibernate design flaw.  I expect the framework to help me with concurrency, not to make things worse by throwing on me code ill designed for handling it.   The worst case is where concurrency problem is thrown at me and I have no way of solving it.   (Again, these issues appear to be inherited from Spring/Hibernate technology stack.)

We have some functional and integration tests for concurrency, but not that many.  What we done differently is:  we used a lot of 'withNewSession' logic in our tests (which is a good thing).  We also have spent some time simply thinking about what are the potential logical problems of how Hibernate session works.    That has exposed two interesting issues (documented in previous posts) that probably nobody else knew about.

It maybe interesting to compare Grails others frameworks.  Ruby on Rails 'share nothing' philosophy offloads handling of complex concurrency issues to the database engine.  The main motivating factors behind 'share nothing', as I understand, are scalability and concurrency.  Active Record is session-less.   Play/Ebean is like this too.  But Play started to move to JPA/Hibernate.  That is a very surprising move for me.   I always viewed Scala community to be more FP-oriented. 

Being able to simply google to find answers is priceless.  Grails documentation is growing too (and has hard time staying current).  However we rarely had any lack with posting questions and getting answers.  The silence about 'repeatable finder problem' which I have posted in 2 JIRAs,  stack overflow and Grails forum is a good example.  Maybe other people just do not see these problems often enough to notice?  It seems like once we passed certain level of complexity in our project we are sailing these waters on our own.

This maybe less true than it was before, but it seems like people get in trouble when they criticize Hibernate.  .. and resolution of such criticism is to label the critic ignorant or stupid.  That really amazes me. So there is this library where just adding a new query can change the behavior or break your previous code and you cannot criticize that?

Here is a conversation between two fictional programmers A and B (it resembles some I have seen on the internet when searching for answers to my Hibernate questions):
A:  I am finding how Hibernate works hard to swallow,  my application is very hard to maintain.
B:  I have used Hibernate in 4 projects and never had a problem,  the problem maybe you.
.. and the discussion stops here

I am very puzzled by all of this. What does B differently than I do?  Less complex projects - is that it?

How did that ever work?
Did you ever experience this?  I am looking at some code, typically a test that is failing and scratching my head how did that ever work?  I have experienced situations where, for example, a test had a local variable declared and defined in one closure that was used in another closure.   This test has been passing for months on developer computers and on Jenkins...  I made some not relevant change and it started to fail because of not defined variable error...

The only logical explanation I can come up with is some incremental compilation issue that is not resolved by cleaning.  That, or .. I am crazy.

Clean, clean-all, super-clean, delete folders  ...
And then sometimes nothing works until you clean, super-clean, restart IntelliJ,  reboot...   Again this is a complex technology stack.

It is improving, syntax detection works better and better. Ability to drill into source code is improving.  Still there are some glitches but considering how complex this must be, it is impressive.
I have used STS in the past but not for some time, so I cannot comment on STS.

Overall Productivity:
As a developer,  I much rather focus on complex business functionality than on figuring out Spring or Hibernate gotchas or concurrency issues.   This is a hard project management problem.  How can we estimate cost of new functionality if resolving gotchas takes significant chunk of our time? What gotchas?  Please look at part1 - part 6 posts on this blog site.

Not being able to trust in unit or integration tests is another big problem.  Functional tests have many benefits, but these tests are slow to run and running them after each code change is not very feasible.  

It would be good to figure out some stats for what percentage of time is spent on dealing with framework related issues. I have no such stats but it feels like Grails/Hibernate does not scale well to more complex projects...  but then, would I prefer to use Spring/Hibernate directly? NO! NO! NO!

Maybe we are seeing more of Spring/Hibernate issues because Grails 'overlay' allows us to spot problems that have been previously overlooked by developers bogged down with writing verbose Java code?   'No good deed ever goes unpunished' , am I blaming Grails framework for making things better?  Quite possibly, I do.

Finally, I think a big lesson from all of this is maybe that Hibernate and 'outside of the box' thinking do not mix.  Keeping things orthodox and simple may help reduce the volume of issues that I and you will see.   My previous projects have been very vanilla like,  I have seen some problems but not enough to write home (or blog) about.  True, I knew less back then on what can go wrong, so there is some ostrichism in this advice, but the likelihood of things going wrong appears to be significantly smaller when things are kept simple.

Next Posts:
I want to take a short break from Grails/Hibernate 'bashing' and write about what absolutely fascinates me:  My dream is to have ability to write code that has no bugs at all. Is that even possible?  (I you read my "I don't like" series, you probably can see why that is my dream.... and yes it is possible!)
I also want to discuss this questions: Is Groovy 'functional', how functional is Grails?

After that: I will talk about challenges and problems I see with testing in Grails.


  1. I understand your frustration. From my observations Grails is a great platform for simple CRUD applications. You just write a few lines of code, magic happens, and you're done. But what if the "magic" doesn't work right? As you've mentioned, the more complex your project gets the more difficult Grails becomes. The documentation is often stale and doesn't delve very deep plus the comunity is rather small, so when I hit a magic problem I often resort to trial and error. As you've noted there are many bugs in the Grails framework, most of them minor, but it seems there's so few people doing advanced Grails programming they aren't work fixing.

    So sure, if your application is small and you need something simple up and running, Grails is great. But if you think your product will grow and expand, expect to do a lot of trailblazing.

  2. domainObject.discard() is probably what you are looking for.

    The domain complexity, and business rules you are describing would be a problem for any framework/language, it has less to do with Grails. Especially, for business rules are specialized frameworks, DROOLS for example. Concurrency? Another huge subject matter that so many frameworks and products try to tackle. Grails is not one of those frameworks. If you have multiple threads and multiple locations working with the same object, Grails will provide the hooks, but it will be up to your team to roll (or integrate) the right solution.

    I think you were infatuated with Grails, originally, because it does what it does so well. That you were disappointed when you realized that it solves a specific problem and wasn't magic after all.

    Hibernate/Gorm simply store stuff for you in a persistent (relational) database, that's it, nothing more nothing else. These frameworks have very little to say about business rules, or concurrency.

    Perhaps the code base is missing some layers, some framework integrations? Trying to codify so much business logic onto relational database constraints seems... lite

  3. So sure, if your application is small and you need something simple up and running, Grails is great. But if you think your product will grow and expand, expect to do a lot of trailblazing.


    So sure when your application is simple, it is simple, and when it gets complex, it gets complex.