REST 4: Web App with some REST on a side

Probably my last post of the series inspired by the RESTfulweb services book.   This post is about these very few topics where I have problem in fully agreeing with the authors. 

JavaScript Ajax

More and more of web apps are Java Script heavy and end up moving data back and forth with Ajax.  The book suggest this to be a great opportunity for REST.  I have some skepticism.

I can be persuaded that SPA (single page load - applications with only one HTML page sporting an empty page body, and lots of JavaScript) are more likely to be implemented with true REST.   (Authors call these type of projects Ajax projects and make a point that these are really true Web Services under the hood.)   But I am still skeptical.

Web application development tends to focus on ... well  on the user interface development.  Unless project has explicit goal to create web service interface easily accessible by the 'rest' of the world,  the implementation will end up a REST-RPC hybrid at best.

I am probably repeating myself here. My point is that REST is not on the path of least resistance for Web App development.  Simply put: in today's task focused development a true REST will not make into the story line.  As long as JavaScript gets the data it needs and sever gets the data it needs from JavaScript the development goals are satisfied.  REST as part of Ajax will most likely only mean using GET, POST, PUT, DELETE verbs.  The data will likely be send in JSON format and will contain whatever is needed by the UI: and not a meaningful representation of a resource. The URLs will be constructed by JavaScript and will likely use ? and database IDs.  status codes 500 and 200 only (well maybe 404 and 401 but that will be about it) ...  

Grails withFormat and Rails respond_to

It seems that both Grails and Rails made it not very easy for the programmer to separate controllers serving the same URL base:  to have one for serving REST and a twin for serving web pages.

The seemingly easiest and most tempting alternative is to use withFormat (Grails) or respond_to (Rails) blocks of code.  The book seems to approve of the Rails respond_to method (Grails is not covered).
This is tempting because you can do stuff like this:

    def list() {
        def books = Book.list()

        withFormat {
            html { [bookList:books] } 
            json { render books as JSON } 
            xml { render books as XML }
        }
    }

I do not like this approach.  Here is why:

If your web application is serving HTML pages to the user the controller is already busy doing what is it supposed to be doing in the MVC web framework: flowing the pages (we are not talking Ajax here).

Page flow and serving REST uniform interface are 2 different problems and have very little in common.

Separation of Concerns?: Ironically, my first post about REST compared it to OO.

Let me rephrase this: in an MVC web application serving web pages it is the Controller's job is to decide what the user will see next.  In a RESTful Web Service it is the client responsibility to decide what to do or see next.  If there is a user sitting behind a REST client (it does not have to be) the UI presentation flow is the REST client responsibility. The difference is that of push vs pull.

Using the same controller for flowing pages and serving REST is mixing apples and oranges.

So my take on this is:  If you are writing a RESTfull Web Service and are using Grails or Rails than using withFormat/respond_to is a great idea.  You are serving different representations of the same thing and it is convenient to see the rendering logic for all of these representations in one place.

If you are adding REST on a side of a Web app, then keeping your REST logic together with page flow logic is a bad idea.

Concluding Remarks:

Reading the book was overall a great learning experience for me.  Hopefully I have managed to stir some unREST into my coworkers and friends with these posts.

I am not a REST expert.  I have still a lot to learn.  I am looking forward to discussing this stuff.

My next step should be to read up on semantic web but this will need to wait a bit ... I got myself a haskell book...

REST (3): why NOT easy: the details

Continuation of the previous post motivated by RESTful WebServices book.

In the previous post I have tried to identify the elements of REST and pointed out general problems around software development of RESTful web services.  I am considering REST based on the HTTP protocol only. 
REST is important concept and can yield many benefits to applications which decide to be strictly RESTful.

Here is a more low level detailed list of REST things which are not very easy to do.  (It is my private what I can screw up (or I did screw up) and where I should be careful list):

URLS

• Constructing URL is NOT client job:  I believe we will see very few applications where this is met and it is obviously hard to achieve.  It requires excellent 'connectedness'. Designs where client needs to know how to construct URLs are not RESTful.
• ? Restrictions:  Except for few spelled out exceptional cases using ? is not fashionable any more.  Unfortunatelly ?-marks are so easy to use:  just throw some parameters in and you done, you never have to rethink you URLs, you can simply add stuff to them to piggyback new functionality.  If you do that it is not RESTful.
• No Database IDs:  Rails with ActiveRecord and Grails with GORM make using IDs easy.  These are not very useful especially when the service is not well connected. Imagine a fantasy Active Directory exposing REST interface, you want to get a phone number for jsmith.  Would you prefer to issue a GET to: .../users/jsmith  or  .../users/2001234?   Frankly, the second URL format is useless, yet you will find it in most Rails or Grails projects which claim RESTfulness and in fact this is what you get if you use Rails REST scaffold.
• Lack of support for templates:  (example: /users/{userId}/posts/{topic}).  Few tools support templates.  GRAILS has UrlMapping file, but there are limitation to it as well.  I believe Rails, as I understand it, is more limited than Grails.
• No Verbs please:  This may require extra effort and even rethinking of what are your resources.  URLs represent resources, so URLs containing verbs (.../myserver/mydatabase/runDefragmenter) are not RESTful.  How about POSTing to: .../myserver/mydatabase/defragjobs instead?  API-like thinking is very much ingrained in us.   API and REST should not be used in the same sentence (unless you think of the uniform interface).

Uniform Interface

HTTP protocol is the base for achieving the uniformity.  This goes beyond the 4 (or 6, or 8) HTTP verbs and requires understanding of the HTTP protocol itself with advanced use of headers, status codes, etc.  Defining HTTP protocol and explaining how it applies to REST is not what I want to do here.  I just want to point out that RESTful is not RESTful without it.  Here are some examples.

• Status Codes:  Example 1: consider a PUT to a URL representing a resource with unique constraint on its name.  The PUT tries to rename the resource but a resource with the new name already exists.  I bet many apps will simply return status 500.  Correct code should be: 409 (Conflict).   

Example 2:  As above, consider a PUT request to rename a resource and assume that resource name is part of the URL.  I bet may web services will simply return 200 and a new representation leaving to the client to 'construct' the new URL.  This is not optimal use of HTTP protocol and is not RESTful.

• Other Header Stuff:  There is a bit to learn there and this will probably be overlooked by many developers writing RESTful apps.  This could create lack of consistency. 

• Uniform Interface vs Serving Web Pages to the users  Typical 'MVC' web frameworks have you focus on flow of web pages.  Programming REST Uniform Interface is different.  Making both pieces of logic coexist is harder than it looks.  I will explain more in the next post.

Formatting resource representations

Standard Interface helps but its benefits are less inspiring if REST client coded by someone else has no way of understanding the content.

Coming up with a standard vocabulary for your problem domain may be hard, one may simply not exist.  Adopting something like Atom Feeds or XHTML may seem artificial or awkward (yet often are the best choices you have).

Grails or Rails make it easy to spill out XML or JSON, but this will not be any standard vocabulary and your resources will not be connected.

Even much simpler problem of assuring that error messages are served in a format expected by the clients are often surprisingly easy to overlook and may not so easy to implement.

Statelessness.

REST has no place for cookies, session objects, etc. Simply put: don't claim your app is RESTful if you store server state between HTTP requests.  Hypermedia is the engine of application state in REST.

I may be opinionated on this but to me it is not just about REST:  if you programming web app and use session object, you should rethink your design.  But then since people use server side state on web apps a lot, they must be finding this technique convenient and the benefits of being stateless not worth the extra effort to them. 

My personal opinion is that a possible exception to this rule is use of some standard security infrastructure (like Spring Security) which may have stateful implementation and you have no control over it.  This still would violate REST, but you may have not much choice.

POST is the worst of the 6 verbs.

I you can avoid using it you should.  But that is obviously not easy.

I am not elaborating on this one ... go and read the book ;)

Connectedness.

If the client has to construct URLs then the server is not RESTful.  Ability to discover URLs is what differentiates REST from API.

I wrote a bit about it in my previous post.  This is the hardest thing to do right and will impact your decisions on representation formats. This topic is out of scope for this post.

My next post will focus on the coexistence of Web App and REST

REST (2): why NOT easy

Continuation of the previous post motivated by my reading of the RESTful Web Services book.

This post is a high level overview of the topic.  In my next posts I will provide more concrete examples of why REST is not so easy.  I am talking about REST in the context of HTTP protocol only.  I assume the reader has basic familiarity what are the 6 HTTP verbs (TRACE and CONNECT do not count ;) and what REST is or at least what is typically painted to be.

When you develop RESTful web services you will be dealing with

• Resources (think: flight reservation as a resource example)
• URLs (which are the resource addresses, think:  http://.../{userName}/travel/reservations/flights/{tripName})
• Resource Representations (for example some XML vocabulary,  think: XHTML document describing flight reservation, seat assignments, etc)

Thus, designing your web service application you will need to think about:

• How to define URLs (some ways are more RESTful than others)  
• What standards to use to define resource representations (some standards are more RESTful than others)

In addition, the book identifies following properties which make things RESTful (or as the book calls them ROA-Resource Oriented): 

• Addressability (measure of how well you can access your resources with URLs, ideally each resource should have a URL) 
• Uniform Interface (requires that the web service exposes subset of the 6 HTTP verbs and uses HTTP protocol itself to manage resources,  think: GET gets current flight reservation, PUT makes changes to it, DELETE deletes it,  think of using HTTP header and status codes) 
• Statelessness (no state on the server, state is in hypermedia)
• Connectedness (measure how well are your resource representations linked,  think: link or a tags linking to user account with list of other reservations this user has made, link to the airline flight itself, link or form to a ticket resource (to buy the ticket for this reservation), maybe a form tag defining what can be changed using PUT, etc) 

The first expense of adding REST to your project will be some design time:  (1) split your application into many resources, (2) create URLs for all the resources,  (3) map your problem domain to the Uniform Interface (that includes using standard status codes,  user of more advanced HTTP header stuff),  (4) design your representations (make formatting decisions), (5) connect the resources with some type of hypermedia.

There are 4 big problems categories which make it hard:

PROBLEM 1:  Tool Support: 

Most tools including Grails or Rails are designed to do something else with ease (to quickly create web apps) and the simplifying decisions which made them successful also makes doing other things (like creating a RESTful web service) hard.  The book lamented about Rails (Grails) has not been covered.

If you are using tools designed for REST (like restlet) this point does not apply.

PROBLEM 2:  Finding Standard Vocabulary for Resource Representation:
Some form of XML seem to be best bet, but a vocabulary designed for your problem domain simply may not exist.  Using XHTML maybe a good option.  There may be some relevant microformats for your problem domain.

PROBLEM 3:  How to implement Connectedness: 

There are very few tools or even standards which could help you to connect your resources.  Connectedness may be the most important part of being RESTful and is the hardest to implement.  The point is that without it REST it is just another API only with fewer verbs and many, many nouns. Sure there is a benefit of not needing to know which verb to use (you have to when using APIs), but there is a tradeoff of having to know many nouns (URLs).   Connectedness implies that REST client can discover the URLs and it does not need to know them and should not be constructing them!

Here is where semantic web becomes important.  Things like microformats, RDF, etc.
XHTML has link and a tags and forms (HTML 5 forms get better).  Atom feeds/APP, etc also define cross linking between resources.  Microformats play a big role in qualifying the 'connections' between resources. 

Future RESTful clients will be able to semantically know the links.

HTTP protocol itself can help in connectedness a bit with its location header and various status codes: 201 (Created), and many in the 3xx (Redirection) range.  But mostly connecting resources will be the job of your resource representations.  No free lunch here. To connect your resources some more serious work needs to be done and you will be looking at exposing you data trough things like Atom feeds.

PROBLEM 4:  Confusion: 

Remember the big picture from my last post?:  APIs are all different, REST is trying to be the same.  The uniformity will not happen if everyone is confused about what REST is, yet claims to support it.

REST is often understood as simply some level of support for the 4 verbs in HTTP (GET, POST, PUT and DELETE).  I maybe repeated myself here, but this trivialized view is demonstrated by most frameworks (including Ext JS, Rails, Grails). Adopting this simplified view point creates confusion and probably impedes support for more meaningful REST functionality these frameworks could provide.

Example:  RESTful JSON. JSON is not connectable (at least not in any standard way), so how can web service be truly RESTful if it is serving JSON only?

This was my high level overview.  Next post will deal with more concrete examples of why it is hard to REST (and maybe easier to stay BUSY ;)

REST (1): misunderstood and important, how and why

I want to take a break from declarative/curly posts and write a bit about REST. I hope this will help spawn some discussion among developers I work with and know.
I am on last pages of (the rightfully acclaimed) RESTful web services, by Richardson and Ruby book.  The reading felt a bit repetitive, it felt like hammering of ideas into my head, but they probably needed hammering.  Overall a great book, one that opened my mind to what REST is about (or should be about).

Somewhere half way through the book things started to click.   (I am trying to share some of these clicks in this and in a next few posts.)  I read quite a bit about REST before reaching for the book.  The first question is why did it take so long to click?  

I came out of this reading with a much better understanding of the HTTP protocol itself, so this must be a part of the learning curve.  But, I also blame my slowness on two things: first reason is the amount of confusion surrounding REST,  second reason is harder to explain:  REST is somewhat complex if you look at it as a set of implementation guidelines (the how),  it is also quite straightforward if you look at it as what it is for (the why).   It is hard to understand the how until you get the why.

Developers like to focus on the mechanics of things (the how).  Developers also have deadlines and work in a task-oriented environment where there maybe many whys but the goals of Uniform Interface, ease of future integration and benefits of semantic linking are probably not likely to be among the whys which pay their paychecks.  REST become such a buzzword that it ends up being used even if the why is not on the radar.  REST gets half-implemented but holds on to its full name. This fuels confusion.

REST Confused: 

The term REST is used a lot these days, I hear it at work,  I see a lot about it on the web.  Many developers are very interested in it and know a lot about it (way more than I do).  Last year's Uberconf REST offerings were very well attended.  At the same time, there is quite a bit of misunderstanding and confusion about what REST is.  (I was among the confused before reading this book.)  And I think that the industry-wide confusion on this topic will be winning.
The book authors decided to stay away from the REST term most of the time and have used the term ROA (Resource Oriented) to avoid the confusion.  (I stay with the terms REST and RESTful.)   
The book also introduced the therm REST-RPC Hybrid to describe many existing web services and to differentiate them from the true REST.  There is quite a bit of wit in this self-contradicting term.

It is just like Object Oriented
A somewhat forgotten old paper by Tim Rentsch (back in 1982) included a prophecy about what OO will become:  "Every manufacturer will promote his products as supporting it. Every manager will pay lip service to it. Every programmer will practice it (differently). And no one will know just what it is."

One more old OO related quote stays in my brain and does not want to leave:  I was at some sort of a conference and talking to vendors in the booth area.  One of them (he knew he is talking to a developer) was doing his spiel and said this: '... and we will be object oriented by the end of the next quarter'.   I am not trying to contribute ideas for a next Dilbert episode.  The scary thing is: that does not sound like something the marketing group would came up with on their own. I fear it came from the developers. 

I think REST became like OO from these last 2 quotes. Part of the problem is that, like OO, everyone wants to claim it, yet it is not so easy to implement (my next posts will talk more why it is hard) but the lip service is easy so the term is used a lot.

Here is a quote from the above book: "Both REST and web services have become buzzwords. They are chic and fashionable. These terms are artfully woven into PowerPoint presentations by people who have no real understanding of the subject".

Side NOTE:  a similarity between REST and OO:  both are often used as check boxes and both are really progress bars.

THE WHY:
I believe the fact that the term REST is misused and misunderstood causes lack of its true adoption.  We are missing out on some amazing opportunities.  REST could be a big step towards fulfilling a fantasy in which computer programs can surf the web the way humans do today.

Why REST is such a good idea?:  REST is the opposite of API.  APIs are all different, REST services are trying to be all the same.  

Let me quote the book:

"One alternative to explaining everything is to make your service look like other services.  If all services exposed the same representation formats, and mapped URIs to resources in the same way... well, we can't get rid of client programming altogether, but clients could work at higher level than HTTP."

I came up with this imprecise, sort-of definition of REST focusing on the why:  You wrote a very RESTful app if you have exposed your app functionality to the programmable web without API specification and without ambiguity.

I think the problem is that people think that REST is all about using GET, PUT, POST, DELETE (maybe OPTIONS, HEAD), some will know that POST as it stands today is better not used, but that is where it often ends.  We think of the mechanics and loose the big picture.  The net result is that we use the term REST a lot,  but what we develop typically has nothing to do with being easily accessible by the 'rest' of the programmable web.   

Example:

Say, you are developing a web application serving web pages and need to expose Ajax data from server to your Java Script library in a few places (just a GET request for simplicity). 

JSON is not very RESTful (point one: developers should be aware of the fact) but, if you decide on using JSON, still ask yourself these questions: 

• How cryptic is the URL, does it look anything like a location of a resource?  Any database IDs in it? (Would you expect external REST client to know your database IDs?)
• Can someone not very familiar with this project get any information from your JSON? 
• How much is this JSON specific to the current version of the UI:
• Is it just the fields that you need or a more meaningful representation? 
• Are you pre-formatting the data for your UI on the server?
• How is your error handling and status codes?  (say, you listing employees in a department and someone will issue a URL GET with invalid department: will your service just return 500 with the error message in plain text? - hey you know your JavaScript would never do that so why care about correct status codes or consistent content type?...).
• And finally how can another developer discover this URL?  Do you expect them to simply construct the URLs?

Lots of REST is simply common sense if you understand the why.

We are missing on big opportunities:

The first reason is so other developers can access/integrate easily with your application. 

The second reason is so that the future semantic web will find the data exposed by your app.

Such goal should be very important for businesses which want their products to be 'known' or educational institutions, organizations defining standards, many government institutions which want their data to be transparent and available to programs at large who may want to use it, study it, analyze it. 

These 2 reasons should create a big push for writing RESTful apps, but the reality is the exact opposite. Today businesses run hundreds of applications, these applications do not know how talk to each other.  Any integration is complex and expensive.  New applications are being developed (I participated in developing some of them) and REST is last thing on anyone's mind. 

The targets of semantic web today are not government, science, or even manufactured goods (although, Best Buy made recent news to the contrary), but tweeter, facebook, blogger, etc.  Who likes whom on facebook will benefit from all of this,  but is there any semantic web development for, say, endangered species?  This is not to criticize social web, kudos to them for innovating, this is to criticize everyone else who should be part of this and decides to stay behind.

Next post will have more mechanics:

I think a big part of the problem is that good REST is not so easy to implement. If it is hard and it is a buzzword people are bound to overuse it.  Buzzwords loose their meaning very fast, just like OO term did.

My next post will be about what is so hard about REST.