Software has two ingredients: opinions and logic (=programming). The second ingredient is rare and is typically replaced by the first.
I blog about code correctness, maintainability, testability, and functional programming.
This blog does not represent views or opinions of my employer.

Sunday, August 3, 2014

I don't like Hibernate/Grails, part 2, repeatable finder problem: trust in nothing!

I was hoping for some 'inalienable truths' developer can rely on.... Like, that things that should obviously return true never return false.
(A more correct technical term for this is property but I find inalienable truth more fun).

The assert code from last post is one such example:
    assert BankAccount.findAllByBranch(myBranch).every{
    it.branch == myBranch
  }

Repeatable finders:
(I use this term as a reverse analogy to non repeatable reads.)  In Hibernate/GORM the above assertion does not need to be true.  For each finder/getter hibernate stores returned objects in its session and when next finder/getter is called hibernate will use the stored objects whenever it can.  It will not refresh them.   So you have to assume that any finder will return some (or many) of the domain objects found by previous finders.  What if something has happened that hibernate session does not know about between the time of your current finder and the time previous finders ran?

So here is one example showing how to break the above BankBranch assert:

(assume ac1.branch == branch2)

... HTTP request for User1:
... 'componentA' executes:
println 'Tracing something about '+BankAccount.findByName('ac1')

... some other expensive computation executes

... HTTP request for User2:
def acc = BankAccount.findByName('ac1')
acc.branch = branch1
acc.save(...)

... HTTP request for User1 continues:
... 'componentB' executes:
def myBranchAccounts = BankAccount.findAllByBranch(branch1) 

(myBranchAccounts includes ac1 but Hibernate returns old, not refreshed version of it so ac1.branch == branch2 is still true)

... myBrancheAccounts are rendered on a view page

(User1 is presented a list of all accounts from branch1 including ac1 which is jolly displayed showing branch2. User1 is surprised.)

This is not necessarily a strict concurrency problem. You may have code which bypasses Hibernate (maybe uses Groovy.Sql class directly) and get into very similar issues.

It is also interesting to think about compoentA and componentB code from the point of view of the unit test coverage leak problem I described in the 'part 1' post.

Here are 2 other inalienable truths (properties) that are no longer:
Uniquness constraints:
My BankAccount was declared with unique constraint on the name field (database enforced uniqueness on the name column). So if I do this:

  def accounts = BankAccount.findAll()

I will never see the same name repeated, right?  Wrong.  Here is a concurrent usage that shows how that breaks:

...HTTP request for User1:
... 'componentA' executes:
println 'Tracing something about ' +BankAccount.findByName('ac1')

... some other expensive computation executes

...HTTP request for User2:
def ac1 = BankAccount.findByName('ac1')
def ac2 = BankAccount.findByName('ac2')
ac1.name = 'ac1_b'
ac1.save(...)
ac2.name = 'ac1'
ac2.save(...)

... HTTP request for User1 continues:
... 'componentB' executes:
def allAccounts = BankAccount.findAll() 

(allAccounts contain old amount in ac1 with ac1.name == 'ac1'
and ac2 with ac2.name == 'ac1')

... allAccounts are are rendered on a view page 

(User1 is presented a list of all accounts and account 'ac1' shows up twice. User1 is upset)

You may find it unrealistic that User 2 can perform 2 renames concurrently to a short period between 2 finder calls in one HTTP request.  What if there have been 2 users renaming one object each?  In any rate there are other possible domain objects than Bank Account and other fields that may need to have uniqueness constraint.   I think the issue is demonstrated well enough.

Again, you can get into similar problems if you use some direct Groovy.Sql.

Results which look like uncommitted reads:
If I transfer money between accounts inside a transaction I should never ever see the transfer applied to one account and not the other.  Right?  Wrong again:

...HTTP request for User1:
... 'componentA' executes:
println 'Tracing something about ' +BankAccount.findByName('ac1')

... some other expensive computation executes

... HTTP request for User2:
def ac1 = BankAccount.findByName('ac1')
def ac2 = BankAccount.findByName('ac2')
transferMoney(ac1, ac2, 1000) //transfer 1000 dollars

...HTTP request for User1 continues:
def allAccounts = BankAccount.findAll() 

(allAccounts contain old amount in ac1 but new amount in ac2)

... allAccounts are are rendered on a view page 

(User1 is presented a list of all accounts and the list looks inconsistent.  Where did the 1000 go? User1 is software tester, he/she is now furious, spends hours figuring out what got wrong and the problem magically just disappears. )

And again just use direct Groovy.Sql to get into the same issue without concurrency.

Why Hibernate Works Like This?
I imagine that it must have been tempting to use single Java object to represent single record. This is also purist approach to ORM:  node.children.first().parent.is(node). But with hibernate this may have not been just a temptation.  Hibernate designers decided at some point that Hibernate will be saving objects attached to the session automatically. I imagine that it would be very hard to deal with auto saving if you had more than one domain object representing the same record (which one would you save?)
So why not refresh existing object each time it is retrieved?  Maybe because that would be a serious side-effect ;)   If some objects have been changed locally and also concurrently changed in the database:  have Hibernate designers been concerned about throwing ConcurrentModificationException from a finder?
Well I do not see why, because Hibernate finders already save objects and you are likely to get a collection of interesting save errors when calling a finder.  (Talk about aversion to side-effects!)

Can I be just careful?
Be careful not to pollute hibernate session - that may not be so easy.  For example, consider that the BankBranch class has something like featuredAccount association back to BankAccount.  If that gets eagerly loaded at the time branch1 is retrieved hibernate session is already polluted with one BankAccount at the onset of the first example (without any artificial println statements).

More complex applications may want to do some HTTP filter before and after logic which uses hibernate objects.  Applications may have layers of complexity which share the same hibernate session.  Controlling what is and what is not in that session is unrealistic.

Why is this bad?
Developers experienced with relational databases are likely to expect certain behaviors from the code that with GORM/Hibernate are just gone.  The unexpected behavior may be very intermittent and impossible to troubleshoot.  In my example I have 'broke' the code by inserting a println statement printing something about one account. What if finder 'polluting' hibernate session was executed only under some (rare) conditions?
I think developers tend to think of finders and getters as safe methods to call.   Almost always getters do not mutate anything.  With Hibernate getter/finder have side-effects, one of them is mutated Hibernate session and this is easy to forget when you design and code your application.

I believe that any nontrivial Grails app most likely has issues/bugs related to repeatable finder.
In addition, Hibernate offers NO public API to query what is stored on the session.  So if you think of some programmatic ways to solve this issue think some serious hacking.

How to test for this:
Repeated finder problems arising from the use of direct SQL or use of several hibernate sessions within single HTTP request can be discovered by both integration and functional tests.
The issue is not unit testable even if you think of Unit Tests as 'atomic tests' which are implemented as Grails integration tests.
Unfortunately, in many cases the issue will be triggered by concurrent access to your application. 
Testing concurrency issues is always not trivial.  So I do not really know how to answer this question. 
Ideally, the tools we use will be better designed to handle concurrency (have you read Simon Peyton Jones 'Beautiful Concurrency' http://research.microsoft.com/en-us/um/people/simonpj/papers/stm/?). I am afraid Hibernate maybe never be one of these tools.  

New Session a Solution?
There is one very tempting partial solution to this.
If you really need consistent results from a finder keep it in isolated Hibernate session.(LazyInitalizationException alert flashing, OK I promise not to talk about LazyInitalization. :)
The idea is that if this line (from first example):

def myBrachAccounts = BankAccount.findAllByBranch(branch1) 

runs in a new (and therefore unpolluted) hibernate session the domain objects returned from the finder will all have the most recent values and none of the surprising things will ever happen.
Or ...
Enter next Hibernate gotchas:  DuplicateKeyException (and friends) the subject of my next post.

My current thinking is that using new session on certain 'crucial' selects seems to be a very good option to reduce the impact of repeatable finder problem. This technique could work well, especially if the finder we want to protect is the very last hibernate call in the HTTP request. I will not solve the problem but may reduce its impact. I may return to this thought later.

I believe there is currently no full solution to repeatable finder problem described in this post.

Added 2014/08/12:  Thinking more about impacts:
I find it helpful to think about application code from the point of view of properties (logical consistency rules that need to be true).  Application may rely on such properties explicitly (for example your logic will exception or return incorrect results if name property is not unique), implicitly (it will be embarrassing to show user list of items with seemingly broken uniqueness), and can actually be enforcing such properties (for example Grails code is used to enforce some more complex uniquness rules).

There are many, many possible properties related to domain objects (we have seen only 3 in this post).  Some are derived from software requirements but many have a much lower level nature. For example, each 'where' or 'join' criteria in underlying SQL statements is likely to imply a property (as shown by BankAccount.findByBranch() example).
Some properties are meant to be enforced by application code, some by DB, some by the framework.
Repeatable finder is likely to affect/invalidate large portion of such properties in your application!
And you cannot rely on the fact that even DB enforced property will hold.
The impact seems very big.

(EDITED 2014/09/13) References:  
I have posted a question about this issue here: Stack Overflow Question
I have now posted one (terrible) solution to how one can identify and fix such problems as answer to the Stack Overflow Question.
Here is Grails JIRA ticket for it: http://jira.grails.org/browse/GRAILS-11645
another relevant JIRA: http://jira.grails.org/browse/GRAILS-11644
Forum:  http://groups.google.com/forum/#!topic/grails-dev-discuss/wzekMGC0ibE
Hibernate JIRA:  https://hibernate.atlassian.net/browse/HHH-9367

352 comments:

  1. For some reason I've been under the impression that dynamic finders (findBy*, findAllBy*) do not make use of the instance cache, unless cached queries has been enabled (which is discouraged).

    (This is one reason given to make use of findById() vs get(), to avoid the cached instance.)

    I don't know about what findAll() does regarding the caches but in practical terms I've never made use of this method. (When would one get ALL bank accounts with no filter clauses.)

    Have I lost it?

    ReplyDelete
    Replies
    1. I have based this post on test cases I wrote using GRAILS 2.3.3. The configuration is:

      cache.use_second_level_cache = false
      cache.use_query_cache = false

      What you describe was my understanding too.

      I agree that findAll is not realistic, but you will get the same problem with any finder or criteria query. For example findAllByBranch().

      The test cases are easy enough to create.

      Delete
    2. I have also tested this behavior on 2.4.3 (which is the currently the latest).

      Delete
  2. second_level caching is a different issue... where data get cached across sessions. session level caching is just something that happens as part of hibernate, not much developer control over it.

    ReplyDelete
  3. Hi Robert, Will you please zip up your grails application and post it? Run: grails bug-report

    ReplyDelete
  4. Thanks For Sharing Your Information , Plaese Keep Updating US, Time Went On Reading The Article
    Advanced Java Online CourseAdvanced Java Course In Hyderabad

    ReplyDelete
  5. Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.

    CEH Training In Hyderbad

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    AWS training in chennai

    ReplyDelete
  8. Wonderful article, very useful and well explanation. Your post is extremely incredible. I will refer this to my candidates...
    python Training institute in Chennai
    python Training institute in Bangalore

    ReplyDelete
  9. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    AWS Training in Bangalore

    ReplyDelete
  10. Really very nice blog information for this one and more technical skills are improve,i like that kind of post.
    Java training in Chennai

    Java training in Bangalore

    ReplyDelete
  11. My developer is trying to convince me to move to .net from PHP. I have always disliked the idea because of the expenses. But he’s trying none the less. I’ve been using Movable-type on several websites for about a year and am anxious about switching to another platform. I have heard great things about blogengine.net. Is there a way I can transfer all my Word Press posts into it? Any help would be appreciated.
    Data Science training in Chennai
    Data Science training in bangalore
    Data Science training institute in bangalore

    ReplyDelete
  12. Nice Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.
    Check out : best hadoop certification in chennai
    best hadoop training institute in chennai with placement
    best bigdata hadoop training in chennai
    big data training in chennai velachery

    ReplyDelete
  13. Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.

    Hadoop Interview Questions and Answers


    Hyperion Interview Questions and Answers


    Informatica Interview Questions and Answers

    ReplyDelete
  14. Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here.data science course in dubai

    ReplyDelete
  15. Nice information, valuable and excellent post, thanks to offer such a helpful information here.

    Data Science in Bangalore

    ReplyDelete
  16. Attend the Best Python training Courses in Bangalore From ExcelR. Practical PythonTraining Sessions with Assured Placement From Excelr Solutions.

    python training in bangalore

    ReplyDelete
  17. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
    date analytics certification training courses

    ReplyDelete
  18. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.
    what are solar panel and how to select best one
    learn about iphone X
    top 7 best washing machine
    iphone XR vs XS max
    Samsung a90



    ReplyDelete
  19. Thank you for sharing wonderful information with us to get some idea about that content.
    Microservices Online Training
    Microservices Training in Hyderabad

    ReplyDelete
  20. Si el agua cae al lago, desaparecerá( phụ kiện tủ bếp ). Pero si cae a la hoja de( phụ kiện tủ áo ) loto, brillará como una joya. Caer igual pero( thùng gạo thông minh ) estar con alguien es importante.

    ReplyDelete
  21. I don’t think many of websites provide this type of information.
    Data Science Course in Pune

    ReplyDelete

  22. Its as if you had a great grasp on the subject matter, but you forgot to include your readers. Perhaps you should think about this from more than one angle.
    www.technewworld.in
    How to Start A blog 2019
    Eid AL ADHA

    ReplyDelete
  23. DJ Hire in London, DJ agencies London
    Dj Required has been setup by a mixed group of London’s finest Dj’s, a top photographer and cameraman. Together we take on Dj’s, Photographers and Cameramen with skills and the ability required to entertain and provide the best quality service and end product. We supply Bars, Clubs and Pubs with Dj’s, Photographers, and Cameramen. We also supply for private hire and other Occasions. Our Dj’s, Photographers and Cameramen of your choice, we have handpicked the people we work with

    ReplyDelete
  24. I am impressed by the information that you have on this blog. It shows how well you understand this subject.
    AI learning course malaysia

    ReplyDelete
  25. Very interesting blog Thank you for sharing such a nice and interesting blog and really very helpful article

    Workday Online Training

    ReplyDelete
  26. Nice information, valuable and excellent design, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here.
    pmp certification malaysia

    ReplyDelete
  27. Thanks for sharing this informative blog. It gives a clear view about hibernategrails.
    Advanced Java Online Training

    ReplyDelete
  28. This comment has been removed by the author.

    ReplyDelete
  29. Car Maintenance Tips That You Must Follow


    For everyone who owns it, Car Maintenance Tips need to know.
    Where the vehicle is currently needed by everyone in the world to
    facilitate work or to be stylish.
    You certainly want the vehicle you have always been in maximum
    performance. It would be very annoying if your vehicle isn’t even
    comfortable when driving.
    Therefore to avoid this you need to know Vehicle Maintenance Tips or Car Tips
    Buy New Car visit this site to know more.

    wanna Buy New Car visit this site.
    you dont know about Car Maintenance see in this site.
    wanna know about Car Tips click here.
    know more about Hot car news in here.


    ReplyDelete
  30. LogoSkill,
    Logo Design Company
    is specifically a place where plain ideas converted into astonishing and amazing designs. You buy a logo design, we feel proud in envisioning
    our client’s vision to represent their business in the logo design, and this makes us unique among all. Based in USA we are the best logo design, website design and stationary
    design company along with the flayer for digital marketing expertise in social media, PPC, design consultancy for SMEs, Start-ups, and for individuals like youtubers, bloggers
    and influencers. We are the logo design company, developers, marketers and business consultants having enrich years of experience in their fields. With our award winning
    customer support we assure that, you are in the hands of expert designers and developers who carry the soul of an artist who deliver only the best.

    Logo Design Company

    ReplyDelete
  31. very interesting , good job and thanks for sharing such a good blog.Usually I never comment on blogs but your article is so convincing that I never stop myself to say something about it. You’re doing a great job Man,Keep it up. ExcelR Solutions

    ReplyDelete
  32. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
    machine learning course malaysia

    ReplyDelete
  33. Business Analytics or Data Analytics or data science training in hyderabad is an extremely popular, in-demand profession which requires a professional to possess sound knowledge of analysing data in all dimensions and uncover the unseen truth coupled with logic and domain knowledge to impact the top-line (increase business) and bottom-line (increase revenue).

    ReplyDelete
  34. Usually I never comment on blogs but your article is so convincing that I never stop myself to say something about it. You’re doing a great job Man,Keep it up.Tereza Burki

    ReplyDelete
  35. Awesome Blog, I love your writing style, good job and thanks for sharing this information. You’re doing a great job Man, Keep it up!! Emanualonline Review

    ReplyDelete
  36. Nice article.
    For AWS training in bangalore, Visit:
    AWS training in bangalore

    ReplyDelete
  37. A good blog always comes-up with new and exciting information and while reading I have feel that this blog is really have all those quality that qualify a blog to be a one.
    data science course

    ReplyDelete
  38. Every business these days need to collect data at every point of the manufacturing and sales process to understand the journey of the product.
    This may include applications, clicks, interactions, and so many other details related to the business process which can help define goals in a better way.
    Therefore, we bring you the list of benefits which you can reap with the use of Digital Marketing Course in Sydney in your process of management.
    every business has a single reason for which the interaction of the customer and the seller is established and it is the product to be sold. Therefore, it is very crucial
    that you must add relevance to your product by understanding the needs of the customers with the addition of features and design improvements which can make your product a
    perfect fit for the target audience. This can be easily achieved with the right interpretation skills which you can only get with Data Analytics Certification.

    ReplyDelete




  39. Thank you so much for sharing the article. Really I get many valuable information from the article
    With our Digital Marketing Training, re-discover your creative instinct to design significant
    marketing strategies to promote a product/service related to any organization from any business sector.

    Digital Marketing Course


    ReplyDelete
  40. Tech Gadgets reviews and latest Tech and Gadgets news updates, trends, explore the facts, research, and analysis covering the digital world.
    You will see Some Tech reviews below,

    lg bluetooth headset : You will also wish to keep design and assorted features in mind. The most essential part of the design here is the buttonsof lg bluetooth headset .

    Fastest Car in the World : is a lot more than the usual number. Nevertheless, non-enthusiasts and fans alike can’t resist the impulse to brag or estimate according to specifications. Fastest Car in the World click here to know more.

    samsung galaxy gear : Samsung will undoubtedly put a great deal of time and even more cash into courting developers It is looking for partners and will allow developers to try out
    different sensors and software. It is preparing two variants as they launched last year. samsung galaxy gear is very use full click to know more.

    samsung fridge : Samsung plans to supply family-oriented applications like health care programs and digital picture frames along with games It should stick with what they know and they
    do not know how to produce a quality refrigerator that is worth what we paid. samsung fridge is very usefull and nice product. clickcamera best for travel: Nikon D850: Camera It may be costly, but if you’re trying to find the very best camera you can purchase at this time, then Nikon’s gorgeous DX50 DSLR will
    probably mark each box. The packaging is in a vibrant 45.4-megapixel full-frame detector, the picture quality is simply wonderful. However, this is just half the story. Because of a complex 153-point AF system along with a brst rate of 9 frames per minute. camera best specification. click here to know more.

    visit https://techgadgets.expert/ this site to know more.

    ReplyDelete
  41. Digital Marketing can be defined as a unique marketing strategy that is implemented in digital platforms through Internet Medium to reach the target audience. When compared to traditional marketing, search analytics gives you an extra edge in Digital Marketing. Analytics empowers the business to analyse the success in their business strategies and provides the required data to modify the strategies to suit the market requirements and improve ROI.

    Digital Marketing Course
    Digital Marketing Course in Sydney

    ReplyDelete
  42. Nice Post...I have learn some new information.thanks for sharing.. Machine Learning Training In Bangalore

    ReplyDelete
  43. For IOT Training in Bangalore Visit: IOT Training in Bangalore

    ReplyDelete
  44. How to stop smoking weed ???

    1.Do you want to know How to stop smoking weed or have you been wondering of how to get your dear one to stop smoking weed ?
    Every weed smoker knows deep down that quitting the behavior is an uphill task. And some people don`t know How to quit smoking weed .. They should
    Know How to quit smoking weed by visiting in this https://irvined.co.uk/ website.

    2.Long-term marijuana users may find the withdrawal experience uncomfortable. Marijuana detox helps one to slowly ease off of THC
    until it is completely eliminated from the body system. Marijuana detox also helps to reduce withdrawal symptoms thus making it
    easier for even highly addicted individuals to make full about turn in their weed smoking habits (avoiding relapse).

    3.The decision to stop smoking weed is usually impulsive for many individuals. If you have smoked pot long enough, you probably have plenty of memories
    where you did something and swore to yourself to stop your weed smoking habit going forward. And if you don`t know How to stop smoking pot ...
    Then visit https://irvined.co.uk/ here to know more.

    4.Quitting marijuana will give you the chance to become more responsible and set you in the right direction to progress in your life.
    And the good thing is that you don’t have to try quitting on your own now. ‘ Quit Marijuana The Complete Pack ’ is just a click away at a very affordable price.
    See more details about the guide and current purchase offers at https://quit-weed.com/. You can do this. Regardless of how long you have smoked pot or used marijuana
    in other ways, the quit marijuana pack offers you robust support to ensure that you achieve your goals. To know more information visit https://irvined.co.uk/ here.

    ReplyDelete
  45. This is the exact information I am been searching for, Thanks for sharing the required infos with the clear update and required points. To appreciate this I like to share some useful information.aws training in bangalore

    ReplyDelete
  46. Choose high quality and durable dennys driveshaft replacement parts for your Nissan. Replacement parts are available for your air intake system, body electrical, body mechanical and trim, body sheet metal, brakes, climate control, clutch, cooling system, diesel injection, drive belts, drive shafts and axle, engine electrical, engine parts, exhaust, fuel delivery, steering, suspension, tools and hardware, transmission. Replacement parts keep your Nissan running and looking great, these parts will surely make it more stylish, more fun to drive, more comfortable and convenient, and more high-tech. dennys driveshaft .

    ReplyDelete
  47. Its really helpful for the users of this site. I am also searching about these type of sites now a days. So your site really helps me for searching the new and great stuff.python training in bangalore

    ReplyDelete
  48. Very useful and information content has been shared out here, Thanks for sharing it.google cloud platform training in bangalore

    ReplyDelete
  49. These provided information was really so nice,thanks for giving that post and the more skills to develop after refer that post.blue prism training in bangalore

    ReplyDelete
  50. inking is very useful thing.you have really helped lots of people who visit blog and provide them use full information.sccm training in bangalore

    ReplyDelete
  51. Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well as giving.vmware training in bangalore

    ReplyDelete
  52. Really it was an awesome article,very interesting to read.You have provided an nice article,Thanks for sharing.aws training in bangalore

    ReplyDelete
  53. I know that it takes a lot of effort and hard work to write such an informative content like this.data science training in bangalore

    ReplyDelete
  54. Smart Outsourcing Solutions is the leading web development, ecommerce solution, offshore outsourcing development and freelancing training company in Dhaka Bangladesh please
    visit us:Digital marketing training
    Digital marketing training in dhaka
    Digital Marketing Training In Bangladesh

    ReplyDelete
  55. I am overwhelmed by your post with such a nice topic. Usually I visit your blogs and get updated through the information you include but today’s blog would be the most appreciable. Well done!
    Please check ExcelR Data Science Certification

    ReplyDelete
  56. You might comment on the order system of the blog. You should chat it's splendid. Your blog audit would swell up your visitors. I was very pleased to find this site.I wanted to thank you for this great read!!
    Know more Data Science Course in Pune

    ReplyDelete
  57. This post is really nice and informative. The explanation given is really comprehensive and informative . Thanks for sharing such a great information..Its really nice and informative . Hope more artcles from you. I want to share about the best java tutorial videos for beginners with free bundle videos provided and java training .

    ReplyDelete
  58. We as a team of real-time industrial experience with a lot of knowledge in developing applications in python programming (7+ years) will ensure that we will deliver our best in python training in vijayawada. , and we believe that no one matches us in this context.

    ReplyDelete
  59. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
    data analytics course mumbai

    ReplyDelete
  60. Thank you for excellent article, Please refer below if you are looking for best training institute in hyderabad.
    Microservices Online Training
    Microservices Training in Hyderabad

    ReplyDelete
  61. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.course on big data analytics
    data scientist course malaysia
    data analytics courses

    ReplyDelete
  62. I am a new user of this site so here i saw multiple articles and posts posted by this site,I curious more interest in some of them hope you will give more information on this topics in your next articles.
    data analytics course in bangalore
    business analytics courses
    data analytics courses
    data science interview questions

    ReplyDelete

  63. This post is very simple to read and appreciate without leaving any details out. Great work! data science course

    ReplyDelete
  64. This comment has been removed by the author.

    ReplyDelete
  65. I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end. I would like to read newer posts and to share my thoughts with you.big data analytics malaysia
    data science course malaysia
    data analytics courses
    360DigiTMG

    ReplyDelete
  66. Your blog is very good to see & thanks for sharing &keep sharing
    devops training in Hyderabad

    ReplyDelete
  67. devops training in Hyderabad
    Your blog is very good to see & thanks for sharing &keep sharing

    ReplyDelete
  68. Genyatra provides train ticket, flight ticket, senior citizen yatra services to its Clients across World.
    Ticketing: All types of Domestic and International Flight ticket booking at competitive price. We provide best corporate fare and group fare across world.
    Packages: We create specialized travel packages like family holidays, honeymoons, meetings, pilgrimage tours, special packages for senior citizen tours & women tours.
    Visa and Forex: We Specialize in visa assistance for Individual and Group. We provides foreign currency at best exchange available rates. we provide Travel insurance.
    Flight tkt, teerthyatra, foreign exchange rail ticket

    ReplyDelete
  69. hi..
    Each year, thousands of young children are killed or injured in car crashes. Proper use of car seats helps keep
    children safe. But with so many different seats on the market, many parents find this overwhelming.
    If you are expectant parents, give yourselves enough time to learn how to properly install the car seat
    in your car before your baby is born to ensure a safe ride home from the hospital.
    baby car seater
    The type of seat your child needs depends on several things, including your child's age, size, and developmental
    needs. [url=http://www.best-babycarseats.com]babycarseats[/url] Read on for more information from the American Academy of Pediatrics (AAP) about choosing the most appropriate
    car seat for your child.

    ReplyDelete
  70. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    digital marketing course

    For more info :

    ExcelR - Data Science, Data Analytics, Business Analytics Course Training in Mumbai

    304, 3rd Floor, Pratibha Building. Three Petrol pump, Opposite Manas Tower, LBS Rd, Pakhdi, Thane West, Thane, Maharashtra 400602
    18002122120

    ReplyDelete
  71. Study Machine Learning Training in Bangalore with ExcelR where you get a great experience and better knowledge .
    Business Analytics course

    ReplyDelete
  72. Hi there, First of all great post enjoyed reading and taking note of your worthwhile information! This is one of your best posts to date! I really appreciate that you added links to further reading to document and support your points.
    WordPress Phone Number

    ReplyDelete
  73. AWS
    Mind Q Systems provides AWS training in Hyderabad & Bangalore.AWS training designed for students and professionals. Mind Q Provides 100% placement assistance with AWS training.

    Mind Q Systems is a Software Training Institute in Hyderabad and Bangalore offering courses on Testing tools, selenium, java, oracle, Manual Testing, Angular, Python, SAP, Devops etc.to Job Seekers, Professionals, Business Owners, and Students. We have highly qualified trainers with years of real-time experience.



    ReplyDelete
  74. Big Truck Tow: Heavy Duty towing service san jose

    We're rated the most reliable heavy duty towing san jose service & roadside assistance in San Jose!
    Call us now! We're ready to help you NOW!

    Since 1999, tow truck san jose has provided quality services to clients by providing them
    with the professional care they deserve. We are a professional and affordable Commercial
    Towing Company. BIG TRUCK TOW provides a variety of services, look below for the list of
    services we offer. Get in touch today to learn more about our heavy duty towing


    Click here to Find tow truck near me

    ReplyDelete
  75. Very nice job... Thanks for sharing this amazing Machine Learning Courses and educative blog post!

    ReplyDelete
  76. Wow. That is so elegant and logical and clearly explained. Brilliantly goes through what could be a complex process and makes it obvious.I want to refer about the tableau online training in hyderabad and tableau tutorial videos

    ReplyDelete
  77. Really very happy to say that your post is very interesting. I never stop myself to say something about it. You did a great job. Keep it up.
    We have an excellent IT courses training institute in Hyderabad. We are offering a number of courses that are very trendy in the IT industry. For further information, please once go through our site.
    Best Cyber Security Training Institute in Hyderabad

    ReplyDelete
  78. Wow!! Really a nice Article about Java. Thank you so much for your efforts. Definitely, it will be helpful for others. I would like to follow your blog. Share more like this. Thanks Again.
    Java training in chennai | Java training in annanagar | Java training in omr | Java training in porur | Java training in tambaram | Java training in velachery

    ReplyDelete
  79. Thanks for sharing this wonderful message.
    android training institutes in coimbatore

    data science course in coimbatore

    data science training in coimbatore

    python course in coimbatore

    python training institute in coimbatore

    Software Testing Course in Coimbatore

    CCNA Course in Coimbatore

    ReplyDelete
  80. Effective blog with a lot of information. I just Shared you the link below for Courses .They really provide good level of training and Placement,I just Had Hibernates Classes in this institute,Just Check This Link You can get it more information about the Hibernates course.


    Java training in chennai | Java training in annanagar | Java training in omr | Java training in porur | Java training in tambaram | Java training in velachery

    ReplyDelete
  81. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck. data science course

    ReplyDelete
  82. Talk with Strangerstalk to strangers in Online Free Chat rooms where during a safe environment.
    From friendships to relationships.omegle teen Talk With Stranger is that the best online chatting site.
    Its the simplest alternative to airg chat, Badoo , omegle & mocospace. If you're keen on speaking
    with people on the web ,chat random or want to seek out omegle girls, do free texting or sexting, make new friends.
    you'll find your omegle lady here. Please note this is often not a sexting site so you can't do sexting
    online. this is often a familychatous friendly chat site. we've voice chat if you would like to try to to phone
    chat online. Our most viral is that the 1-1 one on one random chat.talkwithstranger No check in on login needed.
    we've teengers also asanonymous chat older people that want to satisfy new people. Online random chat is that the best
    chatrandom alternative.

    ReplyDelete
  83. SSC Result 2020 Published Date & Time by ssc result 2020
    ssc result 2020
    Education Board of Bangladesh.
    Many of You Search For SSC Result Kobe Dibe on Internet
    as Well as Facebook. The results of Secondary School Certificate
    (SSC)—and its equivalent examinations—for 2020 have been published.
    SSC & Dakhil Result 2020 Published Date is Very Important For T
    he Students Who Attend The SSC Exam 2020.

    ReplyDelete
  84. Study ExcelR Business Analytics Courses where you get a great experience and better knowledge.

    Business Analytics Courses .

    ReplyDelete
  85. I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more....data analytics courses

    ReplyDelete
  86. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

    data science interview questions

    ReplyDelete
  87. his is really a very good article about Java.Thanks for taking the time to discuss with us , I feel happy about learning this topic.
    AWS training in chennai | AWS training in annanagar | AWS training in omr | AWS training in porur | AWS training in tambaram | AWS training in velachery

    ReplyDelete
  88. Nice Post. Very informative Message and found a great post. Thank you.
    Data Science Training in Hyderabad

    ReplyDelete
  89. I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…
    Machine Learning Courses Very good points you wrote here..Great stuff...I think you've made some truly interesting points.Keep up the good work.

    ReplyDelete
  90. we are an online store for all kind of mens and womens watches in india only.ڈپلیکیٹ برانڈڈ گھڑیاں آن لائن

    ReplyDelete
  91. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data scientist course in hyderabad with placement

    ReplyDelete
  92. An overwhelming web journal I visit this blog, it's unfathomably amazing. Unusually, in this present blog's substance made inspiration driving truth and reasonable. The substance of data is enlightening.


    Full Stack Course Chennai
    Full Stack Training in Bangalore

    Full Stack Course in Bangalore

    Full Stack Training in Hyderabad

    Full Stack Course in Hyderabad

    Full Stack Training

    Full Stack Course

    Full Stack Online Training

    Full Stack Online Course



    ReplyDelete
  93. Attend The Data Science Course From ExcelR. Practical Data Science Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Course.data science courses

    ReplyDelete
  94. Leave the city behind & drive with us for a Thrilling drive over the Desert Dunes & Experience a lavish dinner with amazing shows in our Desert Camp.
    desert safari dubai

    ReplyDelete

  95. There is noticeably a bundle to know about this. I assume you made certain nice points in features also . battery reconditioning scams

    ReplyDelete
  96. This is such an awesome asset, to the point that you are giving and you give it away for nothing.our article has piqued a lot of positive interest. I can see why since you have done such a good job of making it interesting. Visit

    ReplyDelete
  97. Hey, Nice one information

    Online IT Software Courses Training ICT South Bopal - Ahmedabad

    Best Programming Training at ICT


    Institute of Computer Training - ICT Bopal

    ReplyDelete
  98. The way you write, you are really a professional blogger. zoe bray cotton

    ReplyDelete
  99. Thank you for taking the time and sharing this information with us. It was indeed very helpful and insightful while being straight forward and to the point.

    Python Training in Gurgaon
    Advanced Excel /VBA training in Gurgaon
    Selenium Training in Gurgaon

    ReplyDelete
  100. Nice & Informative Blog !
    If you are looking for the best accounting software that can help you manage your business operations. call us at QuickBooks Customer Support Number 1-855-974-6537.

    ReplyDelete
  101. Nice Blog !
    QuickBooks is a premium accounting solution that handles everything from inventory to payroll.To get instant support for such problems, reach our experts via QuickBooks Customer Service Number 1-877-751-0742.

    ReplyDelete
  102. Hey! Amazing work. With full of knowledge. Our Team resolve any glitches occurring while utilizing the software. Looking Quickbooks For Mac Support Phone Number 1-855-756-1077. Our experts will assist you to fulfil your accounting needs. The solutions are accurate and time-saving.

    ReplyDelete
  103. I’m happy I located this blog! From time to time, students want to cognitive the keys of productive literary essays composing. Your first-class knowledge about this good post can become a proper basis for such people. nice one
    data science training

    ReplyDelete
  104. Nice & Informative Blog !
    Make sure to call us at our QuickBooks Customer Service Phone Number 1-(855)-974-6537, and gain instant solutions to all your queries and issues on a single call.

    ReplyDelete
  105. Hey!! Great work. You have a very informative blog .You are doing well. Keep it up. We will also provide Quickbooks Customer Service Phone Number to alter Quickbook’s issues. If you have any issues regarding Quickbooks dial +1-877-948-5867 for getting instant help.

    ReplyDelete

  106. Your work is very good and I appreciate you and hopping for some more informative posts. ExcelR Data Science Course In Pune

    ReplyDelete
  107. Nice Post !
    To fix such issue, you must contact our experts via QuickBooks Customer Support Number 1-855-974-6537 and get permanent ways to solve QuickBooks problems. Our team consists of highly qualified professionals who provide the best ways to troubleshoot QuickBooks problems.

    ReplyDelete
  108. Nice Blog !
    Need quick help in QuickBooks? Don’t get annoyed!! Just reach out to our QuickBooks Customer Service Number 1-855-974-6537, and acquire excellent service benefits. We have a technical team of QB experts who can troubleshoot all sorts of complicated error codes without taking too much time.

    ReplyDelete
  109. Hey! Good blog. I was facing an error in my QuickBooks software, so I called QuickBooks Error Code 15106 (855)-756-1077. I was tended to by an experienced and friendly technician who helped me to get rid of that annoying issue in the least possible time.

    ReplyDelete
  110. Hey! Mind-blowing blog. Keep writing such beautiful blogs. In case you are struggling with issues on QuickBooks software, dial QuickBooks For MAC Support (855)756-1077. The team, on the other end, will assist you with the best technical services.

    ReplyDelete
  111. ExcelR provides Business Analytics Course. It is a great platform for those who want to learn and become a Business Analytics Courses. Students are tutored by professionals who have a degree in a particular topic. It is a great opportunity to learn and grow.

    Business Analytics Courses

    ReplyDelete
  112. Hey! Mind-blowing blog. Keep writing such beautiful blogs. In case you are struggling with issues on QuickBooks software, dial QuickBooks Phone Number (877)948-5867. The team, on the other end, will assist you with the best technical services.

    ReplyDelete
  113. ExcelR provides Business Analytics Courses. It is a great platform for those who want to learn and become a Business Analytics Course. Students are tutored by professionals who have a degree in a particular topic. It is a great opportunity to learn and grow.

    Business Analytics Courses
    Business Analytics course
    Business Analytics course in pune

    ReplyDelete
  114. ExcelR provides data analytics courses. It is a great platform for those who want to learn and become a data analytics Course. Students are tutored by professionals who have a degree in a particular topic. It is a great opportunity to learn and grow.

    data analytics courses
    data analytics course

    ReplyDelete
  115. ExcelR provides data analytics courses. It is a great platform for those who want to learn and become a data analytics Course. Students are tutored by professionals who have a degree in a particular topic. It is a great opportunity to learn and grow.

    data analytics courses
    data analytics course

    ReplyDelete
  116. ExcelR provides Business Analytics Courses. It is a great platform for those who want to learn and become a Business Analytics Course. Students are tutored by professionals who have a degree in a particular topic. It is a great opportunity to learn and grow.


    Business Analytics Courses
    Business Analytics Course

    ReplyDelete
  117. Nice Blog
    QuickBooks is one of the best accounting software that is developed to manage all the accounting tasks of a business. In case you want quick solutions for QuickBooks problems, call us on QuickBooks Customer Service Number and get the most feasible solutions for QuickBooks errors.

    ReplyDelete
  118. Nice Blog
    QuickBooks is one of the best accounting software that is developed to manage all the accounting tasks of a business. In case you want quick solutions for QuickBooks problems, call us on QuickBooks Customer Service Number and get the most feasible solutions for QuickBooks errors.

    ReplyDelete
  119. Nice & Informative Blog !
    QuickBooks Error 248 is an error that usually occurs when you are working on QuickBooks software. If you are also struggling with the same error, we have solutions for you.

    ReplyDelete
  120. Hey! Lovely blog. Your blog contains all the details and information related to the topic. In case you are a QuickBooks user, here is good news for you. You may encounter any error like QuickBooks Error, visit at QuickBooks technical support phone number for quick help.

    ReplyDelete
  121. Nice & Informative Blog !
    If QuickBooks Error 15101 is still not fixed, immediately contact on QuickBooks Support Number and connect with technical experts to fix the issue without a minute delay.

    ReplyDelete
  122. Thank you for your valuable content.very helpful for learners and professionals. You are doing very good job to share the useful information which will help to the students . if you are looking for
    Best Machine Learning Training in Gurgaon
    then Join iClass Gyansetu

    ReplyDelete
  123. Thank you for your valuable content.very helpful for learners and professionals. You are doing very good job to share the useful information which will help to the students . if you are looking for
    Best Machine Learning Training in Gurgaon
    then Join iClass Gyansetu

    ReplyDelete
  124. Hey! Nice Blog, I have been using QuickBooks for a long time. One day, I encountered QuickBooks Error 404 in my software, then I called QuickBooks Error 404 They resolved my error in the least possible time.

    ReplyDelete
  125. incredible article distributed here by you. i've for a long while been itching to adapt new things with respect to this subject, and i have unquestionably adapted new things today. reviews

    ReplyDelete
  126. Hey! What a wonderful blog. I loved your blog. QuickBooks is the best accounting software, however, it has lots of bugs like QuickBooks Error. To fix such issues, you can contact experts via QuickBooks Customer Support Number

    ReplyDelete
  127. Thanks for sharing such useful information with us. I hope you will share some more info about your blog. Please keep sharing. We will also provide QuickBooks Customer Service Number for instant help.

    ReplyDelete
  128. Thanks for sharing such useful information with us. I hope you will share some more info about your blog. Please keep sharing. We will also provide QuickBooks Error 1723 for instant help.

    ReplyDelete
  129. Thanks for sharing such useful information with us. I hope you will share some more info about your blog. Please keep sharing. We will also provide QuickBooks Customer Service Phone Number for instant help.

    ReplyDelete

  130. Amazing blog.Thanks for sharing such excellent information with us. keep sharing...


    ReplyDelete
  131. Hey! Nice Blog, I have been using QuickBooks for a long time. One day, I encountered QuickBooks Customer Service in my software, then I called QuickBooks Customer Service Number. They resolved my error in the least possible time.

    ReplyDelete

  132. Hey! Mind-blowing blog. Keep writing such beautiful blogs. In case you are struggling with issues on QuickBooks software, dial QuickBooks Customer Service Phone Number . The team, on the other end, will assist you with the best technical services.

    ReplyDelete

  133. Thank you for excellent article.You made an article that is interesting. Hair Fall Control Hair Oil

    ReplyDelete
  134. this article is interesting to read and useful.keep up the good work.Angular training in Chennai

    ReplyDelete
  135. AMAZING BACHELORETTE PARTY SUPPLIES We strive to have a positive impact on small to medium businesses, customers, employees, the economy, and communities. Surjmor bring together smart, passionate builders with different backgrounds and goals, who share a common desire to always be learning and inventing on behalf of our customers. With all the family of business that are a part of us, our goals is providing customers with the best service possible.

    xxxtoys.top

    ReplyDelete
  136. Best Erotic Bonage Blindfolds Restraint We strive to have a positive impact on small to medium businesses, customers, employees, the economy, and communities. Surjmor bring together smart, passionate builders with different backgrounds and goals, who share a common desire to always be learning and inventing on behalf of our customers. With all the family of business that are a part of us, our goals is providing customers with the best service possible.

    https://xxxtoys.top/

    ReplyDelete
  137. Hey! Well-written blog. It is the best thing that I have read on the internet today. Moreover, if you are looking for the solution of QuickBooks Software, visit at QuickBooks Customer Service to get your issues resolved quickly.

    ReplyDelete
  138. Hey! What a wonderful blog. I loved your blog. QuickBooks is the best accounting software, however, it has lots of bugs like QuickBooks Error. To fix such issues, you can contact experts via QuickBooks Support Phone Number

    ReplyDelete
  139. Your blog has wonderful information regarding software Development, I also have some valuable information regarding the Best Guidewire Development Services in USA, Canada and India hopefully, this will be very helpful for you.

    ReplyDelete
  140. I am very much pleased with the contents you have mentioned. I wanted to thank you for this great article. akatsuki puffer jacket

    ReplyDelete
  141. Thanks for sharing
    Online Training | Classroom | Virtual Classes
    DevOps Training in Hyderabad with 100% placement assistance
    1860 testers placed in 600 companies in last 8 years
    Real time expert trainers
    Indutry oriented training with corporate casestudies
    Free Aptitude classes & Mock interviews

    ReplyDelete
  142. Hey! Excellent work. Being a QuickBooks user, if you are struggling with any issue, then dial QuickBooks Phone Number (855)444-2233. Our team at QuickBooks will provide you with the best technical solutions for QuickBooks problems.

    ReplyDelete
  143. Hey! Well-written blog. It is the best thing that I have read on the internet today. Moreover, if you are looking for the solution of QuickBooks Enterprise Support (855)756-1077, visit at QuickBooks Customer Service Number (888)233-6656 to get your issues resolved quickly.

    ReplyDelete
  144. Outstanding article! I want people to know just how good this information is in your article. Your views are much like my own concerning this subject. I will visit daily your blog because I know. It may be very beneficial for me. thejustreviews

    ReplyDelete
  145. This post is so interactive and informative.keep update more information...
    DevOps Training in Anna Nagar
    DevOps Training in Chennai

    ReplyDelete
  146. Excellent read, I just passed this onto a colleague who was doing a little research on that. And he actually bought me lunch because I found it for him smile So let me rephrase that. https://thejustreviews.com

    ReplyDelete
  147. Hiiii!!
    Wonderfull bolg. i love it if youy are looking for Quickbooks costumer service you can contact us at. +1 855-786-5155,NH.

    ReplyDelete
  148. Hey!
    Well-written blog. It is the best thing that I have read on the internet today. Moreover, if you are looking for the solution of QuickBooks Software, visit at QuickBooks Customer Service+1 888-272-4881 to get your issues resolved quickly.

    ReplyDelete
  149. Good to visit your weblog again, it has been months for me. Nicely this article that i've been waiting for so long. I will need this post to total my assignment in the college, and it has exact same topic together with your write-up. Thanks, good share.
    data science training in hyderabad

    ReplyDelete
  150. Thanks for posting this info. I just want to let you know that I just check out your site. Fonzie Leather Jacket

    ReplyDelete