Using G1 for Big Data Streams to Cassandra

Background; Adello does mobile ads using a medium size Hadoop cluster (http://www.technavio.com/blog/top-16-hadoop-technology-companies), taking in, among others, the Smaato, Flurry, OpenX, and OMAX feeds. Combined, this is a fair amount of data to handle, about a terabyte per day. Cassandra's turned out to be great for this task, even if we had to fix a race condition and develop new read/write paths from Pig as the existing ones were not able to handle the analysis load (see https://issues.apache.org/jira/browse/CASSANDRA-6788 and https://issues.apache.org/jira/browse/CASSANDRA-6665).

The standard garbage collection config for Cassandra is not bad, but it couldn't quite cope with the 20GB+ heaps we had before we really optimized our Cassandra config. In particular, too much data was pushed into the old space. This creates 30-50 second long stop-the-world GC cycles every 2-4 hours. This was a serious issue when using the token aware writing of Astyanax (https://github.com/Netflix/astyanax), a problem partly solved by optimizing fail-overs, retries, and using Hystrix (https://github.com/Netflix/hystrix) to drop the outliers. Overall, I found Netflix' Github account to be a open-pit gold mine :)


Even though we're no longer using G1, I found the experiences worth sharing. The G1 collector was running in production for a few weeks, and here are our observations:

Initially, it seemed to be a fire-and-forget change; once we switched from CMS, we've immediately saw a major reduction in the time and frequency of long GC cycles.

However, too much data was still pushed into old space. This is caused to the memory-wasteful read/write paths of Cassandra over Thrift.

We soon realized that the time spent on the full cycles increased linearly with the Cassandra's uptime. Over the course of a week, the full GC cycle went from taking 10 seconds to 60 seconds, reclaiming the same amount of memory. This is mostly caused by fragmentation, which G1 was supposed handle better than CMS. Internally, G1 arranges the space into blocks that can be reclaimed/defragmented separately. Clearly, the simplest solution was to restart Cassandra nodes on a regular basis, like Spotify does (http://www.slideshare.net/planetcassandra/8-axel-liljencrantz-23204252). However, with over 4TB per node, Cassandra takes roughly 5 minutes to start, making a serial restart a lengthy process.

To combat the fragmentation issue, we turned to the experimental options for G1. The "-XX:G1NewSizePercent" defaults to 5 and "-XX:G1MaxNewSizePercent" defaults to 60 (they're described here: http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html). Given that Cassandra is wasteful with memory, most of the connection-related data is ready to be reclaimed almost instantly. By using "-XX:G1NewSizePercent=60" and "-XX:G1MaxNewSizePercent=95", we keep almost all the data off old space. When using the defaults, the old space would slowly grow to about 12GB, with the new settings, it stayed at about 1GB. Given that the young collection is more parallel, this was huge benefit both for responsiveness and max heap use.

Some notes: setting the "-XX:InitiatingHeapOccupancyPercent" doesn't seem to have much impact. It defaults to 45, but even setting it to 5 or 0 doesn't mean the GC will step in earlier. Reducing the max heap size, however, makes the GC kick in. This even made it possible to run Cassandra with less than 8GB of RAM, but the risk/benefit wasn't there for us. Also, we used "-XX:MaxGCPauseMillis=100", compared to the default of 200. However, this has little impact as G1 typically collected 7GB of new space in 50ms, which is pretty fantastic performance.

Cassandra's been great for the most part. The main problem is that it has an extremely ungraceful degradation. If you leave Cassandra's intended design space, or run with a non-optimized config, it will quickly turn into a constant source of pain. However, once you've figured out the best config for your application, it just keeps running, with minimal management and almost zero stability issues.


Cheers,
Christian

77 comments:

  1. It realizes your indiscreet however savvy approach. To remain on the ball, it gains by the investigation. data science course in pune

    ReplyDelete
    Replies
    1. Great Article Cloud Computing Projects

      Networking Projects

      Final Year Projects for CSE

      JavaScript Training in Chennai

      JavaScript Training in Chennai

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

      Delete
  2. Attend The Data Science Courses in Bangalore From ExcelR. Practical Data Science Courses in Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Courses in Bangalore.
    ExcelR Data Science Course Bangalore

    ReplyDelete
  3. Thanks for sharing your valuable information to us, it is very useful.
    digital marketing course

    ReplyDelete
  4. Such a very useful article. I have learn some new information.thanks for sharing.
    data scientist course in mumbai

    ReplyDelete
  5. You might comment on the order system of the blog. You should chat it's splendid. Your blog audit would swell up your visitors. I was very pleased to find this site.I wanted to thank you for this great read!!
    Data Analytics Course in Mumbai

    ReplyDelete
  6. Excelr is providing emerging & trending technology training, such as for data science, Machine learning, Artificial Intelligence, AWS, Tableau, Digital Marketing. Excelr is standing as a leader in providing quality training on top demanding technologies in 2019. Excelr`s versatile training is making a huge difference all across the globe. Enable ?business analytics? skills in you, and the trainers who were delivering training on these are industry stalwarts. Get certification on "
    best data science course in hyderabad"
    and get trained with Excelr.

    ReplyDelete
  7. Such a very useful Blog. Very interesting to read this article. I have learn some new information.thanks for sharing. know more about

    ReplyDelete
  8. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!
    ExcelR data science

    ReplyDelete
  9. Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.
    ExcelR Data Analytics courses

    ReplyDelete
  10. Good to become visiting your weblog again, it has been months for me. Nicely this article that i've been waited for so long. I will need this post to total my assignment in the college, and it has exact same topic together with your write-up. Thanks, good share.
    Please check ExcelR Data Science Course in Pune

    ReplyDelete
  11. Awesome..I read this post so nice and very imformative information...thanks for sharing
    Click here for data science course

    ReplyDelete
  12. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more. excelr data science

    ReplyDelete
  13. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    ExcelR data science course in mumbai

    ReplyDelete
  14. You might comment on the order system of the blog. You should chat it's splendid. Your blog audit would swell up your visitors. I was very pleased to find this site.I wanted to thank you for this great read!!
    data analytics courses online

    ReplyDelete
  15. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
    data analytics course mumbai

    ReplyDelete
  16. Actually I read it yesterday but I had some thoughts about it and today I wanted to read it again because it is very well written.
    .Please check ExcelR Data Science Course in Pune

    ReplyDelete
  17. Attend The Machine Learning courses in Bangalore From ExcelR. Practical Machine Learning courses in Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Machine Learning courses in Bangalore.
    ExcelR Machine Learning courses in Bangalore

    ReplyDelete
  18. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful.

    data analytics courses

    data science interview questions

    business analytics courses

    data science course in mumbai

    ReplyDelete
  19. Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

    business analytics course

    data analytics courses

    data science interview questions

    data science course in mumbai

    ReplyDelete
  20. This is a wonderful article, Given so much info in it, Thanks for sharing. CodeGnan offers courses in new technologies and makes sure students understand the flow of work from each and every perspective in a Real-Time environmen python training in vijayawada. , data scince training in vijayawada . , java training in vijayawada. ,

    ReplyDelete
  21. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our courses of pmp certifications
    pmp certifications | https://www.excelr.com/pmp-training-in-mumbai

    ReplyDelete
  22. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!

    Invisalign orthodontists

    ReplyDelete
  23. Truly, this article is really one of the very best in the history of articles. I am a antique ’Article’ collector and I sometimes read some new articles if I find them interesting. And I found this one pretty fascinating and it should go into my collection. Very good work!data science course
    360DigiTMG

    ReplyDelete
  24. Super site! I am Loving it!! Will return once more, Im taking your food additionally, Thanks.
    Know more Data Scientist Courses

    ReplyDelete
  25. The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
    Artificial Inteligence course in Patna
    AI Course in Patna

    ReplyDelete
  26. It's really nice and meaningful. it's really cool blog. Linking is very useful thing. You have really helped lots of people who visit blog and provide them useful information.
    More Information of ExcelR

    ReplyDelete
  27. Data Science Courses
    It's really nice and meaningful. it's really cool blog. Linking is very useful thing. You have really helped lots of people who visit blog and provide them useful information.

    ReplyDelete
  28. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data science Interview Questions

    ReplyDelete
  29. Attend The Data Science Training Bangalore From ExcelR. Practical Data Science Training Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Training Bangalore.
    Data Science Training Bangalore
    Data Science Interview Questions

    ReplyDelete
  30. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data Science Course

    ReplyDelete
  31. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. keep it up.
    data analytics course in Bangalore

    ReplyDelete
  32. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance

    ReplyDelete
  33. Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article.

    Correlation vs Covariance

    ReplyDelete
  34. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  35. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  36. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one.
    data science training in guwahati

    ReplyDelete
  37. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, best online data science courses

    ReplyDelete
  38. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  39. I will really appreciate the writer's choice for choosing this excellent article appropriate to my matter.Here is deep description about the article matter which helped me more.
    I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts.
    Data Analytics Courses
    I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!

    ReplyDelete
  40. Really nice and interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing
    data science training in mumbai

    ReplyDelete
  41. Attend The Data Science Courses From ExcelR. Practical Data Science Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Courses.
    Data Science Courses

    ReplyDelete
  42. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  43. Just saying thanks will not just be sufficient, for the fantasti c lucidity in your writing. I will instantly grab your rss feed to stay informed of any updates.<a href="https://www.excelr.com/business-analytics-training-in-pune/”> Courses in Business Analytics</a>
    Very nice article, I enjoyed reading your post, very nice share, I want to twit this to my followers. Thanks!.

    ReplyDelete
  44. I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…
    Machine Learning Courses in Pune I really enjoy reading and also appreciate your work.

    ReplyDelete
  45. Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm

    ReplyDelete
  46. Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
    Data Analyst Course

    ReplyDelete
  47. Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Simple Linear Regression
    Correlation vs covariance
    data science interview questions
    KNN Algorithm
    Logistic Regression explained

    ReplyDelete
  48. Data Analytics course PuneI am a new user of this site so here i saw multiple articles and posts posted by this site,I curious more interest in some of them hope you will give more information on this topics in your next articles.
    I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!

    ReplyDelete
  49. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing data science training in Hyderabad

    ReplyDelete
  50. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

    Simple Linear Regression

    Correlation vs covariance

    KNN Algorithm

    Logistic Regression explained

    ReplyDelete
  51. Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm
    Logistic Regression explained

    ReplyDelete
  52. Attend The Data Science Courses From ExcelR. Practical Data Science Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Courses.
    Data Science Courses

    ReplyDelete
  53. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

    data science interview questions

    ReplyDelete
  54. I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…
    Machine Learning Courses I really enjoy reading and also appreciate your work. I read that Post and got it fine and informative. Please share more like that...
    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site.

    ReplyDelete
  55. This is my first time visit here. From the tons of comments on your articles.I guess I am not only one having all the enjoyment right here! ExcelR Data Science Courses

    ReplyDelete
  56. A Computer Scientist figures out how to plan programming frameworks and gains top to bottom learning of the hypothesis of calculationdata scientist courses

    ReplyDelete
  57. Thank you for sharing wonderful information with us.Really useful for everyonedata scientist courses

    ReplyDelete
  58. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  59. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

    data science interview questions

    ReplyDelete
  60. I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…
    Data Science Courses Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained! Cool stuff you have and you keep overhaul every one of us

    ReplyDelete
  61. Very nice blogs!!! i have to learning for lot of information for this sites…Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing, data sciecne course in hyderabad

    ReplyDelete
  62. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  63. very well explained. I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Logistic Regression explained
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm

    ReplyDelete
  64. Very nice blogs!!! i have to learning for lot of information for this sites…Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing, data science courses

    ReplyDelete
  65. There has been a rise in the demand of data science as a subject in the universities, but unfortunately, there is not a particular curriculum which can be followed in this field since it is a very generalized field. data science course syllabus

    ReplyDelete
  66. hello sir,
    thanks for giving that type of information. I am really happy to visit your blog.Leading Solar company in Andhra Pradesh

    ReplyDelete
  67. After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.PMP Certification

    ReplyDelete
  68. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    Data Scientist Course

    ReplyDelete
  69. I Want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging endeavors.
    Business Analytics Course in Bangalore

    ReplyDelete
  70. I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
    Data Analytics Courses in Bangalore

    ReplyDelete