• GridGain 3.6 Released!

    It’s was Friday 13th… but there’s nothing scary about the latest GridGain 3.6 – our latest release of GridGain’s Real Time Big Data platform that allows anyone easily develop, scale and manage compute and data intensive big data applications using integrated Compute and In-Memory Data Grids with Java, Scala and Groovy native APIs.

    Some of the key new features and enhancements:

    • “Write From Behind Caching” was finally added to IMDGs
    • REST batch operation support introduced (putAll, removeAll)
    • Eviction filters added
    • Segmentation handling for half-connected network sockets
    • Improvements in GridGain Visor
    • New eviction performance optimizations
    • Significant IMDG preloading performance enhancements
    • New optimization for lock-free reads and low contention writes
    • Cumulative bug fixes and multitude of performance

    If you don’t have one – get one!

  • Offbeat: Scala by the end of 2011 – No Drama but Frustration is Growing

    Scala

    I wanted to put our two cents when Yammers’ personal email was leaked week or so ago but kind of waited to let the dust settle. We are active Scala developers with several years and production code behind us – and so we have this rare “real live” experience with Scala…

    In two years we went from having Scala as a very strategic choice to have it simply as an interesting (and mostly enjoyable) technology that we use to develop some (and rather small) parts of our platform. I still think, as I always did, that Scala is wonderfully elegant and relatively simple language. For the first time someone has combined OOP and FP in a extremely practical general purpose programming language.

    But our team’s enthusiasm significantly decreased over the last 12 months as it became obviously clear that Scala won’t be the eventual Java replacement for most of the enterprise Java development. Moreover, more and more people are starting to painfully realize that Scala won’t even make any significant dent on Java eco-system at all – remaining forever in 1-2% territory for very specialized projects where FP focus or DSL capabilities override any other requirements. With Java 8 coming up quickly, with Kotlin, Ceylon and Clojure nibbling on the edges (even if some of them don’t yet exist!), Groovy going strong – Scala simply doesn’t have much room to grow or maneuver.

    Not being widely used means little interest for us, independent ISVs, to provide support and integration for it.

    2011 was yet another clear indication that Scala eco-system is experience growth pains:

    • Typesafe perceived troubles and lack of direction
    • Ridicule about parallel collections in 2.9 (one of its major new features)
    • Public “I’m fed up” email leaks
    • SBT vs. Maven, Lift vs. Play, Scalaz vs. common sense…
    • Scrutiny of few larger projects exposing rather alarming ineptitude in: scala.io, scala.xml, scala.actor, parallel collections, scaladoc implementation
    • Glaring performance holes in the compiler known for years yet still ignored
    • Compilation speed, binary backward compatibility woes continued (year after year)
    • Utter lack of attention to API documentation – “read the source code instead” mantra
    • IDEs integration

    Some of this can be, of course, explained by the fact that most of the Scala eco-system was originally developed by University postdocs and students that lack non-academia experience in building commercial software (of that size and magnitude). Be it as it may – most of us would still expect the better outcome… On the other hand Typesafe seems to be pushing the language development and maintenance onto the community again and that scares me (read above) – preferring rather to concentrate on the Stack.

    And then – there is a community… Oh – how I wish sometimes that Scalaz would never exist (along with some of the idiot savants behind it)! This Scala-wannabe-Haskel library is probably single-handedly responsible for a big share of “Scala is too complex” syndrome that sealed the fate of Scala at least for now. Having almost zero engineering applicability in the real world – it somehow inexplicably defined the public’s perception on Scala’s usability. While being interesting academically (and written by very smart guys indeed) – I wished it would never exist. Really.

    Having said all that – I still love Scala and that’s what I use almost exclusively when I write code for GridGain (still do!). Heck – we even use one trait from Scalaz! Despite the numerous growth problems – I still can’t resist Scala elegance and almost an algebraic simplicity and pragmatism that is so rare these days. It’s the combination of simplicity of the language and the power it gives to developer that attracts me to Scala.

    My wish list for Scala in 2012 thus is pretty simple: stop the bleeding, no new boneheaded features, and plenty of bug fixes and core hardening.

  • GridGain’s Round A Funding

    We announced this week that we’ve closed round A funding for our company (see the official press release) led by RTP Ventures. This is a significant milestone for us and clear manifestation of how far we’ve come as a project and as a company to this date.

    GridGain has always been an engineering company first and foremost – and will remain so going forward. We’ve pushed back on getting funded for as long as we could to make sure we build a product and prove it in the market place first. Getting institutional investment means agreeing on much more articulated business rhythm and we wanted to have the flexibility to pause, reassess, pivot and make sure we develop GridGain platform organically just way we see it from the engineering stand point.

    By the mid of 2011 we’ve reached the point where we believed we needed to get proper organization in place as we were swamped with new business. Few months later we’ve got several terms sheets on the table.

    As we look forward into the next year GridGain 4.0 in Q112 will be one of our the most significant product announcements further solidifying our technology lead in Real Time Big Data processing. Stay tuned!

    And by the way… WE ARE HIRING both in Saint Petersburg, Russia and Foster City, CA.

  • GridGain is hiring!

    GridGain

    GridGain Systems is rapidly growing and we are hiring talented Java/Scala top-level engineers in both of our locations:
    - Saint Petersburg, Russia
    - Foster City, CA, USA

    Come work for one of the hotest startups in the Silicon Valley! Cutting edge work and top level benefits and compensation. We also guarantee sleepless nights, unsolvable problems and insane debugging of complex distributed systems just in the first week :)

    All the details are here.

  • GridGain 3.5 Released!

    We’ve just released GridGain 3.5 – our latest stable release of GridGain’s Real Time Big Data platform that allows anyone easily develop, scale and manage compute and data intensive JVM based applications using integrated Compute and In-Memory Data Grid with Java, Scala and Groovy native APIs.

    Some of the key new features and enhancements:

    • Significant performance improvements throughout the product
    • Previously deprecated APIs have been removed
    • Improvements and bug fixed in GridGain Visor
    • Enhancements to GridProjection interface
    • Ability to programmatically start remote nodes in bulk
    • Customizable closure-based MapReduce
    • Enhancements in affinity and co-location
    • Bug fixed in eviction policies in Data Grid
    • Significant improvements and performance enhancements in Swap SPI

    Download GridGain 3.5 today!

  • GridGain presents at 1DevDay, Detroit, MI on November 5th

    1DevDay

    GridGain will be presenting it’s “Live Coding” talk at 1DevDay in Detroit, MI on November 5th. If you are interested in “Functional Cloud Computing with Scala and GridGain” – come by and listen to the talk. No slides, only live Scala coding of distributed applications for 60 minutes. Several MapReduce and Data Grid apps from scratch and working right in front of your eyes.

    Hope to you see you there!

  • Dennis Ritchie – RIP

    I’ve got Dennis Ritchie’s (and Brian Kernighan’s) book “The C Programming Language” when I was 15. I remember it quite well because I got in English (it wasn’t translate yet to Russian) and it took me a while to read it (often with vocabulary). It took me a lot longer to learn C and follow it through coding… I’ve had AT&T 7300 Unix workstation at home that I was hacking everyday after school. Graphics were pretty pail comparing to Amiga/Atari (which interestingly were all based on the same Motorola 68000 CISC processor) – but it was genuine Unix environment and I believe it’s what got me hooked every since.

    I think that book along with Rob Pike’s (and again along with Brian Kernighan) book “The Unix Programming Environment” were the two most influential books in my early years – if not overall. Somehow they opened up the inner-beauty and almost an mathematical elegance of Unix and its main language – C language.

    Since I was 15 till early 20 I was writing programs exclusively in C/C++ and Unix – thanks to wonderful language created by Dennis Ritchie and his equally wonderful book.

    It’s sad to see the legend go…

  • PDC Theorem – The Underlying Principle of Real-Time Distributed Processing

    Few months ago I blogged about real-time processing in enterprise systems of all stripes, and since then I’ve talked quite a bit about the underlying principle of such processing. I call it a PDC theorem as in “Processing, Data, and Co-Location”.

    The idea behind PDC is pretty simple. Real-time response in a highly distributed system is not achievable unless the following 3 rules are followed:

    • Processing must be distributable for in-memory computation
    • Data storage must be distributable (i.e. partitioned) for in-memory storage
    • Co-location must be ensured between processing and data units to provide locality of remote operations

    Few important notes:

    • We, of course, are talking about business or perceptual real-time (a.k.a. Near Real-Time or nRT) and not about hardware real-time. Perceptual real-time response is not well defined but you can conceptually visualize it as the time the user of the system willing to wait for the response that he or she expects right away… In most cases it means few seconds or less. In rarer cases like FOREX trading, for example, the real-time would mean microseconds.
    • It is critically important that your processing supports algorithmic parallelization. Not all tasks can be parallelized and therefore not all tasks can be optimized for real-time processing. However, many of the typical business and social graph tasks can be split into multiple sub-tasks executing in parallel – and therefore are trivially parallelizable.
    • Data have to be partitioned and stored in-memory. Any outside calls to get data from NoSQL storage, file systems like HDFS, or traditional SQL storage renders any real-time attempts useless in most cases. This is one of the most critical design element and it is often overlooked. In other words – in no time the remote processing should escape the boundaries of the local JVM it is executing on.
    • Co-location of the processing and data (a.k.a affinity-based routing referring to the fact that should be an affinity between the computation and the data this computation needs) is the main mechanism to ensure that there is no noise data transfer between remote nodes where a task is being processed. Such unnecessary data transfer will violate the locality principle of the remote operations making real-time processing often unachievable.

    It’s also quite obvious that PDC theorem doesn’t guarantee the real-time processing – it merely states these three rules are necessary but not enough on their own for a real-time response. Latencies of the atomic remote operations will often dictate whether or not real-time response is achievable in practice.

    It is interesting to note that combination of PDC and CAP theorems really defines the fundamentals of high performance distributed programming today.

Follow

Get every new post delivered to your Inbox.

Join 403 other followers