HighLoad++ 2009: IMDG Presentation

On 12th of October I gave a presentation dedicated to concept of In-Memory Data Grids on russian conference dedicated to high load systems. My talk laid little aside interests of most participants, but it seemed to induce some interest. I even get an invitation to do the same talk for employees of Mail.ru corporation.

PRESENTATION RUSSIAN ONLY

HighLoad++ 2009 In-Memory Data Grids

View more presentations from aih1013.

Wonderful analogy between programming & sleeping

Just found a great article about Analogy between sleeping & programming (russian only). When stated, similarity becomes mostly obvious.

Full-text & faceted search over In-Memory-Data-Grid

Modern in memory data-grid solutions provide different facilities for execution of queries over whole stored data set with different levels of sophistication . Coherence provides Query facilities (one time full scan & continuous quering with Cost-Based-Optimized). Gigaspaces has JDBC Query interface with ability to use hash & B-Tree indexes. The solutions work quite well for big part of problem areas. However for heavy loads & complex multi-createrias queries the facilities may become a bottleneck quickly.

But there is a class of workloads producing high query loads onto IMDGs. Item catalogs (in retailing companies for example) are hit by diverse stream of multi-createria queries.Typically query you see likes :

give me cell phones with MP3 support, Java and in red color.

Fortunately, Compass framework allows to do such queries effectively. It allows to builds inverse indexes with Apache Lucene and store them in a grid. The capability originates in very modular design of Lucene framework. All index IO operations hidden by abstraction of FileDirectory.

For now Compass provides implementations for Coherence, GigaSpaces and Terracota introducing unprecedented ability to build a vertical search solution on top of In-Memory DataGrid.



Moreover, Compass has sophisticated object-to-document mapping system which allows to make stored objects searchable just by addition of Java annotations or XML mapping files. Also mapping can be built in runtime.

However, inspite the great code, Composs has scarce documentation. It may take significant time to dive into the code and docs to get what you want. But results will overcome all your expectations. Search engines on top of data grids easily overcome any old-generation search technologies.

Ship It Review

Content of the book can be naturally divided into 3 parts:
- Code of Software Developer
- Advices about development tools and considerations
- Trace Bullet Development
I'll review each part independently.

Part dedicated to relations in developers team and management is absolutely brilliant. It gives wonderful inside to team dynamics, main factors influencing changes and intrateam relations. And most importantly it provides solutions for common problems of day-to-day developer’s life. No doubt, the chapter will be interesting for both newbies and seasoned team leaders.

Another part of the book is dedicated to development tools and processes. There is nothing novel here. In our days every developer knows Version Control System should present. And it would be better to have a continuous integration server if your team has more than 1 guy. So the part can be skipped if you have knowledge about modern development processes.

And the last part is dedicated to so-called Trace Bullet Development (TBD) process. While the wording sounds cool, usefulness of the process is rather arguable. Don't get me wrong; described design process (yes, this is not a Development process, this is a Software Design Process) will work. Creation of vertical of functioning system components is a good starting point for any project. But TBD pushes it forward too much. Interfaces created early tend to be error-prone. And after the interfaces were designed, they tend to become unchangeable. So line by line they become monsters which have no connection with the original elegant idea.

Concept of implementation of architecturally important user-stories or use-cases looks much better for me. Needless to say, as in any book there are several out-dated or even harmful advices. For example, "design all interfaces to be remote" . Such methodology used without appropriate care will make system too complex without real need.

Anyway, the book worth reading. This is the best masterpiece I’ve read since "The One Minute Manager".