This post is the first in a series about Big Data. In it, we’d like to tell you how we at JetBrains see Big Data, and consequently, how we’re creating products for it.<\/p>\n\n\n\n
Next parts:<\/p>\n\n\n\n
Table of contents:<\/p>\n\n\n\n
The world of big data can seem mysterious, hidden behind a curtain of unknown and weird words. It\u2019s time to clear up this mystery and define Big Data.<\/p>\n\n\n\n
As any term that has been overhyped at some point, the term \u201cBig Data\u201d has become convoluted with vast meaning. I will use the three definitions that I feel are most accurate:<\/p>\n\n\n\n
This is dependent on each piece of hardware, so we can\u2019t define a universal, static value for what constitutes \u201cbig data\u201d. I remember my ancient Intel 80386 \u2013 its 16 MB memory meant that anything more than 8 MB would be classed as \u201cbig data\u201d.<\/p>\n\n\n\n
100 MB of data looks small now, but it was considered huge in the past and required sophisticated algorithms to process.<\/p>\n\n\n\n
Today, Big Data is much bigger in absolute terms, but still requires sophisticated processing, distributed computing, and special storage formats.<\/p>\n\n\n\n
3V (pronounced as triple-v) stands for Volume, Velocity, and Variety. Scaling on 3V means that you won\u2019t have to re-architecture your storage, jobs, and processes if volume, velocity, or variety will grow, say, ten times.<\/p>\n\n\n\n
It\u2019s hard to say what \u201cten times\u201d means in terms of variety, but data tends to change frequently and rapidly in terms of form and velocity.<\/p>\n\n\n\n
As you might have guessed, this definition is primarily determined by software.<\/p>\n\n\n\n
Let\u2019s not forget why data, big or small, matters in the first place \u2013 to do business. Taking this into consideration, defining \u201cBig Data\u201d in terms of business applications is useful.<\/p>\n\n\n\n
Successful businesses are almost always data driven, and usually focus on making business reliable, predictable, and consistent. Doing these things well, however, requires more data than merchants had during, say, the Middle Ages. The modern business model, user-centric, and working with each person differently, is not possible without large amounts of data<\/p>\n\n\n\n
For example, most big e-commerce companies have huge clickstreams (streams of user-generated events) based on marketing that predicts which goods will be more popular than others.<\/p>\n\n\n\n
Now that we understand what \u201cBig Data\u201d is, let\u2019s try to understand who the consumers are.<\/p>\n\n\n\n
There are three main categories of internal customers:<\/p>\n\n\n\n
Management needs reports to understand what\u2019s going on in the company, improve existing plans, and create new plans.<\/p>\n\n\n\n
Product managers want to improve their products through experimentation and need data to analyze the results of experiments and propose new ideas.<\/p>\n\n\n\n
Marketing needs data to analyze marketing metrics, such as COA (cost of acquisition), LTV (lifetime value), and so on. They also need data to build successful marketing companies.<\/p>\n\n\n\n
This is how we understand what big data is and who consumes the results of working with big data.<\/p>\n\n\n\n
At JetBrains, our main projects for big data include:<\/p>\n\n\n\n