Datalore
Collaborative data science platform for teams
Drama & Company Accelerated Data Research by 80% With Datalore
About Drama & Company
As an IT company focused on “connecting business opportunities”, Drama & Company has made a big splash in the career recruitment industry with its flagship app, Remember. With over 4.5 million active users in the Republic of Korea, Remember is a testament to the company’s commitment to using data to drive business opportunities.
Embracing Challenges to Improve Data Products Delivery
Drama & Company has a wealth of user data spanning various areas, including business card information, recruitment history, research figures, and advertising metrics. Unsurprisingly, handling such extensive data came with its own set of challenges.
“When creating an account in the app, users only provide a few bits of personal information. To bring more value to other app users, our team needs to enrich user profiles with data from other sources. Hence, we need a quick and reliable process for data research, processing, refining, and delivery.”
Before, the company used a standalone Jupyter server for its data research needs. However, this setup often faced performance hiccups, especially when certain researchers used a lot of computation resources. Transferring the research results to a data pipeline, distributing the code written by researchers, and other similar tasks proved to be tough and time-consuming.
Given the challenges they were facing, Drama & Company realized the need for a more robust and scalable data science platform. The Big Data Center Team at Drama & Company had two major goals. First, they wanted to improve the quality of their data and expand its range. Second, they aimed to increase data utilization across the business to make their organization more competitive.
Their requirements for the new data science platform included:
- Easy access to their data engineering infrastructure on AWS, including integrations with Amazon Glue, Amazon Athena, Amazon S3, and Amazon AuroraDB.
- Seamless code reviews between researchers who are not comfortable using Git and data engineers
- Personalized work environments for each researcher to promote collaboration.
- Support for an on-premises installation with a secure authentication process
- A system that allowed tasks to keep running no matter the browser status.
After reviewing various options, Drama & Company chose Datalore. It was available as an on-premises installation with SSO authentication in-place, offered integrations with AWS data sources, had embedded easy-to-use notebook versioning, real-time collaboration on notebooks and in team workspace, along with a Background-computation option which allowed running notebooks even when the browser is switched off. Moreover, JetBrains was a provider Drama & Company already trusted.
Redefining Collaboration and Building Synergy With Datalore
Through Datalore, Drama & Company has refined its data research and management. Collaboration is key at Drama & Company. Their team has four squads: Data Planning & Engineering, Data Reliability Engineering, Biz Card, and Data Platform Engineering. Each squad has its own role, but they all work together very closely.
“Datalore’s intuitive interface and robust functionalities empower us to seamlessly explore and analyze datasets. This drives increased productivity and promotes informed decision-making across the team.”
Here’s how a typical workflow looks with Datalore at Drama & Company:
- First, as soon as a project kicks off, researchers and data engineers jointly create a separate workspace for it and refine the research goals and methodology.
- Second, data engineers set up the necessary data analysis environment with required SQL databases and S3 buckets in the project workspace.
“Having no-code connections to Amazon Glue, Athena, S3, and AuroraDB, as well as the opportunity to combine SQL and Python in one notebook, helped us cut down the resources needed for data lookup, verification, and rework, resulting in faster research times.”
Youngrae Lee, Big Data Center Team Lead at Drama & Company
- Third, the researchers conduct their data analysis in Datalore notebooks and pass over their research for a code review by a data engineer. Data engineers work together with the researcher to make real-time changes or corrections to the code if needed. This process also serves as a learning experience for the researchers, helping them improve their writing of code for efficient data infrastructure.
- Last, if the analyzed data needs to be moved immediately to a production environment, the data engineer downloads the notebook file and handles the pipelining. If data needs further validation, the researchers use Datalore’s Scheduling feature to test the notebook for more extended periods of time on new data chunks.
“One of the best things about Datalore for the team is that, unlike Jupyter, their work doesn’t stop even if they’ve stepped away from the browser. This means they can confidently leave their work and know the heavy notebook computation continues in the background.”
Youngrae Lee, Big Data Center Team Lead at Drama & Company
Results
The adoption of Datalore has led to a significant increase in the team’s efficiency. Notably, collaboration between various subsets of the data team has improved, reducing the time spent on common tasks by over 50%. Thanks to smooth data integrations, improved code reviews, and faster iteration cycles, data analysis has sped up by 80%, significantly enhancing the overall capacity of the team members.
“Collaborating with my team has become significantly more efficient thanks to Datalore. The platform excels in sharing and analyzing data and code, allowing for seamless collaboration and review. Unlike our previous tools, Datalore enables us to split notebooks into separate sheets. This functionality allows us to work on different stages of data processing in distinct sheets, significantly improving the readability and structure of our analysis reports.”
These significant productivity gains have enabled the Big Data Center Team to successfully improve the quality and scope of data products, thereby bolstering their business competitiveness through more effective data utilization.
“I think many companies have the same concerns as us. I hope many companies will experience maximizing and streamlining collaboration from analysis to data development through Datalore and share a lot of know-how.”
Subscribe to Datalore News and Updates
