{"id":310569,"date":"2023-01-05T16:03:53","date_gmt":"2023-01-05T15:03:53","guid":{"rendered":"https:\/\/blog.jetbrains.com\/?post_type=big-data-tools&#038;p=310569"},"modified":"2023-01-06T13:39:13","modified_gmt":"2023-01-06T12:39:13","slug":"view-the-current-state-of-variables-in-zeppelin-notebooks","status":"publish","type":"big-data-tools","link":"https:\/\/blog.jetbrains.com\/zh-hans\/big-data-tools\/2023\/01\/05\/view-the-current-state-of-variables-in-zeppelin-notebooks","title":{"rendered":"View the Current State of Variables in Zeppelin Notebooks"},"content":{"rendered":"\n<p>Do you remember a long<a href=\"https:\/\/blog.jetbrains.com\/zh-hans\/idea\/2020\/10\/ztools-for-apache-zeppelin\"> time ago when we introduced<\/a> a feature in the Big Data Tools plugin called <a href=\"https:\/\/github.com\/JetBrains\/ztools\" target=\"_blank\" rel=\"noopener\">ZTools<\/a> that allows you to view the current state of variables in a Zeppelin notebook? Do you remember that we made significant changes to ZTools implementation almost a year ago?<\/p>\n\n\n\n<p>It\u2019s our tradition to make big announcements about this part of <a href=\"https:\/\/jb.gg\/bdt\" target=\"_blank\" rel=\"noopener\">Big Data Tools<\/a> yearly, and today we\u2019re announcing multiple exciting changes. If you&#8217;d rather just check them out&nbsp; yourself, here is the link to the plugin:<\/p>\n\n\n    <iframe loading=\"lazy\" frameborder=\"none\" width=\"245px\" height=\"48px\" src=\"https:\/\/plugins.jetbrains.com\/embeddable\/install\/12494\"><\/iframe>\n\n\n\n\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">New name<\/h2>\n\n\n\n<p>Most importantly, this feature is now called &#8220;State Viewer&#8221;. We decided to change the name because State Viewer can implement variable viewing for other notebooks too, not just for Zeppelin. There were a few other names being considered, and the choice wasn\u2019t easy.&nbsp;<\/p>\n\n\n\n<p>Here is how it looks:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"1600\" height=\"1050\" src=\"https:\/\/blog.jetbrains.com\/wp-content\/uploads\/2023\/01\/image.png\" alt=\"\" class=\"wp-image-310570\"\/><\/figure>\n\n\n\n<p>In the same picture, you can see that the State Viewer is now enabled by default!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Thick server to thick client<\/h2>\n\n\n\n<p>The first significant change we made was the move from a &#8220;thin client&#8221; to a &#8220;thin server&#8221; model. This means that the IDE doesn&#8217;t need to install anything into Zeppelin&#8217;s interpreter. In turn, this is beneficial when you do not fully control the Zeppelin instance.<\/p>\n\n\n\n<p>How does it work? Well, these changes are evolutionary rather than revolutionary. Previously, State Viewer used background cell execution only to call a method from a library we installed into the Zeppelin interpreter. Now the plugin acts differently, executing the data gathering logic from Zeppelin and sending it to the State Viewer window.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"1600\" height=\"618\" src=\"https:\/\/blog.jetbrains.com\/wp-content\/uploads\/2023\/01\/image-1.png\" alt=\"\" class=\"wp-image-310581\"\/><\/figure>\n\n\n\n<p>The code of the hidden (disappearing) cell is quite impressive too. It starts like this:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"1600\" height=\"892\" src=\"https:\/\/blog.jetbrains.com\/wp-content\/uploads\/2023\/01\/image-2.png\" alt=\"\" class=\"wp-image-310592\"\/><\/figure>\n\n\n\n<p>Don\u2019t take our word for it \u2013 check out the code yourself! To do so, enable &#8220;Debug mode&#8221; in the settings:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"450\" height=\"456\" src=\"https:\/\/blog.jetbrains.com\/wp-content\/uploads\/2023\/01\/image-3.png\" alt=\"\" class=\"wp-image-310603\"\/><\/figure>\n\n\n\n<p>This leads us to the following significant change: the State viewer settings have been revamped.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">State viewer settings<\/h2>\n\n\n\n<p>The fully expanded view of State Viewer settings looks like this:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"1136\" height=\"1524\" src=\"https:\/\/blog.jetbrains.com\/wp-content\/uploads\/2023\/01\/image-4.png\" alt=\"\" class=\"wp-image-310614\"\/><\/figure>\n\n\n\n<p>It&#8217;s absolutely massive, right?<\/p>\n\n\n\n<p>As we can see in the previous screenshot, it consists of 3 main parts:<\/p>\n\n\n\n<p>1. Common Interpreter Settings<\/p>\n\n\n\n<p>2. Variable Introspector settings<\/p>\n\n\n\n<p>3. SQL Introspector settings<\/p>\n\n\n\n<p>Hopefully the name &#8220;Common Interpreter Settings&#8221; is self-explanatory. You can enable or disable the collection of info about variables from Spark SQL, and it allows you to enable debug mode. Admittedly this is most useful for us, the plugin developers, but you can use it to get a deeper understanding of the plugin\u2019s inner workings.Variable Introspector Settings<\/p>\n\n\n\n<p>Valuable feedback from our customers has given us an understanding of exciting corner cases when introspecting variables. Variable Introspector Settings addresses all of the potential issues you might encounter. Need to introspect longer texts? Increase the limit of the String size. Now it takes more time to extract them all? Increase the timeout while waiting for Zeppelin to answer. Have to work with deeply nested structures? You can fine-tune the maximum dig depth.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">SQL Introspector settings<\/h3>\n\n\n\n<p>We didn&#8217;t even realize how useful SQL Introspector would be for some of our customers needed to find out how complex usage scenarios can be across lots of schemas and tables, which led us to the next exciting solution:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"694\" height=\"202\" src=\"https:\/\/blog.jetbrains.com\/wp-content\/uploads\/2023\/01\/image-5.png\" alt=\"\" class=\"wp-image-310625\"\/><\/figure>\n\n\n\n<p>Depending on the size of your Spark database, you can use different strategies when you need to pull changes from the Spark database. Sometimes databases are so extensive that we will decide not to pull changes automatically at all.<\/p>\n\n\n\n<p>All of these changes led us to switch on State Viewer by default in this release.<\/p>\n\n\n\n<p>We also learned that when our customers work with Spark, they work with more than just the default catalog. Sometimes, the \u201cdefault\u201d catalog is absolutely massive, and they need to filter data from it somehow. That\u2019s why we introduced the following filter in the SQL Introspector:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"1148\" height=\"272\" src=\"https:\/\/blog.jetbrains.com\/wp-content\/uploads\/2023\/01\/image-6.png\" alt=\"\" class=\"wp-image-310636\"\/><\/figure>\n\n\n\n<p>Here you can filter and add more catalogs to look for data for autocompletion, as well as limit the subset of queried tables inside a catalog.<\/p>\n\n\n\n<p>If you use AWS Glue or Hive Metastore, you might also find this checkbox useful:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" loading=\"lazy\" width=\"400\" height=\"58\" src=\"https:\/\/blog.jetbrains.com\/wp-content\/uploads\/2023\/01\/image-7.png\" alt=\"\" class=\"wp-image-310647\"\/><\/figure>\n\n\n\n<p>Your company\u2019s Glue is likely to be incredibly large, and you only need the data you put there during your session.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">State Viewer is now enabled by default<\/h2>\n\n\n\n<p>We\u2019ve introduced many improvements since our last blog post on this feature. We are very grateful to our customers for their ongoing support and continuous feedback \u2013 it was possible to implement many changes because of them! We are now confident that Big Data Tools is stable and flexible enough for our general audience to be able to use it continuously for these reasons:<\/p>\n\n\n\n<p>1. It no longer requires changes in Zeppelin.<\/p>\n\n\n\n<p>2. It allows you to fine-tune variable introspection to your use case.<\/p>\n\n\n\n<p>3. It allows you to tune SQL introspection according to the context and complexity of your tasks.<\/p>\n\n\n\n<p>4. The mechanics of hidden cells have been improved.<\/p>\n\n\n\n<p>5. We have a method for our users to check and deeply understand the plugin\u2019s functionality. took an extra step to allow our customers to \u201ccheck up on us\u201d and make sure we\u2019re doing only what we say we\u2019re doing, and no more.<\/p>\n\n\n\n<p>If you&#8217;re interested in trying Big Data Tools, you can easily find the plugin <a href=\"https:\/\/jb.gg\/bdt\" target=\"_blank\" rel=\"noopener\">here<\/a>.<\/p>\n","protected":false},"author":1234,"featured_media":311050,"comment_status":"closed","ping_status":"closed","template":"","categories":[],"tags":[7178,7180,91,7177,7182,7179,6320],"cross-post-tag":[],"acf":[],"_links":{"self":[{"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/big-data-tools\/310569"}],"collection":[{"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/big-data-tools"}],"about":[{"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/types\/big-data-tools"}],"author":[{"embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/users\/1234"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/comments?post=310569"}],"version-history":[{"count":10,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/big-data-tools\/310569\/revisions"}],"predecessor-version":[{"id":311091,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/big-data-tools\/310569\/revisions\/311091"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/media\/311050"}],"wp:attachment":[{"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/media?parent=310569"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/categories?post=310569"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/tags?post=310569"},{"taxonomy":"cross-post-tag","embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/cross-post-tag?post=310569"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}