{"id":34021,"date":"2020-03-03T22:39:53","date_gmt":"2020-03-03T13:39:53","guid":{"rendered":"https:\/\/blog.jetbrains.com\/kr\/?p=610"},"modified":"2020-03-03T22:39:53","modified_gmt":"2020-03-03T13:39:53","slug":"update-on-big-data-tools-plugin-spark-hdfs-parquet-and-more-ko","status":"publish","type":"post","link":"https:\/\/blog.jetbrains.com\/ko\/2020\/03\/03\/update-on-big-data-tools-plugin-spark-hdfs-parquet-and-more-ko\/","title":{"rendered":"Big Data Tools \ud50c\ub7ec\uadf8\uc778 \uc5c5\ub370\uc774\ud2b8: Spark, HDFS, Parquet \uc9c0\uc6d0 \ubc0f \uae30\ud0c0 \ub2e4\uc591\ud55c \uae30\ub2a5 \uc81c\uacf5"},"content":{"rendered":"\ub9c8\uc9c0\ub9c9 \uc5c5\ub370\uc774\ud2b8 \uc774\ud6c4\ub85c \uc624\ub79c \uc2dc\uac04\uc774 \uc9c0\ub0ac\uc2b5\ub2c8\ub2e4. \uae30\uc5b5\ud558\uc2e4\uc9c0 \ubaa8\ub974\uaca0\uc9c0\ub9cc \uc791\ub144\uc5d0 \uc800\ud76c\ub294 IntelliJ IDEA\uc640 Apache Zeppelin, S3\uc640\uc758 \ud1b5\ud569, \uadf8\ub9ac\uace0 Apache Spark\uc640\uc758 \uc2e4\ud5d8\uc801 \ud1b5\ud569\uc744 \ubc1c\ud45c\ud588\uc2b5\ub2c8\ub2e4. Apache Spark\uc640\uc758 \ud1b5\ud569 \uc81c\ud488\uc740 \uc2e4\ud5d8\uc801\uc778 \uae30\ub2a5\uc73c\ub85c \ucd9c\uc2dc\ub418\uc5b4 \ubd88\uc548\uc815\ud55c \uc5c5\ub370\uc774\ud2b8 \ucc44\ub110\uc5d0\uc11c\ub9cc \uc0ac\uc6a9\ud560 \uc218 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. \ud558\uc9c0\ub9cc \uc88b\uc740 \uc18c\uc2dd\uc774 \uc788\uc2b5\ub2c8\ub2e4. \uc624\ub298 \ub4dc\ub514\uc5b4 Spark \uc9c0\uc6d0\uc744 \uacf5\uac1c\uc801\uc73c\ub85c \uc81c\uacf5\ud558\ub294 \uc0c8\ub85c\uc6b4 \ubc84\uc804\uc758 \ud50c\ub7ec\uadf8\uc778\uc774 \ucd9c\uc2dc\ub429\ub2c8\ub2e4. \uc774 \ubc84\uc804\uc5d0\uc11c\ub294 HDFS \ubc0f Parquet\ub3c4 \uc9c0\uc6d0\ub429\ub2c8\ub2e4.\r\nSpark \ubaa8\ub2c8\ud130\ub9c1\r\nSpark \ud1b5\ud569 \uc81c\ud488\uc774 \uacf5\uac1c \uc5c5\ub370\uc774\ud2b8\ub85c \uc81c\uacf5\ub418\uc5c8\uc73c\uba70, \uadf8 \uae30\ub2a5\uc5d0 \uad00\ud574 \uac04\ub7b5\ud558\uac8c \uc54c\ub824 \ub4dc\ub9ac\uaca0\uc2b5\ub2c8\ub2e4.\r\nSpark \uc791\uc5c5\uc744 \ubaa8\ub2c8\ud130\ub9c1\ud558\ub824\uba74 \ub2e4\uc74c\uacfc \uac19\uc774 Big Data Tools Connections(\uc5f0\uacb0) \uc124\uc815\uc73c\ub85c \uc774\ub3d9\ud558\uc5ec Spark History Server\uc758 URL\uc744 \ucd94\uac00\ud558\uae30\ub9cc \ud558\uba74 \ub429\ub2c8\ub2e4.\r\n\r\n\ucd94\uac00\ud558\uace0 \ub098\uc11c \uc124\uc815\uc744 \ub2eb\uace0 IDE \ucc3d\uc758 \uc6b0\uce21 \ud558\ub2e8\uc5d0 \uc788\ub294 Spark \ub3c4\uad6c \ucc3d\uc744 \uc5ec\uc138\uc694. Spark \ub3c4\uad6c \ucc3d\uc740 \uc644\ub8cc\ub418\uac70\ub098 \uc2e4\ud589 \uc911\uc778 Spark \uc560\ud50c\ub9ac\ucf00\uc774\uc158 \ubaa9\ub85d(Applications(\uc560\ud50c\ub9ac\ucf00\uc774\uc158) \ud0ed\uc5d0 \uc788\uc73c\uba70 \uae30\ubcf8\uc801\uc73c\ub85c \uc811\ud600 \uc788\uc74c)\uacfc \uc791\uc5c5 \ubaa9\ub85d, \uc138\ubd80 \uc791\uc5c5\uc744 \ud45c\uc2dc\ud569\ub2c8\ub2e4.\r\n\r\nExecutors(\uc2e4\ud589\uc790) \ud0ed\uc744 \ud074\ub9ad\ud558\uba74 \ud65c\uc131 \ubc0f \ube44\ud65c\uc131\ud654\ub41c \uc2e4\ud589\uc790\uc5d0 \uad00\ud55c \uc815\ubcf4\ub97c \ub2e4\uc74c\uacfc \uac19\uc774 \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.\r\n\r\n\ud604\uc7ac SQL \ud0ed\uc5d0\ub294 \ucd5c\uadfc \uac80\uc0c9\uc5b4 \ubaa9\ub85d\uc774 \ud45c\uc2dc\ub418\uc9c0\ub9cc \uc2e4\uc81c SQL\uc740 \uc544\uc9c1 \ud3ec\ud568\ub418\uc9c0 \uc54a\uc2b5\ub2c8\ub2e4. \ub610\ud55c Spark\uc640 \ud568\uaed8 Kerberos\ub97c \uc0ac\uc6a9\ud558\ub294 \uacbd\uc6b0 IDE\uc5d0\uc11c \uc11c\ubc84 \uc5f0\uacb0\uc744 \ud5c8\uc6a9\ud558\uc9c0 \uc54a\uc744 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \ubb38\uc81c\ub294 \ub2e4\uc74c \uc5c5\ub370\uc774\ud2b8 \uc911\uc5d0\uc11c \ud574\uacb0\ub418\ub3c4\ub85d \uc791\uc5c5\ud558\uace0 \uc788\uc2b5\ub2c8\ub2e4. Kerberos\ub97c \uc0ac\uc6a9\ud558\uace0 \uacc4\uc2e0 \uacbd\uc6b0 \uc54c\ub824\uc8fc\uc2dc\uba74 \uc774 \uc791\uc5c5\uc744 \ub2e4\ub978 \uac83\ubcf4\ub2e4 \uc6b0\uc120\uc21c\uc704\uc5d0 \ub450\uaca0\uc2b5\ub2c8\ub2e4.\r\nHDFS\r\n12\uc6d4\uc5d0 \ub3c4\uc785\ud55c S3 \uc9c0\uc6d0\uacfc \ub9c8\ucc2c\uac00\uc9c0\ub85c \uc774\uc81c \uc774 \ud50c\ub7ec\uadf8\uc778\uc744 \ud1b5\ud574 HDFS \uc11c\ubc84\uc5d0 \uc5f0\uacb0\ud558\uc5ec IDE\uc5d0\uc11c \ud30c\uc77c\uc744 \ud0d0\uc0c9\ud558\uace0 \uad00\ub9ac\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc774 \uae30\ub2a5\uc744 \uc0ac\uc6a9\ud558\ub824\uba74 \ub2e4\uc74c\uacfc \uac19\uc774 Big Data Tools Connections(\uc5f0\uacb0) \uc124\uc815\uc73c\ub85c \uc774\ub3d9\ud558\uc5ec HDFS \uad6c\uc131\uc744 \ucd94\uac00\ud558\uba74 \ub429\ub2c8\ub2e4.\r\n\r\n\ud604\uc7ac\ub294 \ub8e8\ud2b8 \uacbd\ub85c\uc640 \uc11c\ubc84\uc5d0 \uc5f0\uacb0\ud558\ub294 \ubc29\ubc95(\uad6c\uc131 \ud30c\uc77c \ub514\ub809\ud130\ub9ac \ub610\ub294 \uba85\uc2dc\uc801 URI)\uc744 \uc9c0\uc815\ud574\uc57c \ud569\ub2c8\ub2e4.\r\nHDFS \uc11c\ubc84\ub97c \uad6c\uc131\ud558\uba74 Big Data Tools \ub3c4\uad6c \ucc3d\uc5d0 \ud574\ub2f9 \uc11c\ubc84\uac00 \ub2e4\uc74c\uacfc \uac19\uc774 \ud45c\uc2dc\ub429\ub2c8\ub2e4(\uad6c\uc131\ud55c \uacbd\uc6b0 Apache Zeppelin Notebook \ubc0f S3 \ubc84\ud0b7 \uc606\uc5d0 \ud45c\uc2dc\ub428).\r\n\r\nBig Data Tools \ub3c4\uad6c \ucc3d\uc5d0\ub294 \uad6c\uc131\ub41c \uc11c\ubc84\uc5d0 \uc800\uc7a5\ub41c \ud30c\uc77c \ubc0f \ud3f4\ub354\uac00 \ud45c\uc2dc\ub429\ub2c8\ub2e4. S3\uc758 \uacbd\uc6b0\uc640 \ub9c8\ucc2c\uac00\uc9c0\ub85c HDFS\uc758 CSV \ubc0f Parquet \ud30c\uc77c\uc744 \ud2b8\ub9ac\uc5d0\uc11c \ud655\uc7a5\ud558\uc5ec \ud30c\uc77c \uc2a4\ud0a4\ub9c8\ub97c \ud45c\uc2dc\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ud30c\uc77c \ub610\ub294 \ud3f4\ub354\uc5d0\uc11c \ucee8\ud14d\uc2a4\ud2b8 \uba54\ub274\ub97c \uc2e4\ud589\ud558\uba74 \ub2e4\uc591\ud55c \uc561\uc158\uc774 \uc81c\uacf5\ub429\ub2c8\ub2e4.\r\n\r\n\uc774 \uc635\uc158\uc744 \uc0ac\uc6a9\ud558\uba74 \uc5d0\ub514\ud130\uc5d0\uc11c \ud30c\uc77c\uc744 \uad00\ub9ac\ud558\uac70\ub098 \ub85c\uceec \ucef4\ud4e8\ud130\ub85c \ubcf5\uc0ac\ud558\uac70\ub098 \ubbf8\ub9ac \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ubbf8\ub9ac\ubcf4\uae30\ub97c \uc0ac\uc6a9\ud558\uba74 \ud30c\uc77c \ub0b4\uc6a9\uc744 \ucef4\ud4e8\ud130\uc5d0 \uc644\uc804\ud788 \ubcf5\uc0ac\ud558\uc9c0 \uc54a\uace0\ub3c4 \ud30c\uc77c \ub0b4\uc6a9\uc758 \uccab \ubc88\uc9f8 \uccad\ud06c\ub97c \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4.\r\nParquet\r\n\uc704\uc5d0\uc11c \uc5b8\uae09\ud588\ub4ef\uc774 \uc774\ubc88 \uc5c5\ub370\uc774\ud2b8\uc5d0\uc11c\ub294 Parquet \ud30c\uc77c\uc5d0 \ub300\ud55c \uc870\uae30 \uc9c0\uc6d0\uc744 \uc81c\uacf5\ud569\ub2c8\ub2e4. \uc774\uc81c IDE\uc5d0\uc11c Parquet \ud30c\uc77c\uc744 \uc5f4\uace0 \uadf8 \ub0b4\uc6a9\uc744 \ub2e4\uc74c\uacfc \uac19\uc774 \ud14c\uc774\ube14\ub85c \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4.\r\n\r\nParquet \ud30c\uc77c\uc744 \uc5f4\uba74 \ud50c\ub7ec\uadf8\uc778\uc5d0\uc11c \uc804\uccb4 \ub0b4\uc6a9\uc774 \uc544\ub2cc \uccab \ubc88\uc9f8 \ubd80\ubd84\ub9cc \ud45c\uc2dc\ud569\ub2c8\ub2e4. \uc774\ub294 \ub9e4\uc6b0 \ud070 \ud30c\uc77c\ub85c \uc791\uc5c5\ud560 \ub54c \ud2b9\ud788 \uc720\uc6a9\ud569\ub2c8\ub2e4.\r\nSpark\uc640 \ub9c8\ucc2c\uac00\uc9c0\ub85c \ud30c\uc77c\uc5d0 \uc561\uc138\uc2a4\ud558\ub824\uba74 \uc11c\ubc84\uc5d0 \ubb3c\ub9ac\uc801\uc73c\ub85c \uc561\uc138\uc2a4\ud574\uc57c \ud569\ub2c8\ub2e4. \uc989, \uc11c\ubc84\uac00 SSH \ud130\ub110 \ub4a4\uc5d0 \uc788\ub294 \uacbd\uc6b0 \ud130\ub110\uc744 \uc9c1\uc811 \uc124\uc815\ud574\uc57c \ud569\ub2c8\ub2e4. \ud30c\uc77c\uc5d0 \uc561\uc138\uc2a4\ud560 \ub54c \ubb38\uc81c\ub098 \ubd88\ud3b8\ud55c \uc0ac\ud56d\uc774 \uc788\ub294 \uacbd\uc6b0 \uc54c\ub824\uc8fc\uc138\uc694. \uc5ec\ub7ec\ubd84\uaed8\uc11c \uc54c\ub824\uc8fc\uc2dc\uc9c0 \uc54a\uc73c\uba74, \uc800\ud76c\ub294 \uc544\uc9c1 \uc9c0\uc6d0\ub418\uc9c0 \uc54a\ub294 \ud2b9\uc815 \uc2dc\ub098\ub9ac\uc624\ub97c \uc778\uc9c0\ud558\uc9c0 \ubabb\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc758\uacac\uc744 \ube68\ub9ac \uc81c\uacf5\ud574 \uc8fc\uc2e4\uc218\ub85d \uc88b\uc2b5\ub2c8\ub2e4.\r\n\uc624\ub298\uc740 \uc5ec\uae30\uae4c\uc9c0\uc785\ub2c8\ub2e4. \uc9c0\uae08\uae4c\uc9c0 \uc5c5\ub370\uc774\ud2b8 \uc815\ubcf4\ub294 Scala \ube14\ub85c\uadf8\uc5d0\ub3c4 \uac8c\uc2dc\ub418\uc5c8\uc73c\uba70 \uc774 \uae00\uc740 IntelliJ IDEA \ube14\ub85c\uadf8\uc5d0 \ucc98\uc74c\uc73c\ub85c \uac8c\uc2dc\ub41c \uc5c5\ub370\uc774\ud2b8 \uc18c\uc2dd\uc785\ub2c8\ub2e4. \uac8c\uc2dc \uc704\uce58\uac00 \ubc14\ub010 \uc774\uc720\ub294 \uc774\uc81c \uc774 \ud50c\ub7ec\uadf8\uc778\uc5d0\uc11c Apache Zeppelin\uacfc Scala \uc9c0\uc6d0\ub9cc \uc81c\uacf5\ud558\ub294 \uac83\uc774 \uc544\ub2c8\ub77c \ube45\ub370\uc774\ud130 \uc791\uc5c5\uc744 \uc704\ud55c \ud6e8\uc52c \ub2e4\uc591\ud55c \ub3c4\uad6c\ub97c \ud1b5\ud569\ud558\uae30 \ub54c\ubb38\uc785\ub2c8\ub2e4.\r\n\uc774 \uc5c5\ub370\uc774\ud2b8\uc5d0\uc11c \uc791\uc5c5\ub41c \uc804\uccb4 \ubc84\uadf8 \uc218\uc815 \ubaa9\ub85d\uc744 \ud655\uc778\ud558\ub824\uba74 \ub9b4\ub9ac\uc2a4 \uc815\ubcf4\ub97c \ud655\uc778\ud574\uc8fc\uc138\uc694.\r\n\ub9c8\uc9c0\ub9c9\uc73c\ub85c \ud50c\ub7ec\uadf8\uc778 \uae30\ub2a5\uc758 \uc0ac\uc6a9 \ubc29\ubc95\uc5d0 \uad00\ud55c \ub3c4\uc6c0\uc774 \ud544\uc694\ud558\uc2e0 \uacbd\uc6b0 \ubb38\uc11c\ub97c \ud655\uc778\ud574 \uc8fc\uc138\uc694. \uadf8\ub798\ub3c4 \uc5ec\uc804\ud788 \ub3c4\uc6c0\uc774 \ud544\uc694\ud558\uc2e0\uac00\uc694? \uc774\uacf3\uc758 \ub313\uae00\uc774\ub098 Twitter\uc5d0\uc11c \uc5b8\uc81c\ub4e0\uc9c0 \uba54\uc2dc\uc9c0\ub97c \ub0a8\uaca8 \uc8fc\uc138\uc694.\r\nP.S.: \uc774 \ud50c\ub7ec\uadf8\uc778\uc740 \uc544\uc9c1 \uac1c\ubc1c \ucd08\uae30 \ub2e8\uacc4\uc5d0 \uc788\uc73c\ubbc0\ub85c \uc5ec\ub7ec \ud1b5\ud569 \uae30\ub2a5\uc5d0\uc11c \ubaa8\ub4e0 \uc2dc\ub098\ub9ac\uc624\ub97c \uc9c0\uc6d0\ud558\uc9c0 \uc54a\uc744 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uadf8\ub807\uae30 \ub54c\ubb38\uc5d0 \uc9c0\uae08 \uc5ec\ub7ec\ubd84\uc758 \uc758\uacac\uc774 \ub9e4\uc6b0 \ud544\uc694\ud569\ub2c8\ub2e4. \uc911\uc694\ud55c \uc0ac\uc6a9\uc790 \uc2dc\ub098\ub9ac\uc624(\uc608: \ud2b9\uc815 \uc778\uc99d \uc720\ud615 \ub610\ub294 \uae30\ud0c0 \ud2b9\uc815 \uc0c1\ud669)\uac00 \uc9c0\uc6d0\ub418\uc9c0 \uc54a\ub294 \uac83\uc744 \ud655\uc778\ud558\uc2e0 \uacbd\uc6b0 \uc774\uacf3\uc758 \ub313\uae00, \uc774\uc288 \ud2b8\ub798\ucee4 \ub610\ub294 \uc758\uacac \uc124\ubb38\uc870\uc0ac\ub97c \ud1b5\ud574 \uc54c\ub824\uc8fc\uc138\uc694.\r\n\r\n\ubcf8\ubb38\uc740 Andrey Cheptsov\uc758 Update on Big Data Tools Plugin: Spark, HDFS, Parquet and More\ub97c \ubc88\uc5ed\ud558\uc5ec \uae30\uc7ac\ud55c \uae00\uc785\ub2c8\ub2e4.","protected":false},"excerpt":{"rendered":"\ub9c8\uc9c0\ub9c9 \uc5c5\ub370\uc774\ud2b8 \uc774\ud6c4\ub85c \uc624\ub79c \uc2dc\uac04\uc774 \uc9c0\ub0ac\uc2b5\ub2c8\ub2e4. \uae30\uc5b5\ud558\uc2e4\uc9c0 \ubaa8\ub974\uaca0\uc9c0\ub9cc \uc791\ub144\uc5d0 \uc800\ud76c\ub294 IntelliJ IDEA\uc640 Apache Zeppelin, S3\uc640\uc758 \ud1b5\ud569, \uadf8\ub9ac\uace0 Apache Spark\uc640\uc758 \uc2e4\ud5d8\uc801 \ud1b5\ud569\uc744 \ubc1c\ud45c\ud588\uc2b5\ub2c8\ub2e4. Apache Spark\uc640\uc758 \ud1b5\ud569 \uc81c\ud488\uc740 \uc2e4\ud5d8\uc801\uc778 \uae30\ub2a5\uc73c\ub85c \ucd9c\uc2dc\ub418\uc5b4 \ubd88\uc548\uc815\ud55c \uc5c5\ub370\uc774\ud2b8 \ucc44\ub110\uc5d0\uc11c\ub9cc \uc0ac\uc6a9\ud560 \uc218 \uc788\uc5c8\uc2b5\ub2c8\ub2e4. \ud558\uc9c0\ub9cc \uc88b\uc740 \uc18c\uc2dd\uc774","protected":false},"author":{"name":"Jessie Cho","link":"https:\/\/blog.jetbrains.com\/ko\/author\/jessie-chojetbrains-com"},"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[],"tags":[588,5488,591,592,5337,5491],"cross-post-tag":[],"acf":[],"featured_image":null,"_links":{"self":[{"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/posts\/34021"}],"collection":[{"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/users\/964"},{"embeddable":true,"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/users\/964"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/comments?post=34021"}],"version-history":[{"count":0,"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/posts\/34021\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/media?parent=34021"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/categories?post=34021"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/tags?post=34021"},{"taxonomy":"cross-post-tag","embeddable":true,"href":"https:\/\/blog.jetbrains.com\/ko\/wp-json\/wp\/v2\/cross-post-tag?post=34021"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}