Debugging containerized Go applications

Containers are increasingly popular for deploying applications and Go applications make no exceptions from this. But how should we debug these applications that are running in the isolated environment a container offers?

For this, we turn back to the way we build these containers and include a copy of Delve, the Go debugger, and we adjust how we create/launch our application.

Let’s consider a basic web application:

We’ll place the code above in a package under GOPATH, for example $GOPATH/src/hello/hello.go

One important note is that your project should be in a valid Go Workspace, also known as GOPATH. You can read more about Go Workspaces in the official documentation.

Before we continue, make sure you have installed the latest Docker plugin from our repository via: Settings | Plugins | Install JetBrains Plugin… | Docker integration.

Now, let’s start using Docker to compile and run this application.

We have a simple, multi-stage, Docker container which contains the building stage and the final container that results from this.

Place this in the root of your package, in our example under $GOPATH/src/hello/Dockerfile.

To run this, we’ll have to create a new Run Configuration, and expose our application’s port, 8080.

Application Run Configuration

To test this, we’ll run the container and then launch a simple request from the IDE. To run requests from the IDE, you can create a file with the “.http” extension and then type the request as below:

Debug Container - Run Application

Let’s transform this container into one that can be debugged. Our goal is to keep as much parity as possible with the original container so that the functionality is not affected by our debugging version.

We must first compile our application with all optimizations turned off, to allow Delve to extract as much information as possible from the binary. The second part involves actually running delve into the container.

Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

Run Application with Debugging Settings

GoLand will automatically map the source code location for us, so even our source code is on an operating system, and we are compiling/using the binary on a different operating system, everything will work without having to do anything special for this case.

After running this container configuration, now the last step is to connect to the debugging container and see the debugger working. Run our container and then go to “Run | Edit Configurations…” and add a new Go Remote configuration, fill in the remote host IP and port number and then run it like any other configuration.

Remote Debugging Configuration

Rerunning a request will stop the debugger as expected and you can see all the values as if you would debug a local running process.

Debug Container - Run Application with Debugger

And that’s it. Using GoLand’s debugging facilities, you can debug not only applications that run on your machine but also applications that run on remote computers or even in containers. And if you are using Kubernetes, you can also install a plugin to help you edit Kubernetes resources.

If you have not done so already, download GoLand today and send us feedback in the comments below or using our issue tracker on how we can further improve your development experience!

About Florin Pățan

Developer Advocate at JetBrains
This entry was posted in Features, Tutorial and tagged , , . Bookmark the permalink.

17 Responses to Debugging containerized Go applications

  1. Viktor says:

    For the second version of the Dockerfile I get this error:

    Could not create config directory: user: Current not implemented on linux/amd64.could not launch process: fork/exec /server: operation not permitted

    • Florin Pățan says:

      Hi Viktor,

      Please make sure you follow the steps described after the Dockerfile here:

      > Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

      The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

      • Brian Topping says:

        Hi Florin, great post! Constructing this from scratch would have been a high barrier to entry with these technologies. Really grateful for your sharing.

        Point of note: Your blog software has mangled a lot of text here. -- turned into , " turned into . I only figured it out after looking at your graphic.

        • Florin Pățan says:

          Hi Brian,

          Thank you for your feedback.
          My apologies for the issue created by the blog platform, my colleagues will look into this as soon as possible, thank you for letting us know about the issues.

          Kind regards,
          Florin

  2. dongqi says:

    the log is:
    Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000
    2018/07/10 08:22:54 starting server…

    the container is running. But when I debug the “Remote Debug”, there is something wrong: “could not find /home/…./hello.go:11” and the debug panel showed nothing.

    • Florin Pățan says:

      Hi Dongqi,

      Please make sure you follow the steps described after the Dockerfile here:

      > Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

      The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

      Thank you.

      • jota says:

        still an issue with me also

        • Florin Pățan says:

          Hi Jota,

          Can you please provide more details as to what’s not working?

          Also, please note that due to some technical issues with our blogging platform, -- turned into and " is turned into . My apologies for this.

          Kind regards,
          Florin

  3. Hey nice blog,

    So this works when you are running your application, so am stuck at debugging tests.

    So I can use the usual feature of just running test from the idea and everything works including the breakpoints. (it has been really helpful by the way)

    Now the problem is at running tests that are not supported by my os for instance of of the feature am testing is “plugin” but thats not supported in osx at the moment.
    How do I run this test with delve and use breakpoints in my idea

    • Florin Pățan says:

      Hi, thank you!

      Unfortunately, debugging when using the plugins feature of Go is currently not supported by Go, and it looks like 1.11 won’t change anything for that (at least as far as I got with reading the changelog).

      > How do I run this test with delve and use breakpoints in my idea

      If you are using plugins, you cannot, it’s a limitation of Go, as mentioned above. If not, tests should work with the debugging feature out of the box. If they don’t please, let me know what’s going on and how to reproduce this and I’ll be happy to help you out.

  4. Marek says:

    Hi,
    Finally I got this working. It’s a cool feature.

    I had some issues though. When I copied the options from the blog I didn’t realized at the first time that double dash and double quotes characters are replaced by another characters. So afterwards I typed options myself in docker configuration correctly and I instantly got the warning: “Can’t parse command line options: Unrecognized option: –security-opt=apparmor=unconfined”. When I clicked “Apply” the warning pop-up appeared. I closed it and clicked “OK” in “Run/Debug Configurations”, and I got the same warning window again. So there was no other way to close it but “Cancel”. When I opened it again I noticed that the configuration was saved.

    When I ran Docker file I got the following message in deploy log:
    “Failed to deploy ‘hello-server Dockerfile: Dockerfile’: org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: –security-opt=apparmor=unconfined”

    I removed this option from the configuration and IDE was able to build an image and run it. Actually I was also able to debug the application without security-opt :-)

    Thx

    • Florin Pățan says:

      Hi Marek,

      Please make sure you are using the latest version of the IDE as well as the latest version of the Docker plugin. If you are, can you please share the Docker plugin version with me? As far as I’m aware, this should not work without the “security-opt” option, so I’m not sure why it’s not accepted by the Docker plugin or why it worked without it.

      Thank you.

      • Marek says:

        Hi Florin,

        I used GoLand 2017.3.4 (build 173.4674.50) and Docker plugin 173.3727.15. I updated GoLand to version 2018.1.5 and there is no warning message about –security-opt :-)
        I supposed that the warning might have been connected to the error from docker build command with –security-opt: “The daemon on this platform does not support setting security options on build” (I’m working on Ubuntu 16.04 with Docker version 18.03.1-ce, build 9ee9f40).
        I did another attempt with the newest GoLand and no security-opt. It worked differently. Without saying much detail I was able to debug only once, so with –security-opt it works much better.

        Thanks

      • Marek says:

        Hi Florin,

        I spent a lot of time on this topic today. The reason is that actually it didn’t work for me with the newest GoLand 2018.1.5 and the newest docker plugin 181.5087.20. I tried various combinations with and without –security-opt and it behaved differently each attempt. Even the same configuration didn’t work for me the same each time. In some cases I couldn’t stop docker container and reset seemed the only option.

        I decided to try it out again on another computer that never had go, docker, and GoLand. System is different (Linux Mint 19) and docker version is a bit higher (Docker version 18.06.0-ce, build 0ffa825). I had the same problems and unpredictable results. The latest status from this computer is that it doesn’t matter if the security-opt is set or not. Debugging worked, although in some previous attempt I experienced the same issues including impossibility to stop container.

        Unfortunately I was not able to achieve the same on the first computer. Configuration that works for me on the second one doesn’t work on the first one. I think it’s worth mentioning that before each attempt I deleted all the docker containers and images in order to have the same starting point.

        There is a difference in behaviour on both computers, In the second one I have to send a request to the application to start debugging. On the first one it starts debugging instantly when I run “Remote Debug”. And I don’t know why. So I can debug once but the second request is processed without stopping at the breakpoint.

        If you have any suggestions what could be wrong on the first computer I would appreciate any tip.

        Thanks

  5. Mark Robinson says:

    Hey Florin,

    Great work, but there are some problems on Windows (of course)

    In the docker launch config:
    You need to add “–security-opt seccomp:unconfined” to the command line options
    “USER=root” to the environment variable configuration

    These are required otherwise Delve will fail to launch

    • Florin Pățan says:

      Hi Mark, thanks for the reply.

      Normally, adding --security-opt=”apparmor=unconfined” --cap-add=SYS_PTRACE should be enough to get it going. I’m using Windows as host OS as well, and I never hard to add those options to containers. Is this some pre-release Docker version? As for user root, that’s the default user in most base container images.

Leave a Reply

Your email address will not be published. Required fields are marked *