Debugging containerized Go applications

Containers are increasingly popular for deploying applications and Go applications make no exceptions from this. But how should we debug these applications that are running in the isolated environment a container offers?

For this, we turn back to the way we build these containers and include a copy of Delve, the Go debugger, and we adjust how we create/launch our application.

Let’s consider a basic web application:

We’ll place the code above in a package under GOPATH, for example $GOPATH/src/hello/hello.go

One important note is that your project should be in a valid Go Workspace, also known as GOPATH. You can read more about Go Workspaces in the official documentation.

Before we continue, make sure you have installed the latest Docker plugin from our repository via: Settings | Plugins | Install JetBrains Plugin… | Docker integration.

Now, let’s start using Docker to compile and run this application.

We have a simple, multi-stage, Docker container which contains the building stage and the final container that results from this.

Place this in the root of your package, in our example under $GOPATH/src/hello/Dockerfile.

To run this, we’ll have to create a new Run Configuration, and expose our application’s port, 8080.

Application Run Configuration

To test this, we’ll run the container and then launch a simple request from the IDE. To run requests from the IDE, you can create a file with the “.http” extension and then type the request as below:

Debug Container - Run Application

Let’s transform this container into one that can be debugged. Our goal is to keep as much parity as possible with the original container so that the functionality is not affected by our debugging version.

We must first compile our application with all optimizations turned off, to allow Delve to extract as much information as possible from the binary. The second part involves actually running delve into the container.

Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

Run Application with Debugging Settings

GoLand will automatically map the source code location for us, so even our source code is on an operating system, and we are compiling/using the binary on a different operating system, everything will work without having to do anything special for this case.

After running this container configuration, now the last step is to connect to the debugging container and see the debugger working. Run our container and then go to “Run | Edit Configurations…” and add a new Go Remote configuration, fill in the remote host IP and port number and then run it like any other configuration.

Remote Debugging Configuration

Rerunning a request will stop the debugger as expected and you can see all the values as if you would debug a local running process.

Debug Container - Run Application with Debugger

And that’s it. Using GoLand’s debugging facilities, you can debug not only applications that run on your machine but also applications that run on remote computers or even in containers. And if you are using Kubernetes, you can also install a plugin to help you edit Kubernetes resources.

If you have not done so already, download GoLand today and send us feedback in the comments below or using our issue tracker on how we can further improve your development experience!

About Florin Pățan

Developer Advocate at JetBrains
This entry was posted in Features, Tutorial and tagged , , . Bookmark the permalink.

8 Responses to Debugging containerized Go applications

  1. Viktor says:

    For the second version of the Dockerfile I get this error:

    Could not create config directory: user: Current not implemented on linux/amd64.could not launch process: fork/exec /server: operation not permitted

    • Florin Pățan says:

      Hi Viktor,

      Please make sure you follow the steps described after the Dockerfile here:

      > Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

      The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

  2. dongqi says:

    the log is:
    Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000
    2018/07/10 08:22:54 starting server…

    the container is running. But when I debug the “Remote Debug”, there is something wrong: “could not find /home/…./hello.go:11” and the debug panel showed nothing.

    • Florin Pățan says:

      Hi Dongqi,

      Please make sure you follow the steps described after the Dockerfile here:

      > Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

      The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

      Thank you.

  3. Hey nice blog,

    So this works when you are running your application, so am stuck at debugging tests.

    So I can use the usual feature of just running test from the idea and everything works including the breakpoints. (it has been really helpful by the way)

    Now the problem is at running tests that are not supported by my os for instance of of the feature am testing is “plugin” but thats not supported in osx at the moment.
    How do I run this test with delve and use breakpoints in my idea

    • Florin Pățan says:

      Hi, thank you!

      Unfortunately, debugging when using the plugins feature of Go is currently not supported by Go, and it looks like 1.11 won’t change anything for that (at least as far as I got with reading the changelog).

      > How do I run this test with delve and use breakpoints in my idea

      If you are using plugins, you cannot, it’s a limitation of Go, as mentioned above. If not, tests should work with the debugging feature out of the box. If they don’t please, let me know what’s going on and how to reproduce this and I’ll be happy to help you out.

  4. Marek says:

    Hi,
    Finally I got this working. It’s a cool feature.

    I had some issues though. When I copied the options from the blog I didn’t realized at the first time that double dash and double quotes characters are replaced by another characters. So afterwards I typed options myself in docker configuration correctly and I instantly got the warning: “Can’t parse command line options: Unrecognized option: –security-opt=apparmor=unconfined”. When I clicked “Apply” the warning pop-up appeared. I closed it and clicked “OK” in “Run/Debug Configurations”, and I got the same warning window again. So there was no other way to close it but “Cancel”. When I opened it again I noticed that the configuration was saved.

    When I ran Docker file I got the following message in deploy log:
    “Failed to deploy ‘hello-server Dockerfile: Dockerfile’: org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: –security-opt=apparmor=unconfined”

    I removed this option from the configuration and IDE was able to build an image and run it. Actually I was also able to debug the application without security-opt :-)

    Thx

    • Florin Pățan says:

      Hi Marek,

      Please make sure you are using the latest version of the IDE as well as the latest version of the Docker plugin. If you are, can you please share the Docker plugin version with me? As far as I’m aware, this should not work without the “security-opt” option, so I’m not sure why it’s not accepted by the Docker plugin or why it worked without it.

      Thank you.

Leave a Reply

Your email address will not be published. Required fields are marked *