Debugging containerized Go applications

Containers are increasingly popular for deploying applications and Go applications make no exceptions from this. But how should we debug these applications that are running in the isolated environment a container offers?

For this, we turn back to the way we build these containers and include a copy of Delve, the Go debugger, and we adjust how we create/launch our application.

Let’s consider a basic web application:

We’ll place the code above in a package under GOPATH, for example $GOPATH/src/hello/hello.go

One important note is that your project should be in a valid Go Workspace, also known as GOPATH. You can read more about Go Workspaces in the official documentation.

Before we continue, make sure you have installed the latest Docker plugin from our repository via: Settings | Plugins | Install JetBrains Plugin… | Docker integration.

Now, let’s start using Docker to compile and run this application.

We have a simple, multi-stage, Docker container which contains the building stage and the final container that results from this.

Place this in the root of your package, in our example under $GOPATH/src/hello/Dockerfile.

To run this, we’ll have to create a new Run Configuration, and expose our application’s port, 8080.

Application Run Configuration

To test this, we’ll run the container and then launch a simple request from the IDE. To run requests from the IDE, you can create a file with the “.http” extension and then type the request as below:

Debug Container - Run Application

Let’s transform this container into one that can be debugged. Our goal is to keep as much parity as possible with the original container so that the functionality is not affected by our debugging version.

We must first compile our application with all optimizations turned off, to allow Delve to extract as much information as possible from the binary. The second part involves actually running delve into the container.

Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

Run Application with Debugging Settings

GoLand will automatically map the source code location for us, so even our source code is on an operating system, and we are compiling/using the binary on a different operating system, everything will work without having to do anything special for this case.

After running this container configuration, now the last step is to connect to the debugging container and see the debugger working. Run our container and then go to “Run | Edit Configurations…” and add a new Go Remote configuration, fill in the remote host IP and port number and then run it like any other configuration.

Remote Debugging Configuration

Rerunning a request will stop the debugger as expected and you can see all the values as if you would debug a local running process.

Debug Container - Run Application with Debugger

And that’s it. Using GoLand’s debugging facilities, you can debug not only applications that run on your machine but also applications that run on remote computers or even in containers. And if you are using Kubernetes, you can also install a plugin to help you edit Kubernetes resources.

If you have not done so already, download GoLand today and send us feedback in the comments below or using our issue tracker on how we can further improve your development experience!

About Florin Pățan

Developer Advocate at JetBrains
This entry was posted in Features, Tutorial and tagged , , . Bookmark the permalink.

38 Responses to Debugging containerized Go applications

  1. Viktor says:

    For the second version of the Dockerfile I get this error:

    Could not create config directory: user: Current not implemented on linux/amd64.could not launch process: fork/exec /server: operation not permitted

    • Florin Pățan says:

      Hi Viktor,

      Please make sure you follow the steps described after the Dockerfile here:

      > Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

      The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

      • Brian Topping says:

        Hi Florin, great post! Constructing this from scratch would have been a high barrier to entry with these technologies. Really grateful for your sharing.

        Point of note: Your blog software has mangled a lot of text here. -- turned into , " turned into . I only figured it out after looking at your graphic.

        • Florin Pățan says:

          Hi Brian,

          Thank you for your feedback.
          My apologies for the issue created by the blog platform, my colleagues will look into this as soon as possible, thank you for letting us know about the issues.

          Kind regards,
          Florin

  2. dongqi says:

    the log is:
    Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000
    2018/07/10 08:22:54 starting server…

    the container is running. But when I debug the “Remote Debug”, there is something wrong: “could not find /home/…./hello.go:11” and the debug panel showed nothing.

    • Florin Pățan says:

      Hi Dongqi,

      Please make sure you follow the steps described after the Dockerfile here:

      > Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

      The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

      Thank you.

      • jota says:

        still an issue with me also

        • Florin Pățan says:

          Hi Jota,

          Can you please provide more details as to what’s not working?

          Also, please note that due to some technical issues with our blogging platform, -- turned into and " is turned into . My apologies for this.

          Kind regards,
          Florin

  3. Hey nice blog,

    So this works when you are running your application, so am stuck at debugging tests.

    So I can use the usual feature of just running test from the idea and everything works including the breakpoints. (it has been really helpful by the way)

    Now the problem is at running tests that are not supported by my os for instance of of the feature am testing is “plugin” but thats not supported in osx at the moment.
    How do I run this test with delve and use breakpoints in my idea

    • Florin Pățan says:

      Hi, thank you!

      Unfortunately, debugging when using the plugins feature of Go is currently not supported by Go, and it looks like 1.11 won’t change anything for that (at least as far as I got with reading the changelog).

      > How do I run this test with delve and use breakpoints in my idea

      If you are using plugins, you cannot, it’s a limitation of Go, as mentioned above. If not, tests should work with the debugging feature out of the box. If they don’t please, let me know what’s going on and how to reproduce this and I’ll be happy to help you out.

  4. Marek says:

    Hi,
    Finally I got this working. It’s a cool feature.

    I had some issues though. When I copied the options from the blog I didn’t realized at the first time that double dash and double quotes characters are replaced by another characters. So afterwards I typed options myself in docker configuration correctly and I instantly got the warning: “Can’t parse command line options: Unrecognized option: –security-opt=apparmor=unconfined”. When I clicked “Apply” the warning pop-up appeared. I closed it and clicked “OK” in “Run/Debug Configurations”, and I got the same warning window again. So there was no other way to close it but “Cancel”. When I opened it again I noticed that the configuration was saved.

    When I ran Docker file I got the following message in deploy log:
    “Failed to deploy ‘hello-server Dockerfile: Dockerfile’: org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: –security-opt=apparmor=unconfined”

    I removed this option from the configuration and IDE was able to build an image and run it. Actually I was also able to debug the application without security-opt :-)

    Thx

    • Florin Pățan says:

      Hi Marek,

      Please make sure you are using the latest version of the IDE as well as the latest version of the Docker plugin. If you are, can you please share the Docker plugin version with me? As far as I’m aware, this should not work without the “security-opt” option, so I’m not sure why it’s not accepted by the Docker plugin or why it worked without it.

      Thank you.

      • Marek says:

        Hi Florin,

        I used GoLand 2017.3.4 (build 173.4674.50) and Docker plugin 173.3727.15. I updated GoLand to version 2018.1.5 and there is no warning message about –security-opt :-)
        I supposed that the warning might have been connected to the error from docker build command with –security-opt: “The daemon on this platform does not support setting security options on build” (I’m working on Ubuntu 16.04 with Docker version 18.03.1-ce, build 9ee9f40).
        I did another attempt with the newest GoLand and no security-opt. It worked differently. Without saying much detail I was able to debug only once, so with –security-opt it works much better.

        Thanks

      • Marek says:

        Hi Florin,

        I spent a lot of time on this topic today. The reason is that actually it didn’t work for me with the newest GoLand 2018.1.5 and the newest docker plugin 181.5087.20. I tried various combinations with and without –security-opt and it behaved differently each attempt. Even the same configuration didn’t work for me the same each time. In some cases I couldn’t stop docker container and reset seemed the only option.

        I decided to try it out again on another computer that never had go, docker, and GoLand. System is different (Linux Mint 19) and docker version is a bit higher (Docker version 18.06.0-ce, build 0ffa825). I had the same problems and unpredictable results. The latest status from this computer is that it doesn’t matter if the security-opt is set or not. Debugging worked, although in some previous attempt I experienced the same issues including impossibility to stop container.

        Unfortunately I was not able to achieve the same on the first computer. Configuration that works for me on the second one doesn’t work on the first one. I think it’s worth mentioning that before each attempt I deleted all the docker containers and images in order to have the same starting point.

        There is a difference in behaviour on both computers, In the second one I have to send a request to the application to start debugging. On the first one it starts debugging instantly when I run “Remote Debug”. And I don’t know why. So I can debug once but the second request is processed without stopping at the breakpoint.

        If you have any suggestions what could be wrong on the first computer I would appreciate any tip.

        Thanks

  5. Mark Robinson says:

    Hey Florin,

    Great work, but there are some problems on Windows (of course)

    In the docker launch config:
    You need to add “–security-opt seccomp:unconfined” to the command line options
    “USER=root” to the environment variable configuration

    These are required otherwise Delve will fail to launch

    • Florin Pățan says:

      Hi Mark, thanks for the reply.

      Normally, adding --security-opt=”apparmor=unconfined” --cap-add=SYS_PTRACE should be enough to get it going. I’m using Windows as host OS as well, and I never hard to add those options to containers. Is this some pre-release Docker version? As for user root, that’s the default user in most base container images.

  6. Robert K. Bell says:

    Is it possible to set --security-opt="apparmor=unconfined" --cap-add=SYS_PTRACE when using Docker Swarm? We’ve got a set of interdependent services that make heavy use of the Swarm-only Secrets feature.

    • Florin Pățan says:

      It doesn’t look like it’s possible at the moment, if I’m reading this issue https://github.com/moby/moby/issues/25209 correctly. I’ll try and see if there’s any other way to solve this. Unfortunately without those options, due to the restrictions in Docker/the way containers are set up, debugging them won’t work.

  7. iamwwc says:

    I find when I stop debug in goland, delve in container auto stop, I try use –accept-muticlient, but not working

    here is cmd in container

    dlv debug –output /tmp/debug –wd /go/src/chaochaogege.com/onlinecode –headless
    –listen=:8088 –api-version=2 –accept-multiclient
    How I let delve accept multi connection even though goland debug stop?

    • Florin Pățan says:

      If I read delve’s sources correctly, then when the last client of a debug session disconnects in a multiclient mode, the debugger will also terminate.

      My question is what are you trying to do? Do you want to attach the debugger, debug something, then leave the app running?

      I would not recommend doing this in any environment as the way binaries are compiled for debugging greatly reduces their performance so your application will be a lot slower.

      • iamwwc says:

        I am use docker to deploy my app.
        But my develop host is Windows, I need some operation but Linux have different file system structure to Windows. So my codes cannot directly run on Windows.
        leave alone my first commit, I have solve it by Goland’s before build(Restart delve Container again)
        But I cannot get app output. delve put output to stdout, I cannot get logs which my app produce.
        Can we get app’s logs from container and put it into Goland Console?
        I try mount log file to my workspace, and then use goland configuration Logs Log files to be shown in console, But console not show anything.
        Sorry for my bad English

        • Florin Pățan says:

          Delve should proxy the output just fine. Do you have a small sample I could use to replicate the issue? Are you using 2018.2.3?

          Your English is ok but if you prefer another language, please write in it and I will ask my colleagues to helps us.

  8. Chris says:

    When trying to debug I get

    Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000

    [This issue](https://github.com/golang/go/issues/14625) implies that cgo is required

    But cgo is disabled in the Dockerfile

    ENV CGO_ENABLED 0

    If I enable cgo I get

    # runtime/cgo
    exec: "gcc": executable file not found in $PATH

    Tested using
    – macOS High Sierra 10.13.6
    – Goland 2018.2.2
    – FROM golang:1.11.1-alpine3.7

    • Chris says:

      Same behavior with Goland 2018.2.3

    • Florin Pățan says:

      In the above example, we are using a multi-stage container, which can be read as two Docker containers if you want.

      The first part takes care of building the binary and delve, and it does have CGO disabled:


      # Compile stage
      FROM golang:1.10.1-alpine3.7 AS build-env
      ENV CGO_ENABLED 0
      ADD . /go/src/hello

      # The -gcflags "all=-N -l" flag helps us get a better debug experience
      RUN go build -gcflags "all=-N -l" -o /server hello

      # Compile Delve
      RUN apk add --no-cache git
      RUN go get github.com/derekparker/delve/cmd/dlv

      The second part has no need for Go or anything else as our binaries are already built.

      # Final stage
      FROM alpine:3.7

      # Port 8080 belongs to our application, 40000 belongs to Delve
      EXPOSE 8080 40000

      # Allow delve to run on Alpine based containers.
      RUN apk add --no-cache libc6-compat

      WORKDIR /

      COPY --from=build-env /server /
      COPY --from=build-env /go/bin/dlv /

      # Run delve
      CMD ["/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "exec", "/server"]

      Can you share your Dockerfile to understand what’s happening? Thank you.

      • Chris says:

        Using the Dockerfile as per the post I get this error user: Current not implemented on linux/amd64

        > it does have CGO disabled

        Reading up on other issues it sounds like above error happens **because** CGO is disabled.

        I suspect to resolve my issue I need to build with CGO enabled. Changing ENV CGO_ENABLED 0 in the Dockerfile as per the post does not work

        • Florin Pățan says:

          I think I understand now. Yes, if you need to depend on CGO, then you need to remove that line, ENV CGO_ENABLED 0 . And next, if you need gcc , then you can change this line RUN apk add --no-cache git to RUN apk add --no-cache git build-base and move it before the go build line.

  9. VikasG says:

    Will this work in IntelliJ IDEA Ultimate? Our company has a license for IDEA but not GoLand. Thank you.

    • Florin Pățan says:

      Yes, this will work on either GoLand or IntelliJ IDEA Ultimate as IntelliJ IDEA Ultimate ships with the same Docker plugin as GoLand and the Go plugin supports the same features as GoLand has.

  10. VikasG says:

    If I use Dockerfile.1, I see this error on running the container:

    Could not create config directory: user: Current not implemented on linux/amd64.

    If I use Dockerfile.2, I don’t see any error on running the container.

    However, debugger doesn’t stop at any breakpoints in the source.

    —- Dockerfile.1 —-

    # Compile stage
    FROM golang:1.10.1-alpine3.7 AS build-env
    ENV CGO_ENABLED 0
    ADD . /go/src/hello

    # The -gcflags “all=-N -l” flag helps us get a better debug experience
    RUN go build -gcflags “all=-N -l” -o /server hello

    # Compile Delve
    RUN apk add –no-cache git
    RUN go get github.com/derekparker/delve/cmd/dlv

    # Final stage
    FROM alpine:3.7

    # Port 8080 belongs to our application, 40000 belongs to Delve
    EXPOSE 8080 40000

    # Allow delve to run on Alpine based containers.
    RUN apk add –no-cache libc6-compat

    WORKDIR /

    COPY –from=build-env /server /
    COPY –from=build-env /go/bin/dlv /

    # Run delve
    CMD [“/dlv”, “–listen=:40000”, “–headless=true”, “–api-version=2”, “exec”, “/server”]

    —-Dockerfile.2—-

    # Compile stage
    FROM golang:1.10.1-alpine3.7 AS build-env
    ADD . /go/src/hello

    RUN apk add –no-cache libc6-compat build-base

    # The -gcflags “all=-N -l” flag helps us get a better debug experience
    RUN go build -gcflags “all=-N -l” -o /server hello

    # Compile Delve
    RUN apk add –no-cache git
    RUN go get github.com/derekparker/delve/cmd/dlv

    # Final stage
    FROM alpine:3.7

    # Port 8080 belongs to our application, 40000 belongs to Delve
    EXPOSE 8080 40000

    # Allow delve to run on Alpine based containers.
    RUN apk add –no-cache libc6-compat

    WORKDIR /

    COPY –from=build-env /server /
    COPY –from=build-env /go/bin/dlv /

    # Run delve
    CMD [“/dlv”, “–listen=:40000”, “–headless=true”, “–api-version=2”, “exec”, “/server”]

    • Florin Pățan says:

      According to the delve authors, you can ignore the message Could not create config directory: user: Current not implemented on linux/amd64. .

      Do you also have a message right after it saying API server listening at: [::]:40000 (replace 40000 with your port of choice for the debugger)?

      I tried the first Dockerfile and I could replicate the message but when I created and launched the Go Remote debug configuration, everything worked as expected. If this still is an issue for you, can you please let us know? Thank you.

      • VikasG says:

        It is still an issue.

        Docker container logs:

        Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000
        2018/11/08 17:56:44 starting server…

        • VikasG says:

          It is still an issue.

          Docker container logs:

          Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000
          2018/11/08 17:56:44 starting server…

          2) May I know what GoLand version, Go version, docker version, OS you are using? Have you tested this on Mac OS + Docker for Mac?

  11. VikasG says:

    I posted a long post at https://stackoverflow.com/questions/53201915 . Please take a look and try to reply. I added a lot of detail there. Thank you very much.

Leave a Reply

Your email address will not be published. Required fields are marked *