Debugging containerized Go applications

Containers are increasingly popular for deploying applications and Go applications make no exceptions from this. But how should we debug these applications that are running in the isolated environment a container offers?

For this, we turn back to the way we build these containers and include a copy of Delve, the Go debugger, and we adjust how we create/launch our application.

Let’s consider a basic web application:

We’ll place the code above in a package under GOPATH, for example $GOPATH/src/hello/hello.go

One important note is that your project should be in a valid Go Workspace, also known as GOPATH. You can read more about Go Workspaces in the official documentation.

Before we continue, make sure you have installed the latest Docker plugin from our repository via: Settings | Plugins | Install JetBrains Plugin… | Docker integration.

Now, let’s start using Docker to compile and run this application.

We have a simple, multi-stage, Docker container which contains the building stage and the final container that results from this.

Place this in the root of your package, in our example under $GOPATH/src/hello/Dockerfile.

To run this, we’ll have to create a new Run Configuration, and expose our application’s port, 8080.

Application Run Configuration

To test this, we’ll run the container and then launch a simple request from the IDE. To run requests from the IDE, you can create a file with the “.http” extension and then type the request as below:

Debug Container - Run Application

Let’s transform this container into one that can be debugged. Our goal is to keep as much parity as possible with the original container so that the functionality is not affected by our debugging version.

We must first compile our application with all optimizations turned off, to allow Delve to extract as much information as possible from the binary. The second part involves actually running delve into the container.

Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, --security-opt="apparmor=unconfined" --cap-add=SYS_PTRACE.

Run Application with Debugging Settings

GoLand will automatically map the source code location for us, so even our source code is on an operating system, and we are compiling/using the binary on a different operating system, everything will work without having to do anything special for this case.

After running this container configuration, now the last step is to connect to the debugging container and see the debugger working. Run our container and then go to “Run | Edit Configurations…” and add a new Go Remote configuration, fill in the remote host IP and port number and then run it like any other configuration.

Remote Debugging Configuration

Rerunning a request will stop the debugger as expected and you can see all the values as if you would debug a local running process.

Debug Container - Run Application with Debugger

And that’s it. Using GoLand’s debugging facilities, you can debug not only applications that run on your machine but also applications that run on remote computers or even in containers. And if you are using Kubernetes, you can also install a plugin to help you edit Kubernetes resources.

If you have not done so already, download GoLand today and send us feedback in the comments below or using our issue tracker on how we can further improve your development experience!

About Florin Pățan

Developer Advocate at JetBrains
This entry was posted in Features, Tutorial and tagged , , . Bookmark the permalink.

55 Responses to Debugging containerized Go applications

  1. Viktor says:

    For the second version of the Dockerfile I get this error:

    Could not create config directory: user: Current not implemented on linux/amd64.could not launch process: fork/exec /server: operation not permitted

    • Florin Pățan says:

      Hi Viktor,

      Please make sure you follow the steps described after the Dockerfile here:

      > Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

      The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

      • Brian Topping says:

        Hi Florin, great post! Constructing this from scratch would have been a high barrier to entry with these technologies. Really grateful for your sharing.

        Point of note: Your blog software has mangled a lot of text here. -- turned into , " turned into . I only figured it out after looking at your graphic.

        • Florin Pățan says:

          Hi Brian,

          Thank you for your feedback.
          My apologies for the issue created by the blog platform, my colleagues will look into this as soon as possible, thank you for letting us know about the issues.

          Kind regards,
          Florin

  2. dongqi says:

    the log is:
    Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000
    2018/07/10 08:22:54 starting server…

    the container is running. But when I debug the “Remote Debug”, there is something wrong: “could not find /home/…./hello.go:11” and the debug panel showed nothing.

    • Florin Pățan says:

      Hi Dongqi,

      Please make sure you follow the steps described after the Dockerfile here:

      > Before we can continue, we also need to add two more options to the Docker Run Configuration from earlier. First one, we need to expose the port we’ll send and receive data from Delve, port 40000. The second one will be to allow delve to debug the application inside the container, –security-opt=”apparmor=unconfined” –cap-add=SYS_PTRACE.

      The error comes most likely from the missing -security-opt / -cap-add arguments to the Docker daemon. If this is still an issue, please let us know.

      Thank you.

      • jota says:

        still an issue with me also

        • Florin Pățan says:

          Hi Jota,

          Can you please provide more details as to what’s not working?

          Also, please note that due to some technical issues with our blogging platform, -- turned into and " is turned into . My apologies for this.

          Kind regards,
          Florin

      • Nicholas says:

        This is happening for me as well. The debugger is able to connect, but the breakpoints do not work. Hovering over them shows the “could not find file” error mentioned above.

        If I move my project directory to match the directory it is mounted in on the docker, then everything works perfectly.

        ex: Project was located at /var/www/project, and mounted at /app on docker. Debugging didn’t work. Moved the project to /app, then debugging worked.

        It doesn’t seem to automatically map the remote directory to the local.

        • Florin Pățan says:

          GoLand maps the sources automatically, as long as you use GOPATH correctly. If you are using Go Modules and still have an issue, please let us know.

    • John Kleijn says:

      For anyone else having this issue: make sure that the paths __relative__ to $GOPATH are the same on the host and when building the image.

      Eg if your project on in Goland is in $GOPATH/src/host/namespace/project, you would build (GOPATH is /go by default in the Go image):

      ADD . /src/host/namespace/project
      RUN go build -gcflags “all=-N -l” -o /server /src/host/namespace/project

  3. Hey nice blog,

    So this works when you are running your application, so am stuck at debugging tests.

    So I can use the usual feature of just running test from the idea and everything works including the breakpoints. (it has been really helpful by the way)

    Now the problem is at running tests that are not supported by my os for instance of of the feature am testing is “plugin” but thats not supported in osx at the moment.
    How do I run this test with delve and use breakpoints in my idea

    • Florin Pățan says:

      Hi, thank you!

      Unfortunately, debugging when using the plugins feature of Go is currently not supported by Go, and it looks like 1.11 won’t change anything for that (at least as far as I got with reading the changelog).

      > How do I run this test with delve and use breakpoints in my idea

      If you are using plugins, you cannot, it’s a limitation of Go, as mentioned above. If not, tests should work with the debugging feature out of the box. If they don’t please, let me know what’s going on and how to reproduce this and I’ll be happy to help you out.

  4. Marek says:

    Hi,
    Finally I got this working. It’s a cool feature.

    I had some issues though. When I copied the options from the blog I didn’t realized at the first time that double dash and double quotes characters are replaced by another characters. So afterwards I typed options myself in docker configuration correctly and I instantly got the warning: “Can’t parse command line options: Unrecognized option: –security-opt=apparmor=unconfined”. When I clicked “Apply” the warning pop-up appeared. I closed it and clicked “OK” in “Run/Debug Configurations”, and I got the same warning window again. So there was no other way to close it but “Cancel”. When I opened it again I noticed that the configuration was saved.

    When I ran Docker file I got the following message in deploy log:
    “Failed to deploy ‘hello-server Dockerfile: Dockerfile’: org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: –security-opt=apparmor=unconfined”

    I removed this option from the configuration and IDE was able to build an image and run it. Actually I was also able to debug the application without security-opt :-)

    Thx

    • Florin Pățan says:

      Hi Marek,

      Please make sure you are using the latest version of the IDE as well as the latest version of the Docker plugin. If you are, can you please share the Docker plugin version with me? As far as I’m aware, this should not work without the “security-opt” option, so I’m not sure why it’s not accepted by the Docker plugin or why it worked without it.

      Thank you.

      • Marek says:

        Hi Florin,

        I used GoLand 2017.3.4 (build 173.4674.50) and Docker plugin 173.3727.15. I updated GoLand to version 2018.1.5 and there is no warning message about –security-opt :-)
        I supposed that the warning might have been connected to the error from docker build command with –security-opt: “The daemon on this platform does not support setting security options on build” (I’m working on Ubuntu 16.04 with Docker version 18.03.1-ce, build 9ee9f40).
        I did another attempt with the newest GoLand and no security-opt. It worked differently. Without saying much detail I was able to debug only once, so with –security-opt it works much better.

        Thanks

      • Marek says:

        Hi Florin,

        I spent a lot of time on this topic today. The reason is that actually it didn’t work for me with the newest GoLand 2018.1.5 and the newest docker plugin 181.5087.20. I tried various combinations with and without –security-opt and it behaved differently each attempt. Even the same configuration didn’t work for me the same each time. In some cases I couldn’t stop docker container and reset seemed the only option.

        I decided to try it out again on another computer that never had go, docker, and GoLand. System is different (Linux Mint 19) and docker version is a bit higher (Docker version 18.06.0-ce, build 0ffa825). I had the same problems and unpredictable results. The latest status from this computer is that it doesn’t matter if the security-opt is set or not. Debugging worked, although in some previous attempt I experienced the same issues including impossibility to stop container.

        Unfortunately I was not able to achieve the same on the first computer. Configuration that works for me on the second one doesn’t work on the first one. I think it’s worth mentioning that before each attempt I deleted all the docker containers and images in order to have the same starting point.

        There is a difference in behaviour on both computers, In the second one I have to send a request to the application to start debugging. On the first one it starts debugging instantly when I run “Remote Debug”. And I don’t know why. So I can debug once but the second request is processed without stopping at the breakpoint.

        If you have any suggestions what could be wrong on the first computer I would appreciate any tip.

        Thanks

  5. Mark Robinson says:

    Hey Florin,

    Great work, but there are some problems on Windows (of course)

    In the docker launch config:
    You need to add “–security-opt seccomp:unconfined” to the command line options
    “USER=root” to the environment variable configuration

    These are required otherwise Delve will fail to launch

    • Florin Pățan says:

      Hi Mark, thanks for the reply.

      Normally, adding --security-opt=”apparmor=unconfined” --cap-add=SYS_PTRACE should be enough to get it going. I’m using Windows as host OS as well, and I never hard to add those options to containers. Is this some pre-release Docker version? As for user root, that’s the default user in most base container images.

  6. Robert K. Bell says:

    Is it possible to set --security-opt="apparmor=unconfined" --cap-add=SYS_PTRACE when using Docker Swarm? We’ve got a set of interdependent services that make heavy use of the Swarm-only Secrets feature.

    • Florin Pățan says:

      It doesn’t look like it’s possible at the moment, if I’m reading this issue https://github.com/moby/moby/issues/25209 correctly. I’ll try and see if there’s any other way to solve this. Unfortunately without those options, due to the restrictions in Docker/the way containers are set up, debugging them won’t work.

  7. iamwwc says:

    I find when I stop debug in goland, delve in container auto stop, I try use –accept-muticlient, but not working

    here is cmd in container

    dlv debug –output /tmp/debug –wd /go/src/chaochaogege.com/onlinecode –headless
    –listen=:8088 –api-version=2 –accept-multiclient
    How I let delve accept multi connection even though goland debug stop?

    • Florin Pățan says:

      If I read delve’s sources correctly, then when the last client of a debug session disconnects in a multiclient mode, the debugger will also terminate.

      My question is what are you trying to do? Do you want to attach the debugger, debug something, then leave the app running?

      I would not recommend doing this in any environment as the way binaries are compiled for debugging greatly reduces their performance so your application will be a lot slower.

      • iamwwc says:

        I am use docker to deploy my app.
        But my develop host is Windows, I need some operation but Linux have different file system structure to Windows. So my codes cannot directly run on Windows.
        leave alone my first commit, I have solve it by Goland’s before build(Restart delve Container again)
        But I cannot get app output. delve put output to stdout, I cannot get logs which my app produce.
        Can we get app’s logs from container and put it into Goland Console?
        I try mount log file to my workspace, and then use goland configuration Logs Log files to be shown in console, But console not show anything.
        Sorry for my bad English

        • Florin Pățan says:

          Delve should proxy the output just fine. Do you have a small sample I could use to replicate the issue? Are you using 2018.2.3?

          Your English is ok but if you prefer another language, please write in it and I will ask my colleagues to helps us.

  8. Chris says:

    When trying to debug I get

    Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000

    [This issue](https://github.com/golang/go/issues/14625) implies that cgo is required

    But cgo is disabled in the Dockerfile

    ENV CGO_ENABLED 0

    If I enable cgo I get

    # runtime/cgo
    exec: "gcc": executable file not found in $PATH

    Tested using
    – macOS High Sierra 10.13.6
    – Goland 2018.2.2
    – FROM golang:1.11.1-alpine3.7

    • Chris says:

      Same behavior with Goland 2018.2.3

    • Florin Pățan says:

      In the above example, we are using a multi-stage container, which can be read as two Docker containers if you want.

      The first part takes care of building the binary and delve, and it does have CGO disabled:


      # Compile stage
      FROM golang:1.10.1-alpine3.7 AS build-env
      ENV CGO_ENABLED 0
      ADD . /go/src/hello

      # The -gcflags "all=-N -l" flag helps us get a better debug experience
      RUN go build -gcflags "all=-N -l" -o /server hello

      # Compile Delve
      RUN apk add --no-cache git
      RUN go get github.com/derekparker/delve/cmd/dlv

      The second part has no need for Go or anything else as our binaries are already built.

      # Final stage
      FROM alpine:3.7

      # Port 8080 belongs to our application, 40000 belongs to Delve
      EXPOSE 8080 40000

      # Allow delve to run on Alpine based containers.
      RUN apk add --no-cache libc6-compat

      WORKDIR /

      COPY --from=build-env /server /
      COPY --from=build-env /go/bin/dlv /

      # Run delve
      CMD ["/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "exec", "/server"]

      Can you share your Dockerfile to understand what’s happening? Thank you.

      • Chris says:

        Using the Dockerfile as per the post I get this error user: Current not implemented on linux/amd64

        > it does have CGO disabled

        Reading up on other issues it sounds like above error happens **because** CGO is disabled.

        I suspect to resolve my issue I need to build with CGO enabled. Changing ENV CGO_ENABLED 0 in the Dockerfile as per the post does not work

        • Florin Pățan says:

          I think I understand now. Yes, if you need to depend on CGO, then you need to remove that line, ENV CGO_ENABLED 0 . And next, if you need gcc , then you can change this line RUN apk add --no-cache git to RUN apk add --no-cache git build-base and move it before the go build line.

        • Walter says:

          This works for me thanks Chris

  9. VikasG says:

    Will this work in IntelliJ IDEA Ultimate? Our company has a license for IDEA but not GoLand. Thank you.

    • Florin Pățan says:

      Yes, this will work on either GoLand or IntelliJ IDEA Ultimate as IntelliJ IDEA Ultimate ships with the same Docker plugin as GoLand and the Go plugin supports the same features as GoLand has.

  10. VikasG says:

    If I use Dockerfile.1, I see this error on running the container:

    Could not create config directory: user: Current not implemented on linux/amd64.

    If I use Dockerfile.2, I don’t see any error on running the container.

    However, debugger doesn’t stop at any breakpoints in the source.

    —- Dockerfile.1 —-

    # Compile stage
    FROM golang:1.10.1-alpine3.7 AS build-env
    ENV CGO_ENABLED 0
    ADD . /go/src/hello

    # The -gcflags “all=-N -l” flag helps us get a better debug experience
    RUN go build -gcflags “all=-N -l” -o /server hello

    # Compile Delve
    RUN apk add –no-cache git
    RUN go get github.com/derekparker/delve/cmd/dlv

    # Final stage
    FROM alpine:3.7

    # Port 8080 belongs to our application, 40000 belongs to Delve
    EXPOSE 8080 40000

    # Allow delve to run on Alpine based containers.
    RUN apk add –no-cache libc6-compat

    WORKDIR /

    COPY –from=build-env /server /
    COPY –from=build-env /go/bin/dlv /

    # Run delve
    CMD [“/dlv”, “–listen=:40000”, “–headless=true”, “–api-version=2”, “exec”, “/server”]

    —-Dockerfile.2—-

    # Compile stage
    FROM golang:1.10.1-alpine3.7 AS build-env
    ADD . /go/src/hello

    RUN apk add –no-cache libc6-compat build-base

    # The -gcflags “all=-N -l” flag helps us get a better debug experience
    RUN go build -gcflags “all=-N -l” -o /server hello

    # Compile Delve
    RUN apk add –no-cache git
    RUN go get github.com/derekparker/delve/cmd/dlv

    # Final stage
    FROM alpine:3.7

    # Port 8080 belongs to our application, 40000 belongs to Delve
    EXPOSE 8080 40000

    # Allow delve to run on Alpine based containers.
    RUN apk add –no-cache libc6-compat

    WORKDIR /

    COPY –from=build-env /server /
    COPY –from=build-env /go/bin/dlv /

    # Run delve
    CMD [“/dlv”, “–listen=:40000”, “–headless=true”, “–api-version=2”, “exec”, “/server”]

    • Florin Pățan says:

      According to the delve authors, you can ignore the message Could not create config directory: user: Current not implemented on linux/amd64. .

      Do you also have a message right after it saying API server listening at: [::]:40000 (replace 40000 with your port of choice for the debugger)?

      I tried the first Dockerfile and I could replicate the message but when I created and launched the Go Remote debug configuration, everything worked as expected. If this still is an issue for you, can you please let us know? Thank you.

      • VikasG says:

        It is still an issue.

        Docker container logs:

        Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000
        2018/11/08 17:56:44 starting server…

        • VikasG says:

          It is still an issue.

          Docker container logs:

          Could not create config directory: user: Current not implemented on linux/amd64.API server listening at: [::]:40000
          2018/11/08 17:56:44 starting server…

          2) May I know what GoLand version, Go version, docker version, OS you are using? Have you tested this on Mac OS + Docker for Mac?

          • Florin Pățan says:

            You can ignore the message: Could not create config directory: user: Current not implemented on linux/amd64.API . The debug server started ok, since this is the log after that line server listening at: [::]:40000
            2018/11/08 17:56:44 starting server…

            > 2) May I know what GoLand version, Go version, docker version, OS you are using? Have you tested this on Mac OS + Docker for Mac?

            This should work with any GoLand version from 2018.1+ I recommend using the latest GoLand version.

            For Go version, the same applies, always use the latest version possible, or at least Go 1.11.

            This works for all Docker environments, Windows, Linux or macOS.

  11. VikasG says:

    I posted a long post at https://stackoverflow.com/questions/53201915 . Please take a look and try to reply. I added a lot of detail there. Thank you very much.

  12. Tom Vaughan says:

    Everything works OK when I try the first part. That is, running without delve and only port 8080.
    When I go to localhost:8080 the output is:
    HTTP/1.1 200 OK
    Date: Tue, 11 Dec 2018 17:35:11 GMT
    Content-Length: 11
    Content-Type: text/plain; charset=utf-8

    Hello world

    Response code: 200 (OK); Time: 11ms; Content length: 11 bytes

    When I try to get the rest to work everything seems to build OK but the output in the attached console is just:
    API server listening at: [::]:40000

    When I try to go to localhost:8080 i get the error:
    org.apache.http.NoHttpResponseException: localhost:8080 failed to respond

    Any idea what is wrong?

    I am using:
    GoLand 2018.3.1
    Build #GO-183.4588.42, built on December 3, 2018
    JRE: 1.8.0_152-release-1343-b16 amd64
    JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
    Windows 10 10.0

    • Florin Pățan says:

      It looks like everything works ok. Have you tried to also run the Go Remote configuration? This is needed as Delve will block the execution until it receives the start command from a Delve client (in this case the IDE).

  13. Ruian says:

    Hi Florin Pățan,

    I follow your instructions and almost everything work fine.

    My Goland IDE successfully connects to the delve server in the container, but It send my local source file path to delve when setting breakpoints, and then, of course, the delve server reports that it can’t find the location because the binary is compiled inside container with a different path.

    I use Go module, the project is not under GOPATH, and the version of Goland IDE I used is 2018.3.2

    Currently the workaround I used is to make a directory in the container which has the same path as my project directory in local and compile the binary under that directory.

    My question is that is there any other way to do custom path remapping?

    Another question is that is there any option can let me customize the behavior of IDE’s debug button which shown on the left side of my golang test function, so that I can pass the arguments to my container to run delve with target test function?

    Thanks.

    • Florin Pățan says:

      Hi Ruian,

      > Currently the workaround I used is to make a directory in the container which has the same path as my project directory in local and compile the binary under that directory.

      You don’t need the full path to make the automatic mapping work, you just need the directory of the project to be named the same both on the host machine and in the container. There is currently a single known issue with this feature, you can read more about it and the possible workarounds for it on our issue tracker: https://youtrack.jetbrains.com/issue/GO-6736 Please see if this covers your case, and if not, please let us know how to replicate it.

      > My question is that is there any other way to do custom path remapping?

      No, currently we don’t support any custom path remapping as the auto-mapping feature generally works (with the exception of the above issue). What would be the use-case for such a feature?

      > Another question is that is there any option can let me customize the behavior of IDE’s debug button which shown on the left side of my golang test function, so that I can pass the arguments to my container to run delve with target test function?

      This will be addressed when we address https://youtrack.jetbrains.com/issue/GO-3322 which is dedicated to such use-cases. Please feel free to add more comments there as to where/how you’d use the Docker support in your workflow.

      Thank you.

  14. Surendhar says:

    Is there a way to debug console applications?

  15. Andrei says:

    Hi! Thanks for the Golang debugging in Docker.
    Is it possible to debug Angular application with Docker in GoLand?

  16. Maxim Semenov says:

    Sometimes go application doesn’t stop at breakpoints. They display disabled.

    In order for the debugger to correctly match breakpoints, it is necessary that the $GOPATH be equal on the host machine and in the container.

    This helped me solve the problem.

  17. Maxim Semenov says:

    Sometimes go application doesn’t stop at breakpoints. They display disabled.
    In order for the debugger to correctly match breakpoints, it is necessary that the $GOPATH be equal on the host machine and in the container.
    This helped me solve the problem.

Leave a Reply

Your email address will not be published. Required fields are marked *