NGNIX Troubleshooting

Validating Upstream Parameters

To validate parameters passed to upstream services, the request should be dumped by the service or intercepted by another service temporarily. There is a docker container to do this. This will dump the received headers from both http and https communication to the upstream service.

0640 root root docker-compose.yml
http-echo:
  image: mendhak/http-https-echo

location / {
   proxy_pass http://http-echo/;
}

Note

Headers will be dumped directly to the page.

Debug NGINX configs

There is no existing logging functionality in NGINX to write directly to logs from configuration files. Work around by directly injecting debugging headers in configuration files to dump information to logs. NGINX variables may be used as well.

add_header X-debug-message "some message to write $ssl_client_s_dn" always;

Headers are found in the page response.

../../_images/debug-headers.png

If is Evil

If operates as a rewrite and is inherently misunderstood.

Within a location block the only safe operations are:

  • return.

  • rewrite.

All if operations must be explicitly tested for appropriate behavior. Other evil examples.

Dump Loaded NGINX Configuration

Dump the currently loaded configuration in config file formatting. Useful to inspect current nginx state.

nginx -T

NGINX Queries Originate from Wrong Gateway

Docker does not provide a way to set the appropriate default gateway for multi-network containers. This results in non-deterministic source IP routing.

Warning

When a container is connected to multiple networks, its external connectivity is provided via the first non-internal network, in lexical order.

NGINX expresses this bug by forwarding/proxying any traffic over the default gateway for the first lexical named network that appears.

The current fix is to inspect the container and find the first gateway listed in the connected networks. This will be the default gateway for the container.

There is currently no clean way to set a default gateway via compose.

docker inspect proxy_nginx_1

Forward Traffic via Specific Interfaces

NGINX can forward traffic via specific interfaces for location definitions.

0640 root root {SERVICE}/docker-compose.yml
networks:
  custom_net_name:
    external: true

services:
  my_proxy:
    networks:
      my_proxy_network:
        ipv4_address: 172.1.1.1
      custom_net_name:
        ipv4_address: 172.2.1.1

  • custom_net_name is a network defined in another container. Once this is added, the proxy container will be able to do DNS resolution of container names as usual, including proxying traffic to that network.

  • Use IPv4 address for proxy_bind command for specific locations.

location / {
  proxy_bind {NGINX NETWORK IP};
  proxy_pass ...
}

Not Starting with Docker Services

Expresses as NGINX not starting or failing to resolve if running (and backend services were restarted or down at some point).

See Enable NGINX Start/Running with Backends Down to resolve.