Wrong IP in logs (when using docker image behind nginx proxy)

bplein · February 2, 2023, 11:06pm

I am running LibreTranslate using Docker, specifically behind a dockerized nginx proxy.

LibreTranslate is logging the internal IP address of the proxy, 10.0.1.2, not the external address.

The same proxy is correctly passing the necessary headers to other apps running behind it, including Mastodon, Wordpress, and more.

The proxy contains (in addition to other settings), these default settings that are applied to all apps behind it:

# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;

Any thoughts on what could be breaking this? If I need additional proxy headers, I can add them. If this may be a bug, I can open a ticket on Github.

pierotofy · February 5, 2023, 11:23pm

LT will look for a X-Forwarded-For header:

github.com

LibreTranslate/LibreTranslate/blob/main/libretranslate/app.py#L65

    
      
              return ak
          
          

          
def get_json_dict(request):
              d = request.get_json()
              if not isinstance(d, dict):
                  abort(400, description=_("Invalid JSON format"))
              return d
          
          

          
def get_remote_address():
              if request.headers.getlist("X-Forwarded-For"):
                  ip = request.headers.getlist("X-Forwarded-For")[0].split(",")[0]
              else:
                  ip = request.remote_addr or "127.0.0.1"
          
          
    return ip
          
          

          
def get_req_limits(default_limit, api_keys_db, multiplier=1):
              req_limit = default_limit

But it looks like you’re already setting that; perhaps something is off in the proxy software?

Guido · February 9, 2023, 4:26pm

@bplein Did you find a solution in the meantime?

I noticed this also with a similiar setup, a dockerized nginx-proxy in front of the dockerized Libretranslate (all running with docker compose).
When I do set the variable LT_DEBUG to true, then the internal IP of the nginx proxy is shown in the Libretranslate docker logs files instead of the external IP address of the web client which is strange since it is working fine for all other dockerized containers, that they log the external IP address of the web clients.

Guido · February 9, 2023, 5:20pm

it looks like Libretranslate is not logging the remote IP address in the docker logs even in DEBUG mode.
The first IP is always the IP of the nginx proxy.

libretranslate | 172.17.1.2 - - [09/Feb/2023 16:24:36] “GET /languages HTTP/1.1” 200 -

Looking into the docker log of another nginx which is also running behind the ningx proxy I do see the external address, but the first IP is also always the internal proxy IP address:

portal-nginx-1 | 172.17.1.2 - - [09/Feb/2023:16:54:17 +0000] “GET /portal.js?v=1675961657555 HTTP/1.1” 200 16737 “https: //portal. mydomain. net/” “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0” “91.103.xxx.xxx”

And other docker compose logs files do also only show the internal IP address:

wiki-mediawiki-1 | 172.17.1.2 - - [09/Feb/2023:16:10:06 +0000] “GET /index.php?title=Wekan HTTP/1.1” 200 14942 “https: //wiki. mydomain. net/” “Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)”

So I will ask the guys who do the docker compose why the internal IP address of the nginx-proxy is logged instead of the external IP address.

bplein · February 11, 2023, 8:04pm

I also run Mastodon, Wordpress, generic HTML and other services on the same host and all are using the same nginx reverse proxy. I believe I’m doing the correct things here.

bplein · February 11, 2023, 8:05pm

Thanks for taking the time to look into this!

I’ll do some more poking around.

bplein · February 11, 2023, 8:11pm

X-Forwarded-For is a comma separated list of values that are appended onto the list each layer of proxying. I’m wondering if that code is picking up the correct one… (I don’t do any python coding, I can read it but am not sure how it’s handling lists).

I believe the first item in the list is the remote host, following items are proxy IP addresses, so the first should do it. Seems like [0] would be the first, but I’m not sure how request.headers.getlist populates that array.

bplein · February 11, 2023, 8:29pm

Well, I just ran phpinfo() on a wordpress site that is behind the same proxy and it isn’t getting the X-Forwarded-For header. So despite what I thought was the correct setup, it looks like my proxy isn’t adding those headers (I think)… time to start digging.

bplein · February 13, 2023, 9:24pm

Well, I was wrong about that. In order to reduce variables, I created a simple node container that echos back the headers, based on this code: https://github.com/craigforster/node-header-echo/blob/master/server.js

I am running this on the same server in a colocation that runs my libretranslate instance. I am hitting it from my home network

Here are the results:

HTTP headers

Header	Value
host	<redacted>
connection	close
x-real-ip	23.126.xxx.xxx
x-forwarded-for	23.126.xxx.xxx
x-forwarded-proto	https
x-forwarded-ssl	on
x-forwarded-port	443
x-original-uri	/
user-agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15
accept	text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-language	en-US,en;q=0.9
accept-encoding	gzip, deflate, br

bplein · February 13, 2023, 9:51pm

I also found https://github.com/cdfuller/echo-server and brought that up behind my proxy, and the results are:

GET / HTTP/1.1
Host: <redacted>
Connection: close
X-Real-IP: 23.126.xxx.xxx
X-Forwarded-For: 23.126.xxx.xxx
X-Forwarded-Proto: https
X-Forwarded-Ssl: on
X-Forwarded-Port: 443
X-Original-URI: /
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-language: en-US,en;q=0.9
accept-encoding: gzip, deflate, br

bplein · February 13, 2023, 10:00pm

Whoa!

The python based echo server shows the right X-Forwarded-For, but “docker-compose logs” for that app shows the wrong IP.

 docker-compose logs -f --tail=1000
pytest-echo-1  | Echoing from http://0.0.0.0:3246


pytest-echo-1  | 10.0.1.2 - Mon Feb 13 21:48:40 2023 - GET / HTTP/1.1
pytest-echo-1  | 10.0.1.2 - Mon Feb 13 21:48:46 2023 - GET / HTTP/1.1
pytest-echo-1  | 10.0.1.2 - Mon Feb 13 21:56:13 2023 - GET / HTTP/1.1
pytest-echo-1  | 10.0.1.2 - Mon Feb 13 21:56:13 2023 - GET /asdf HTTP/1.1
pytest-echo-1  | 10.0.1.2 - Mon Feb 13 21:56:21 2023 - GET /asdf/asdf/asdf/asdf HTTP/1.1

My head is spinning.

OK, so that little echo server has the correct X-Forwarded-for, but it’s debug output is using the simple client connection …

github.com

cdfuller/echo-server/blob/2318e37a25ebe570108c6f1782fa120815b75e8b/echo.py#L47

    
      
                          else:
                              bytes_left = 0
                      else:
                          request['raw'] += data
                          request['body'] += data.decode('utf-8', 'ignore')
                          bytes_left -= BLOCK_SIZE
          
          
        request_time = datetime.datetime.now().ctime()
          
          
        if verbosity > 0:
                      print(' - '.join([client_address[0], request_time, request['header']['request-line']]))
          
          
        raw_decoded = request['raw'].decode('utf-8', 'ignore')
                  response = "HTTP/1.1 200 OK\nAccess-Control-Allow-Origin: *\n\n{}".format(raw_decoded)
                  if verbosity == 2:
                      print("-"*10)
                      print(response)
                      print("-"*40)
                  connection.sendall(response.encode())
                  connection.close()
          except KeyboardInterrupt:

bplein · February 13, 2023, 10:22pm

I can’t find any logging in LibreTranslate itself (well, not much) and only a small console.log output in tempaltes/app.js.template.

I’m wondering if one of the other libraries that this is built on is doing this logging to the console, and that’s why we are seeing it.