[{"template":"item","lang":"en","langs":["en","zh-CN"],"summary":{"raw":"Previously, I used an idle IP to set up port forwarding to hide my Forgejo server's IP, which worked excellently. However, when I recently set up my email service, I discovered that using iptables for port forwarding over the public network does not allow the application to correctly obtain the client's IP. This was not an issue for informal external applications. However, since my email service uses Greylist to filter spam, the sender's IP is an important parameter. Therefore, I need to obtain the actual client IP.","html":"

Previously, I used an idle IP to set up port forwarding to hide my Forgejo server's IP, which worked excellently. However, when I recently set up my email service, I discovered that using iptables for port forwarding over the public network does not allow the application to correctly obtain the client's IP. This was not an issue for informal external applications. However, since my email service uses Greylist to filter spam, the sender's IP is an important parameter. Therefore, I need to obtain the actual client IP.

\n"},"visible":true,"routable":true,"title":"Retaining Client IP with iptables Port Forwarding on Public Networks","comment":{"enabled":true,"reply":true},"discuss":{"enabled":true},"keywords":["iptables port forwarding","iptables forwarding client ip","forward real ip","forward original ip","forward source ip"],"webmention":{"enabled":true,"accept":true,"callback":{"secret":{"hash":{"sha256":"44bb9da44b424c6dfc9233819e907029ff2ccdfaa89919d3696ddf8db7ff6647"}}}},"pingback":{"enabled":true,"accept":true},"slug":"preserving-client-ip-in-iptables-port-forwarding","taxonomy":{"category":["Internet"],"tag":["GRE","Ubuntu"],"series":[]},"route":"/preserving-client-ip-in-iptables-port-forwarding/","toc":{"enabled":true},"published":{"date":"2024-07-12 07:12"},"modified":{"date":"2024-08-04T17:15:23.457Z"},"data":{"x.com":{"status":null},"nostr":{"note":null}},"license":"CC-BY-NC-SA-4.0","author":[{"email":{"value":{"encrypted":"ODgyZjdiZDg4OTVmMjllOGZkNmFmYg==","algorithm":"aes-256-gcm","iv":"067f6d66f2ba5dd50c8443be"},"hash":{"md5":"8770854a9222b6bf1ee195220793bcf1"}},"url":"https://dallas.lu","name":"Dallas Lu","lang":"en","user":"dallaslu"}],"contributor":[],"sponsor":[],"image":[],"audio":[],"video":[],"photo":[],"content":{"html":"

\n\n

flowchart LR\n C[Client] --> B[Proxy] --> A[Server]

Forwarding at the Gateway

Forwarding a port from the gateway to a server within the internal network is very simple and can be done with a single command:

iptables -t nat -A PREROUTING -p tcp -d 1.1.1.1 --dport 80 -j DNAT --to-destination 10.0.0.2:80\n

flowchart LR\n subgraph Client\n C[3.3.3.3]\n end\n\n subgraph Subnet\n subgraph Proxy\n eth0[1.1.1.1] --> eth1[10.0.0.1]\n eth1 -.-> eth0\n end\n subgraph Server\n eth1 --3.3.3.3->10.0.0.2--> A[10.0.0.2]\n A -.->|10.0.0.2>3.3.3.3| eth1\n end \n end\n\n C --3.3.3.3->1.1.1.1--> eth0\n eth0 -.->|1.1.1.1->3.3.3.3| C

In the internal network server application, the correct client IP can be directly obtained. Whether using tcpdump or netstat, you can see that the IP from the internet has established a connection with the local machine.

Public Network Forwarding

Similarly, you need to specify the forwarding target:

iptables -t nat -A PREROUTING -p tcp -d 1.1.1.1 --dport 80 -j DNAT --to-destination 2.2.2.2:80\n

So far, this forwarding does not work correctly. When 3.3.3.3 from the internet communicates with 1.1.1.1:80, the packet is forwarded to 2.2.2.2:80. When the web server returns the handshake message, its packet should follow its own routing rules and is returned to 3.3.3.3 via 2.2.2.2. This connection cannot be established.

flowchart LR\n subgraph Client\n C[3.3.3.3]\n end\n\n subgraph Proxy\n eth0[1.1.1.1]\n end\n subgraph Server\n eth1[2.2.2.2]\n end \n\n eth0 --3.3.3.3>2.2.2.2-->eth1\n\n C --3.3.3.3->1.1.1.1--> eth0\n\n eth1 ~~~|2.2.2.2->3.3.3.3| C

Therefore, we need to change the source of the packet on 1.1.1.1 to 1.1.1.1 to forward the response from 2.2.2.2, ensuring normal communication and connection with 3.3.3.3.

flowchart LR\n subgraph Client\n C[3.3.3.3]\n end\n\n subgraph Proxy\n eth0[1.1.1.1]\n end\n subgraph Server\n eth1[2.2.2.2]\n end \n\n eth0 --1.1.1.1>2.2.2.2-->eth1\n eth1 -.->|2.2.2.2>1.1.1.1|eth0\n\n C --3.3.3.3->1.1.1.1--> eth0\n eth0 -.->|1.1.1.1->3.3.3.3| C

using SNAT：

iptables -t nat -A POSTROUTING -p tcp -d 2.2.2.2 --dport 80 -j SNAT --to-source 1.1.1.1\n

Using restricted MASQUERADE

iptables -t nat -A POSTROUTING -p tcp -d 2.2.2.2 --dport 80 -j MASQUERADE\n

Using broad MASQUERADE

iptables -t nat -A POSTROUTING ! -o lo -j MASQUERADE\n

This also makes 2.2.2.2 always think that 1.1.1.1 is communicating with it, completely unaware of the existence of 3.3.3.3.

Retaining Client IP with Public Network Forwarding

If a packet sent from 1.1.1.1 to 2.2.2.2 has the source IP as the original client IP, say 3.3.3.3, and the destination as 2.2.2.2. When 2.2.2.2 receives the data, if there are some filtering rules, it may not respond because the route from 2.2.2.2 to 3.3.3.3 is different from where the packet came from.

If we disable these filtering rules, 2.2.2.2 may be more vulnerable to attacks, as it becomes hard to differentiate. If we configure an extra IP on 2.2.2.2 specifically to accept forwarding from 1.1.1.1, it would be wasteful. Changing 2.2.2.2's routing to make all its traffic go through 1.1.1.1 would make it unable to handle direct requests from the internet normally.

Establishing a Dedicated Tunnel

Some proxy implementations, such as Haproxy or Transport Proxy, require 2.2.2.2 to have application-level support. For example, in nginx, you need to explicitly specify proxy_protocol or transport. If the application does not support it, it cannot be used.

Moreover, this scenario only involves direct connection between two nodes, without needing load balancing or high availability. Using iptables is simpler than using proxy software.

Therefore, a better approach is to establish a dedicated tunnel between 1.1.1.1 and 2.2.2.2. There are many ways to establish a tunnel. Similarly, there are many other choices for establishing a tunnel, such as OpenVPN / WireGuard. Given the requirements of this scenario, I chose the simpler GRE.

GRE

Establish the tunnel on 1.1.1.1 and configure the IP address.

ip tunnel add gre1 mode gre remote 2.2.2.2 local 1.1.1.1 ttl 255\nip addr add 10.0.0.1/24 dev gre1\nip link set gre1 up\n

Forward the local public network traffic to the other end of the tunnel.

iptables -t nat -A PREROUTING -p tcp -d 1.1.1.1 --dport 80 -j DNAT --to-destination 10.0.0.2:80\n

Establish the tunnel on 2.2.2.2 and configure the IP address.

ip tunnel add gre1 mode gre remote 1.1.1.1 local 2.2.2.2 ttl 255\nip addr add 10.0.0.2/24 dev gre1\nip link set gre1 up\n

There is no need to disable rp_filter. Add the routing table:

echo \"200 custom\" >> /etc/iproute2/rt_tables\n

Set routing table rules and routing information to let the traffic at this tunnel endpoint return to its origin.

ip rule add from 10.0.0.2 table custom\nip route add default via 10.0.0.1 dev gre1 table custom\n

flowchart LR\n subgraph Client\n C[3.3.3.3]\n end\n\n subgraph Proxy\n subgraph b[1.1.1.1]\n eth1[10.0.0.1]\n end\n end\n subgraph Server \n subgraph a[2.2.2.2]\n eth11[10.0.0.2]\n end \n end\n\n b o--o|GRE| a\n eth1 -->|3.3.3.3->10.0.0.2| eth11\n\n eth11 -.->|10.0.0.2->3.3.3.3| eth1\n C --3.3.3.3->1.1.1.1--> b\n b -.->|1.1.1.1->3.3.3.3| C

Keep alive

crontab -e\n

*/1 * * * * ping -c 60 10.0.0.2\n

Conclusion

Using iptables port forwarding and GRE tunnels, you can easily hide the real IP address while still obtaining the actual client IP. This process does not require installing additional software and is applicable to various applications without needing further adaptation. It's an excellent solution for using an idle VPS as a front-end server, fully utilizing a VPS with a good connection to provide multiple IP entry points for your application, making it faster and more secure.

\n","headings":[{"level":2,"text":"Forwarding at the Gateway","id":"forwarding-at-the-gateway"},{"level":2,"text":"Public Network Forwarding","id":"public-network-forwarding"},{"level":2,"text":"Retaining Client IP with Public Network Forwarding","id":"retaining-client-ip-with-public-network-forwarding"},{"level":3,"text":"Establishing a Dedicated Tunnel","id":"establishing-a-dedicated-tunnel"},{"level":4,"text":"GRE","id":"gre"},{"level":4,"text":"Keep alive","id":"keep-alive"},{"level":2,"text":"Conclusion","id":"conclusion"}],"meta":{"mermaid":true,"prism":true},"links":[]},"newer":"/current-progress-and-outlook-on-comment-system-as-of-aug/","earlier":"/experience-with-postal/"},{"template":"item","lang":"en","langs":["en","zh-CN"],"summary":{"raw":"In PVE, it is easy to expand the disk using its WebUI. So when you create a VM, you don't have to allocate an oversized disk. There is still a chance when you finally realize that the disk is too big. This article describes the experience and methodology of shrinking a disk for an Ubuntu VM.","html":"

In PVE, it is easy to expand the disk using its WebUI. So when you create a VM, you don't have to allocate an oversized disk. There is still a chance when you finally realize that the disk is too big. This article describes the experience and methodology of shrinking a disk for an Ubuntu VM.

\n"},"visible":true,"routable":true,"title":"Shrinking the disk of an Ubuntu VM in PVE","comment":{"enabled":true,"reply":true},"discuss":{"enabled":true},"keywords":["PVE shrink VM disk","PVE reduce VM disk","PVE Reduce Shrink Virtual Machine Hard Disk"],"webmention":{"enabled":true,"accept":true,"callback":{"secret":{"hash":{"sha256":"44bb9da44b424c6dfc9233819e907029ff2ccdfaa89919d3696ddf8db7ff6647"}}}},"pingback":{"enabled":true,"accept":true},"slug":"pve-reduce-ubuntu-vm-disk-size","taxonomy":{"category":["Internet"],"tag":["PVE","Ubuntu"],"series":[]},"route":"/pve-reduce-ubuntu-vm-disk-size/","toc":{"enabled":true},"published":{"date":"2024-02-11 11:02"},"modified":{"date":"2024-07-28T15:12:05.775Z"},"data":{},"license":"CC-BY-NC-SA-4.0","author":[{"email":{"value":{"encrypted":"ODgyZjdiZDg4OTVmMjllOGZkNmFmYg==","algorithm":"aes-256-gcm","iv":"067f6d66f2ba5dd50c8443be"},"hash":{"md5":"8770854a9222b6bf1ee195220793bcf1"}},"url":"https://dallas.lu","name":"Dallas Lu","lang":"en","user":"dallaslu"}],"contributor":[],"sponsor":[],"image":[],"audio":[],"video":[],"photo":[],"content":{"html":"

\n\n

This site is running in a VM with another Ubuntu VM running Nginx as the frontend. The frontend machine was originally allocated 128GB of disk space, but after running for half a year, it's actually taking up less than 10GB of disk space, which isn't a lot of space, and some of the new files are just logs and configuration backups. So I'm planning to reduce the allocated disk space to 32GB for backup and transfer to SSD. If the space requirement grows in the future, I can expand the capacity at any time, or mount another HDD disk as the storage space for logs and backups.

If you have the same need, you can try it as below with a safe backup of your data. Compared with disk reduction ¹ for LXC containers, disk resizing for VMs is more complicated, you need to compress the disk in the VM first to free up space, then go to PVE to operate, and finally go back to the VM to repair the partition.

Information Collection

To avoid IO errors, data loss², you should compress the partition size in the VM first. Before that, you should make a backup of the VM in the PVE management interface.

Next, log in to your Ubuntu VM and view the current disk partition:

lsblk\n

This Ubuntu VM uses the default partition settings when installing the system, and the partition table is MBR. (If you have a GUID partition table, then see Egidio Docile's article,How to manipulate gpt partition tables with gdisk and sgdisk on Linux）

The hard disk is divided into three partitions, the device path is /dev/sda, and the last partition is mounted as the root directory. Run:

df -h\n

Check disk space usage to verify that the tuning solution is feasible.

SystemRescueCD

Next use SystemRescueCD³ for disks

Get the ISO download link from https://www.system-rescue.org/Download/, go to PVE and add the ISO image from the URL. Mount the image in an Ubuntu virtual machine and boot from the CD. Once inside the system, run:

startx\n

to access the desktop environment.

Information Confirmation

Open GParted to view the current partitioning. Run:

lvs\nlvdisplay\n

Confirm the usage of the target partition, and the device path. Here it is assumed to be /dev/ubuntu-vg/ubuntu-lv.

Reduce Disk

WARNINGDouble-check

Reduce Disk in VM

To continue reduce the virtual volume, run:

lvreduce --resizefs --size -96G /dev/ubuntu-vg/ubuntu-lv\n

Resizing a Partition

Use GParted to resize the partition to a suitable size, right-click on the target partition -> Move/Resize to compress the size of the target partition and free up unallocated space.

Resize Disk in PVE

Back to PVE shell, run:

lvresize --size -96G /dev/pve/vm-100-disk-0\n

Note that this step has a chance to be undone if done incorrectly:

lvresize --size +96G /dev/pve/vm-100-disk-0\n

For the PVE system to display the disk size correctly, it is also necessary to perform ⁴:

qm rescan\n

Repairing the Partition Table

Reboot the VM, re-enter SystemRescue, run:

fdisk -l\n

It will prompt GPT PMBR mismatch... . Run:

gdisk /dev/sda\n

Enter v to verify that the partition table is repaired;
Enter w to write to the partition table.

Run fdisk -l again to verify.

Ending

Remove the CD/DVD drive and restart the VM. If you have any problems with your system or data, then bless you that your backup is still there :)

References

\n
Yomi Ikuru. Proxmox - Resizing a LXC Disk. Yomi’s Blog. 2021. ↩
\n
\n
Feng. Proxmox VE (PVE) lvm thin 减小缩减虚拟机硬盘设置的空间大小(Reducing the amount of space set aside for VM hard disks). D3TT. 2019. \"需要先从虚拟机内部缩小分区，然后再执行下边操作，比较繁琐，易翻车，务必先备份数据 (You need to shrink the partition from inside the VM first, and then perform the following operations, more cumbersome, easy to turn over, be sure to back up the data first!)\" ↩
\n
\n
https://pve.proxmox.com/wiki/Resize_disks 'If you reduce (shrink) the hard disk, of course removing the last disk plate will probably destroy your file system and remove the data in it! So in this case it is paramount to act in the VM in advance, reducing the file system and the partition size. SystemRescueCD comes very handy for it, just add its iso as cdrom of your VM and set boot priority to CD-ROM.' ↩
\n
\n
aaron. Reply to lumox in 'How to resize a VM's disk'\n#6. Proxmox Forum. 2020. ↩
\n

\n","headings":[{"level":2,"text":"Information Collection","id":"information-collection"},{"level":2,"text":"SystemRescueCD","id":"systemrescuecd"},{"level":2,"text":"Information Confirmation","id":"information-confirmation"},{"level":2,"text":"Reduce Disk","id":"reduce-disk"},{"level":3,"text":"Reduce Disk in VM","id":"reduce-disk-in-vm"},{"level":3,"text":"Resizing a Partition","id":"resizing-a-partition"},{"level":3,"text":"Resize Disk in PVE","id":"resize-disk-in-pve"},{"level":2,"text":"Repairing the Partition Table","id":"repairing-the-partition-table"},{"level":2,"text":"Ending","id":"ending"}],"meta":{"prism":true},"links":[{"href":"https://linuxconfig.org/how-to-manipulate-gpt-partition-tables-with-gdisk-and-sgdisk-on-linux","type":"mention"},{"href":"https://www.system-rescue.org","type":"mention"},{"href":"https://www.system-rescue.org/Download/","type":"mention"},{"href":"https://yomis.blog/proxmox-resizing-a-disk/","type":"mention"},{"href":"https://www.d3tt.com/view/116","type":"mention"},{"href":"https://pve.proxmox.com/wiki/Resize_disks","type":"mention"},{"href":"https://forum.proxmox.com/threads/how-to-resize-a-vms-disk.79349/post-351732","type":"mention"}]},"newer":"/pve-network-manual-config-and-ipv6/","earlier":"/beancount-periodic-plugin-for-amortization-and-depreciation/"},{"template":"item","lang":"en","langs":["en","zh-CN"],"summary":{"raw":"Gitea is forked from Gogs. It is said that Gogs developers don't accept extermal PR, so they turn to Gitea, and Forgejo is forked from Gitea, which is a community version of Gitea after Gitea was commercialized.","html":"

Gitea is forked from Gogs. It is said that Gogs developers don't accept extermal PR, so they turn to Gitea, and Forgejo is forked from Gitea, which is a community version of Gitea after Gitea was commercialized.

\n"},"visible":true,"routable":true,"title":"Install Forgejo on Ubuntu","comment":{"enabled":true,"reply":true},"discuss":{"enabled":true},"keywords":["self-hosted Git","Install Gitea","self-hosted Github","private Github","reverse proxy"],"webmention":{"enabled":true,"accept":true,"callback":{"secret":{"hash":{"sha256":"44bb9da44b424c6dfc9233819e907029ff2ccdfaa89919d3696ddf8db7ff6647"}}}},"pingback":{"enabled":true,"accept":true},"slug":"install-forgejo-on-ubuntu","taxonomy":{"category":["Internet"],"tag":["Git","Ubuntu","Forgejo","Gitea","Nginx","Self-hosted"],"series":[]},"route":"/install-forgejo-on-ubuntu/","toc":{"enabled":true},"published":{"date":"2022-12-27 12:27"},"modified":{"date":"2024-06-28T14:30:30.902Z"},"data":{},"license":"WTFPL","author":[{"email":{"value":{"encrypted":"ODgyZjdiZDg4OTVmMjllOGZkNmFmYg==","algorithm":"aes-256-gcm","iv":"067f6d66f2ba5dd50c8443be"},"hash":{"md5":"8770854a9222b6bf1ee195220793bcf1"}},"url":"https://dallas.lu","name":"Dallas Lu","lang":"en","user":"dallaslu"}],"contributor":[],"sponsor":[],"image":[],"audio":[],"video":[],"photo":[],"content":{"html":"

\n\n

Add dedicated user

# Ubuntu\nsudo adduser --system --group --disabled-password --shell /bin/bash --home /home/git --gecos 'Git Version Control' git\n

Prepare folders

sudo mkdir -p /var/lib/forgejo/{custom,data,indexers,public,log}\nsudo chown git:git /var/lib/forgejo/{data,indexers,log}\nsudo chmod 750 /var/lib/forgejo/{data,indexers,log}\nsudo mkdir /etc/forgejo\nsudo chown root:git /etc/forgejo\nsudo chmod 770 /etc/forgejo\n

Download installation file

According to the introduction of forgejo's download page ¹, download the installation file:

wget https://codeberg.org/forgejo/forgejo/releases/download/v1.19.3-0/forgejo-1.19.3-0-linux-amd64\nchmod +x forgejo-1.19.3-0-linux-amd64\n

Verify the signature

gpg --keyserver keys.openpgp.org --recv EB114F5E6C0DC2BCDD183550A4B61A2DC5923710\nwget https://codeberg.org/forgejo/forgejo/releases/download/v1.19.3-0/forgejo-1.19.3-0-linux-amd64.asc\ngpg --verify forgejo-1.19.3-0-linux-amd64.asc forgejo-1.19.3-0-linux-amd64\n

Move to local folder

sudo mv forgejo-1.19.3-0-linux-amd64 /usr/local/bin/forgejo\n

Install dependencies

Git

apt install git\n

Mariadb

apt install mariadb-server\nmysql_secure_installation\nmysql -u root -p\n

CREATE DATABASE forgejo;\nCREATE USER 'forgejo'@'localhost' IDENTIFIED BY '<YOUR_PASSWORD>';\nGRANT ALL ON forgejo.* TO 'forgejo'@'localhost';\nFLUSH PRIVILEGES;\nEXIT;\n

Install as service

vim /usr/lib/systemd/system/forgejo.service\n

[Unit]\nDescription=Forgejo\nAfter=network.target\nAfter=mariadb.service\n\n[Service]\n# Modify these two values and uncomment them if you have\n# repos with lots of files and get an HTTP error 500 because\n# of that\n###\n#LimitMEMLOCK=infinity\n#LimitNOFILE=65535\nRestartSec=2s\nType=simple\nUser=git\nGroup=git\nWorkingDirectory=/var/lib/forgejo/\nExecStart=/usr/local/bin/forgejo web -c /etc/forgejo/app.ini\nRestart=always\nEnvironment=USER=git HOME=/home/git GITEA_WORK_DIR=/var/lib/forgejo\n# If you want to bind to a port below 1024 uncomment\n# the two values below\n###\n#CapabilityBoundingSet=CAP_NET_BIND_SERVICE\n#AmbientCapabilities=CAP_NET_BIND_SERVICE\n\n[Install]\nWantedBy=multi-user.target\n

sudo systemctl daemon-reload\nsudo systemctl enable --now forgejo\n\nsudo firewall-cmd --add-port 3000/tcp --permanent\nsudo firewall-cmd --reload \n

Nginx Configuration

You can configure an SSL host with the domain name git.example.com and add reverse proxy config：

location ^~ / {\n        proxy_redirect off;\n        proxy_set_header X-Forwarded-Proto https;\n        proxy_set_header Host $host;\n        proxy_set_header X-Real-IP $remote_addr;\n        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n\n        if (!-e $request_filename) {\n                proxy_pass http://127.0.0.1:3000;\n                break;\n        }\n}\n

Installation Guide

Visit https://git.example.com/install, follow the wizard to complete the installation process.

Adjust Forgejo configuration

vim /etc/forgejo/app.ini\n

(Reference: app.example.ini)

Just restart Forgejo after modifying the configuration.

References

\n
https://forgejo.org/download/ ↩
\n

\n","headings":[{"level":2,"text":"Add dedicated user","id":"add-dedicated-user"},{"level":2,"text":"Prepare folders","id":"prepare-folders"},{"level":2,"text":"Download installation file","id":"download-installation-file"},{"level":3,"text":"Verify the signature","id":"verify-the-signature"},{"level":3,"text":"Move to local folder","id":"move-to-local-folder"},{"level":2,"text":"Install dependencies","id":"install-dependencies"},{"level":3,"text":"Git","id":"git"},{"level":3,"text":"Mariadb","id":"mariadb"},{"level":2,"text":"Install as service","id":"install-as-service"},{"level":2,"text":"Nginx Configuration","id":"nginx-configuration"},{"level":2,"text":"Installation Guide","id":"installation-guide"},{"level":2,"text":"Adjust Forgejo configuration","id":"adjust-forgejo-configuration"}],"meta":{"prism":true},"links":[{"href":"https://codeberg.org/forgejo/forgejo/src/branch/forgejo/custom/conf/app.example.ini","type":"mention"},{"href":"https://forgejo.org/download/","type":"mention"}]},"newer":"/something-about-nostr-protocol/","earlier":"/apache-subversion-behind-nginx/"}]

Tag: Ubuntu

Retaining Client IP with iptables Port Forwarding on Public Networks

Shrinking the disk of an Ubuntu VM in PVE

2024

Install Forgejo on Ubuntu

2022