Document how to change the MTU

Signed-off-by: Jian Wang <[email protected]>

docs/advanced/storagenetwork.md

@@ -140,6 +140,10 @@ Because of the design, Harvester will treat extra and insignificant characters i
 
 :::
 
+#### Change the MTU of the Storage Network
+
+Follow the guide [Change the MTU of Network Config which has Storage Network Attached](../networking/clusternetwork.md#change-the-mut--of-network-config-which-has-storage-network-attached) to change the MTU of the Storage Network.
+
 ### After Applying Harvester Storage Network Setting
 
 After applying Harvester's Storage Network setting, Harvester will stop all pods that are related to Longhorn volumes. Currently, Harvester has some pods listed below that will be stopped during setting.

docs/networking/clusternetwork.md

@@ -93,9 +93,11 @@ The [witness node](../advanced/witness.md) is generally not involved in the cust
 
 :::
 
-#### How to create a new cluster network
+## Configurations
 
-1. To create a cluster network, go to the **Networks > ClusterNetworks/Configs** page and click the **Create** button. You only need to specify the name.
+### Create a new Cluster Network
+
+1. To create a new cluster network, go to the **Networks > ClusterNetworks/Configs** page and click the **Create** button. You only need to specify the name.
 
    ![](/img/v1.2/networking/create-clusternetwork.png)
 
@@ -114,7 +116,7 @@ The method **Select all nodes** works only when all nodes use the exact same ded
 :::
 
 4. Click the **Uplink** tab to add the NICs, and configure the bond options and link attributes. The bond mode defaults to `active-backup`.
-    
+
    ![](/img/v1.2/networking/config-uplink.png)
 
 :::note
@@ -131,3 +133,102 @@ Starting with Harvester v1.1.2, Harvester supports updating network configs. Mak
 To simplify cluster maintenance, create one network configuration for each node or group of nodes. Without dedicated network configurations, certain maintenance tasks (for example, replacing old NICs with NICs in different slots) will require you to stop and/or migrate the affected VMs before updating the network configuration.
 
 :::
+
+### Edit the Network Config
+
+Changes on the existing `Network Config` may affect both the Harvester VMs/workloads and the external devices/systems like Switches/Routers.
+
+#### Change the MTU of Network Config which has no Storage Network attached
+
+You may plan to change the `MTU` of an existing `Cluster Network`. And the [Storage Network](../advanced/storagenetwork.md#harvester-storage-network-setting) is not enabled or not attached to this `Cluster Network`.
+
+The `MTU` on each `Network Config` of an existing custom `Cluster Network` is strictly to be identical. There are many restrictions to change the `MTU`, the following steps should be followed:
+
+1. Stop all the VMs which are attached to the target `Cluster Network`, this can be checked via the [VM Network](./harvester-network.md#create-a-vm-network) and [VM attached Secondary Network](../vm/create-vm.md#secondary-network). When any of the VMs is still running, Harvester will refuse the change.
+
+2. Check the number of `Network Config` of the target `Cluster Network`, if the number is greater than one, then repeat the operations below until there is only one `Network Config` left:
+
+    1. Record the `Node Selector` of a `Network Config`;
+
+    1. Remove this `Network Config`;
+
+3. Change the `MTU` of the last `Network Config`.
+
+4. Check the `MTU` on the selected Harvester nodes via the Linux `ip link` command, the related `*-br` device should be `UP` and with the new `MTU`.
+
+```
+Harvester node $ ip link
+
+                                                  |new MTU|              |state UP|
+3: cn-data-br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
+    link/ether 52:54:00:6e:5c:2a brd ff:ff:ff:ff:ff:ff
+```
+
+5. Test the new MTU on Harvester nodes via command like `ping` to another Harvester node (with the new MTU) or an external device.
+
+```
+Harvester node $ ping 8.8.8.8 -s 8800
+
+PING 8.8.8.8 (8.8.8.8) 8800(8828) bytes of data
+
+# the `-s` specify the ping packet size, which can test if the new MTU really works
+
+```
+
+6. Add those `Network Config` which were removed on Step 2, each should set the `MTU` to the new value. And run step 4, 5 to test.
+
+7. Start the VMs mentioned in step 1.
+
+:::note
+
+- The `MTU` affects both of the Harvester nodes and the infrastructure layer networking devices like Switches and Routers, the careful planning and testing are required to sure the new MTU.
+
+- This method does not work on the built-in `mgmt Cluster Network`.
+
+:::
+
+#### Change the MTU of Network Config which has Storage Network Attached
+
+You may plan to change the `MTU` of an existing `Cluster Network`. And the [Storage Network](../advanced/storagenetwork.md#harvester-storage-network-setting) is enabled or attached to this `Cluster Network`. The `Storage Network` is dedicatedly used by `Longhorn`, the default CSI driver of Harvester cluster. Because the `Longhorn` is responsible for at least the [root disk](../vm/create-vm.md#volumes) of all VMs, this change will affect all the VMs.
+
+The `MTU` on each `Network Config` of an existing custom `Cluster Network` is strictly to be identical. There are many restrictions to change the `MTU`, the following steps should be followed:
+
+1. Stop all the VMs.
+
+2. Disable the Harvester [Storage Network](../advanced/storagenetwork.md#harvester-storage-network-setting). Wait until the [change is applied](../advanced/storagenetwork.md#harvester-storage-network-setting#step-1).
+
+3. Check the number of `Network Config` of the target `Cluster Network`, if the number is greater than one, then repeat the operations below until there is only one `Network Config` left:
+
+    1. Record the `Node Selector` of a `Network Config`;
+
+    1. Remove this `Network Config`;
+
+4. Change the `MTU` of the last `Network Config`.
+
+5. Check the `MTU` on the selected Harvester nodes via the Linux `ip link` command, the related `*-br` device should be `UP` and with the new `MTU`.
+
+```
+Harvester node $ ip link
+
+                                                  |new MTU|              |state UP|
+
+3: cn-data-br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
+    link/ether 52:54:00:6e:5c:2a brd ff:ff:ff:ff:ff:ff
+```
+
+6. Test the new MTU on Harvester nodes via command like `ping` to another Harvester node (with the new MTU) or an external device.
+
+```
+Harvester node $ ping 8.8.8.8 -s 8800
+
+PING 8.8.8.8 (8.8.8.8) 8800(8828) bytes of data
+
+# the `-s` specify the ping packet size, which can test if the new MTU really works
+
+```
+
+7. Add those `Network Config` which were removed on Step 2, each should set the `MTU` to the new value. And run step 4, 5 to test.
+
+8. Enable and set the Harvester [Storage Network](../advanced/storagenetwork.md#harvester-storage-network-setting), note the [Prerequisites](../advanced/storagenetwork.md#prerequisites) are met in the above steps. Wait until the [change is applied](../advanced/storagenetwork.md#harvester-storage-network-setting#step-1).
+
+9. Start the VMs mentioned in step 1.

docs/networking/harvester-network.md

@@ -69,6 +69,14 @@ The [Harvester network-controller](https://github.com/harvester/harvester-networ
 
     ![](/img/v1.2/networking/create-vlan-network.png)
 
+:::note
+
+The `VM Network` inheriates the `MTU` from the `Network Config` of the `Cluster Network`, this makes sure the VM will get the best match and best performance from the hardware. A different `MTU` on `VM Network` is not allowed.
+
+Follow the guide [Change the MTU of Network Config which has Storage Network Attached](./clusternetwork.md#change-the-mut--of-network-config-which-has-storage-network-attached) or [Change the MTU of Network Config which has no Storage Network Attached](./clusternetwork.md#change-the-mut--of-network-config-which-has-no-storage-network-attached) to set a different `MTU`.
+
+:::
+
 1. On the Route tab, select an option and then specify the related IPv4 addresses.
 
     - Auto(DHCP): The Harvester network controller retrieves the CIDR and gateway addresses from the DHCP server. You can specify the DHCP server address.