Floating IP in OpenStack Neutron

• 前言

Floating IP 是相对于Fixed IP而言的，它一般是在VM创建后分配给VM的，可以达到的目的就是，外界可以访问通过这个Floating Ip访问这个VM，VM也可以通过这个IP访问外界。

在OpenStack中，这个Floating IP使用了namespace内的iptables建立NAT 转发机制来达到VM与外界的通讯的。这片文章主要讲述如何使用OpenStack搭建和使用Floating IP.

• Environment Setup

1. Ubuntu 14.04 LTS
2. 2个网卡，分别是eth0（192.168.1.46） 和 eth1（192.168.2.46，这个interface在switch使用vlanid=100）
3. 如果你的机器是VM，请参照http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004099，去开启eth0 和eth1的promisic模式， 这一步很关键，如果没有启用，外界是无法ping通分配给VM的Floating IP的

• Devstack Configuration

1. 启用Nova, Neutron, Cinder, Keystone, Glance
2. 安装OpenvSwitch，然后做如下配置

3. # Add eth1 into br-eth1 and set eth0 and eth1 to promisc mode
   sudo ovs-vsctl br-exists br-eth1 || sudo ovs-vsctl add-br br-eth1
   sudo ovs-vsctl --may-exist add-port br-eth1 eth1
   sudo ip link set dev eth1 promisc on
   sudo ip link set dev eth0 promisc on
   sudo ip addr flush eth1
   sudo ip link set dev eth1 up

• Devstack local.conf

• #  Note: please change XX to your devstack node IP,  change VLAN_START and VLAN_END to your reserved vlan range Here
  # 我的IP是192.168.1.46， 所在网络是192.168.1.0/24
  [[local|localrc]]
  # Set API endpoint host using HOST_IP
  HOST_IP=XX
  # Use to specify the endpoint
  SERVICE_HOST=XX
  
  ADMIN_PASSWORD=welcome
  MYSQL_PASSWORD=welcome
  RABBIT_PASSWORD=welcome
  SERVICE_PASSWORD=welcome
  SERVICE_TOKEN=welcome
  disable_service h-eng
  disable_service h-api
  disable_service h-api-cfn
  disable_service h-api-cw
  disable_service tempest
  #enable_service tempest
  disable_service dstat
  # Enable Neturon
  disable_service n-net
  enable_service q-svc
  enable_service q-agt
  enable_service q-dhcp
  enable_service q-l3
  enable_service q-meta
  enable_service neutron
  
  # stack.sh will freshen each repo on each run if RECLONE
  # is set to yes
  RECLONE=True
  
  # Setting OFFLINE=True to enable stack.sh to run multiple
  # times without an Internet connection
  OFFLINE=False
  
  # Set FLAT_INTERFACE to the Ethernet interface that connects
  # the host to your local network
  FLAT_INTERFACE=eth1
  #FLOATING_RANGE=192.168.1.0/
  #Q_FLOATING_ALLOCATION_POOL=start=192.168.1.100,end=192.168.1.119
  #PUBLIC_NETWORK_GATEWAY=192.168.1.30
  PUBLIC_INTERFACE=eth0
  PHYSICAL_NETWORK=public_eth1
  OVS_PHYSICAL_BRIDGE=br-ex
  #PUBLIC_BRIDGE=br-ex
  
  FIXED_RANGE=192.168.100./
  #FIXED_NETWORK_SIZE=
  NETWORK_GATEWAY=192.168.100.
  
  # IMAGE_URLS accepts a comma separated list of images to pre-load into OpenStack
  IMAGE_URLS=http://download.cirros-cloud.net/0.3.4/cirros-0.3.4-x86_64-disk.img
  [[post-config|$NOVA_CONF]]
  [DEFAULT]
  rpc_response_timeout=
  service_down_time=
  libvirt_iscsi_use_multipath = True
  #force_config_drive = False
  
  [libvirt]
  iscsi_use_multipath = True
  
  [database]
  max_pool_size=
  max_overflow=
  
  [[post-config|/$Q_PLUGIN_CONF_FILE]]
  [ml2]
  tenant_network_types = vlan,flat
  [ml2_type_flat]
  flat_networks = public_eth0
  [ml2_type_vlan]
  network_vlan_ranges = public_eth1:VLAN_START:VLAN_END
  
  [ovs]
  bridge_mappings = public_eth0:br-ex,public_eth1:br-eth1
  enable_tunneling = False

• Setup flat and vlan network​

1. 接下来，我要做两件事，一个是创建一个VLAN ID为100的private network，以后的虚拟机默认是从这个网络分配IP的。
2. 然后是一个flat的public network，这个网络上是用来分配floating IP

创建private network，VLAN　ID为100

stack@openstack-wangp11-:~/devstack$ neutron net-create --provider:network_type vlan  --provider:physical_network public_eth1 --provider:segmentation_id  vlan_100​
stack@openstack-wangp11-:~/devstack$ neutron net-show vlan_100
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 2426ff0d-953f-467f-a564-c4f63d926836 |
| mtu                       |                                     |
| name                      | vlan_100                             |
| port_security_enabled     | True                                 |
| provider:network_type     | vlan                                 |
| provider:physical_network | public_eth1                          |
| provider:segmentation_id  |                                   |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | fab94e2f-f7de-4bf6-8e9b-7ade0fcfad1f |
| tenant_id                 | 8cb8c084ffb84914b41d5044ecbcad4e     |
+---------------------------+--------------------------------------+
stack@openstack-wangp11-:~/devstack$ neutron subnet-create --enable-dhcp  --ip-version  --name vlan_100_subnet01 2426ff0d-953f-467f-a564-c4f63d926836  192.168.46.0/​
stack@openstack-wangp11-:~/devstack$ neutron subnet-show fab94e2f-f7de-4bf6-8e9b-7ade0fcfad1f
+-------------------+----------------------------------------------------+
| Field             | Value                                              |
+-------------------+----------------------------------------------------+
| allocation_pools  | {"start": "192.168.46.2", "end": "192.168.46.254"} |
| cidr              | 192.168.46.0/                                    |
| dns_nameservers   |                                                    |
| enable_dhcp       | True                                               |
| gateway_ip        | 192.168.46.1                                       |
| host_routes       |                                                    |
| id                | fab94e2f-f7de-4bf6-8e9b-7ade0fcfad1f               |
| ip_version        |                                                   |
| ipv6_address_mode |                                                    |
| ipv6_ra_mode      |                                                    |
| name              | vlan_100_subnet01                                  |
| network_id        | 2426ff0d-953f-467f-a564-c4f63d926836               |
| subnetpool_id     |                                                    |
| tenant_id         | 8cb8c084ffb84914b41d5044ecbcad4e                   |
+-------------------+----------------------------------------------------+

接下来，创建Floating IP所在的public network, 网络类型是flat

注意下面：192.168.1.100-192.168.1.119是网络管理员分配给我的IP pool，192.168.1.30是我们网络的router ip

# Step 1: create external flat network
$ neutron net-create --router:external  --provider:network_type flat  --provider:physical_network public_eth0  public_net
# Step : create subnet based on your reservation Here
$ neutron subnet-create  --name public_eth0_subnet01 --allocation-pool start=192.168.1.100,end=192.168.1.119 --gateway​ 192.168.1.30 --disable-dhcp --ip-version   22d01ce7-b4c4-4af2-bc3c-9c3991903b4c 192.168.1.0/
# Step : create a external router
$ neutron router-create  router1
# Step 4: connect router1 with the private subnet via 'neutron router-interface-add <router-id> <private_subnet_id>'
$ neutron router-interface-add 2da62299-6e41-4b24-a1c7-a3d6cc4db1c8 fab94e2f-f7de-4bf6-8e9b-7ade0fcfad1f
# Step 5: Set the external public network  as the router gateway 'neutron router-gateway-set <router-id> <public_subnet_id>'
$ neutron router-gateway-set   73a38db4-20f9-49ba-b855-472be1e2fd45  e0368da6-197e-4c46-bfd1-4897d61b519a
# Step 6: Create floating IP from public network 'neutron floatingip-create  <public network id>'
$ neutron floatingip-create  1452712a-acb3--a5ca-1d838eb2feb9
# Step 7: assicate the VM instance's port in private network to the newly created floating ip port
## get the port id of floating IP (in bold)
$ stack@openstack-wangp11-:~$ neutron floatingip-list
+--------------------------------------+------------------+---------------------+--------------------------------------+
| id                                   | fixed_ip_address | floating_ip_address | port_id                              |
+--------------------------------------+------------------+---------------------+--------------------------------------+
| a2316e0b-6d72-420f-8a23-f8421160d3d4 |                | 192.168.1.102       | 75353e16-5a16-452e-b420-0ada719d625c |
+--------------------------------------+------------------+---------------------+--------------------------------------+
## get the port id of private interface attached to VM instance
$ stack@openstack-wangp11-:~$ neutron port-list
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                            |
+--------------------------------------+------+-------------------+--------------------------------------------------------------------------------------+
| 16a3e39c-48ef-4d4d-bc8a-96b27da7ff3c |      | fa::3e:c1:: | {"subnet_id": "e0368da6-197e-4c46-bfd1-4897d61b519a", "ip_address": "192.168.1.100"} |
| 616d03cd-df6a-465f-a23a-b6bf46e55f7a |      | fa::3e::2e:7f | {"subnet_id": "fab94e2f-f7de-4bf6-8e9b-7ade0fcfad1f", "ip_address": "192.168.46.4"}  |
| 713e4541-7d22--b828-94d714d94ec0 |      | fa::3e:af::7b | {"subnet_id": "fab94e2f-f7de-4bf6-8e9b-7ade0fcfad1f", "ip_address": "192.168.46.2"}  |
| 75353e16-5a16-452e-b420-0ada719d625c |      | fa::3e:::db | {"subnet_id": "fab94e2f-f7de-4bf6-8e9b-7ade0fcfad1f", "ip_address": "192.168.46.3"}

# 上面红色的port是VM在private network的interface
$ stack@openstack-wangp11-:~$ neutron floatingip-associate  a2316e0b-6d72-420f-8a23-f8421160d3d4 75353e16-5a16-452e-b420-0ada719d625c
# Now you can use '192.168.1.102' to login to the VM instance from 192.168..XX network

我在最前面提到，这个Floating IP是通过namespace内的iptables规则实现的，如何查看呢？下面是在neutron node上的一个实例，192.168.1.102是Floating IP，192.168.46.3是private IP

ip netns exec qrouter-<private-network-id> iptables -t nat -S

-A quantum-l3-agent-OUTPUT -d 192.168.1.102/ -j DNAT --to-destination 192.168.46.3
-A quantum-l3-agent-PREROUTING -d 192.168.1.102/ -j DNAT --to-destination 192.168.46.3
-A quantum-l3-agent-float-snat -s 192.168.46.3/ -j SNAT --to-source 192.168.1.102

• 启用nameserver

现在，VM跟外界可以通过192.168.1.102进行通信了，在外界看来，VM的Ip就是192.168.1.102了，但是有一个问题，无法访问网站，也无法sudo apt-get update

必须要启用nameserver才可以访问外面的网址

# add following to /etc/resolv.conf of the VM instance to enable name service

nameserver <your name sever ip>
nameserver <your name server ip >

#and run following to enable
sudo resolvconf -u

• 问题诊断

TODO

• 参考文章

https://www.mirantis.com/blog/configuring-floating-ip-addresses-networking-openstack-public-private-clouds/

https://www.rdoproject.org/Networking_in_too_much_detail

https://blogs.oracle.com/ronen/entry/running_openstack_icehouse_with_zfs