《Terraform 101 从入门到实践》 Terraform在公有云Azure上的应用

《Terraform 101 从入门到实践》这本小册在南瓜慢说官方网站和GitHub两个地方同步更新，书中的示例代码也是放在GitHub上，方便大家参考查看。

---

简介

Azure是微软的公有云，它提供了一些免费的资源，具体可以查看： https://azure.microsoft.com/en-us/free/

本章将介绍如何通过Terraform来使用Azure的云资源。

注册Azure账号

首先要注册一个Azure账号，我选择用GitHub账号登陆，免得又记多一个密码。

跳到GitHub，同意即可：

创建账号时，有一些信息要填，特别是邮箱和手机号比较关键：

同时还需要一张Visa或Master卡，我是有一张Visa的卡，填好后会有一个0元的扣费，不要担心。下面Cardholder Name我填的中文名字，注册成功了。

0元扣费成功后，表示卡是正常的，就可以成功注册了，注册后就可以到Portal查看了。

手动部署虚拟机

为了体验一下Azure，我们先手动创建一个虚拟机，操作入口如下：

需要填写一些配置信息，如主机名、区域、镜像、网络端口等，按需要我打开了22/80/443端口。

完成配置后，点击创建，提示要下载密钥对，必须要在创建的时候下载：

创建完资源后，可以在虚拟机列表查看：

根据用户名和公网IP，我们可以ssh连接到服务器。需要给密钥文件修改权限，太大是不行的，会报错。

$ chmod 400 ~/Downloads/pksow-azure.pem

然后通过下面命令连接：

$ ssh azureuser@20.2.85.137 -i ~/Downloads/pksow-azure.pem
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-1030-azure x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System load:  0.01513671875     Processes:             109
  Usage of /:   4.9% of 28.89GB   Users logged in:       0
  Memory usage: 31%               IPv4 address for eth0: 10.0.0.4
  Swap usage:   0%

0 updates can be applied immediately.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

azureuser@pkslow:~$ free
               total        used        free      shared  buff/cache   available
Mem:          928460      261816      288932        4140      377712      533872
Swap:              0           0           0
azureuser@pkslow:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        29G  1.5G   28G   5% /
tmpfs           454M     0  454M   0% /dev/shm
tmpfs           182M  1.1M  181M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda15      105M  5.3M  100M   5% /boot/efi
/dev/sdb1       3.9G   28K  3.7G   1% /mnt
tmpfs            91M  4.0K   91M   1% /run/user/1000

通过azure-cli创建虚拟机

安装azure-cli

我的电脑是MacOS，安装如下：

$ brew update-reset

$ brew install azure-cli

$ which az
/usr/local/bin/az

$ az version
{
  "azure-cli": "2.44.1",
  "azure-cli-core": "2.44.1",
  "azure-cli-telemetry": "1.0.8",
  "extensions": {}
}

其它系统请参考： https://learn.microsoft.com/en-us/cli/azure/install-azure-cli

权限

通过命令行操作Azure的资源，必然是需要权限的，我们可以通过密码，还可以通过Service Principal等方式来登陆。我们主要使用Service Principal的方式来授权。因此我们先在Portal上创建。

在左侧菜单选择Azure Active Directory，选择应用注册，点击新注册：

注册应用程序：

添加密码：

设置说明和时长：

创建完后要马上记下密码，后面无法再获取密码值：

查看租户

需要查看租户ID，或创建租户：

分配角色

到订阅管理界面： Subscriptions page in Azure portal，查看订阅列表：

点进去后，可以管理访问控制：

把之前创建的Service Principal加进来，分配特定角色：

选择对应的Service Principal：

命令行登陆

完成以上操作后，就可以通过命令行来登陆Azure了：

$ az login --service-principal -u f01d69bf-8ff3-4043-9275-3e0c4de54884 -p B0N8Q~PQu6hTJkBTS5xxxxxxxx******** --tenant 2951528a-e359-4846-9817-ec3ebc2664d4
[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "2951528a-e359-4846-9817-ec3ebc2664d4",
    "id": "cd7921d5-9ba9-45db-bfba-1c397fcaaba3",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Free Trial",
    "state": "Enabled",
    "tenantId": "2951528a-e359-4846-9817-ec3ebc2664d4",
    "user": {
      "name": "f01d69bf-8ff3-4043-9275-3e0c4de54884",
      "type": "servicePrincipal"
    }
  }
]

-u是注册应用的ID；

-p就是之前要记下的密码；

--tenant就是租户ID；

查询之前创建的VM，成功：

$ az vm list -g test --output table
Name    ResourceGroup    Location    Zones
------  ---------------  ----------  -------
pkslow  test             eastasia    1

创建vm

通过命令行创建vm如下：

$ az vm create --resource-group 'test' --name 'pkslow2' --image 'canonical:0001-com-ubuntu-server-jammy:22_04-lts:22.04.202301100' --admin-username 'larry' --admin-password 'Pa!!!ss123' --location 'eastasia'

{
  "fqdns": "",
  "id": "/subscriptions/cd7921d5-9ba9-45db-bfba-1c397fcaaba3/resourceGroups/test/providers/Microsoft.Compute/virtualMachines/pkslow2",
  "location": "eastasia",
  "macAddress": "60-45-BD-57-30-C1",
  "powerState": "VM running",
  "privateIpAddress": "10.0.0.5",
  "publicIpAddress": "20.187.85.53",
  "resourceGroup": "test",
  "zones": ""
}

查询后成功创建，已经有2台虚拟机在运行：

$ az vm list -g test --output table
Name     ResourceGroup    Location    Zones
-------  ---------------  ----------  -------
pkslow   test             eastasia    1
pkslow2  test             eastasia

用Terraform创建vm

权限环境变量设置

当我们使用Terraform来操作Azure时，同样也是需要权限的，配置以下环境变量即可。这些值在前面的内容已经讲过了。

export ARM_SUBSCRIPTION_ID="<azure_subscription_id>"
export ARM_TENANT_ID="<azure_subscription_tenant_id>"
export ARM_CLIENT_ID="<service_principal_appid>"
export ARM_CLIENT_SECRET="<service_principal_password>"

插件和版本

配置Terraform和插件的版本：

terraform {
  required_version = ">= 1.1.3"
  required_providers {

    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.38.0"
    }
  }
}

创建vm

通过azurerm_virtual_machine来创建VM资源：

provider "azurerm" {
  features {}
}

variable "prefix" {
  default = "pkslow-azure"
}

resource "azurerm_resource_group" "example" {
  name     = "${var.prefix}-resources"
  location = "West Europe"
}

resource "azurerm_virtual_network" "main" {
  name                = "${var.prefix}-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_subnet" "internal" {
  name                 = "internal"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefixes     = ["10.0.2.0/24"]
}

resource "azurerm_network_interface" "main" {
  name                = "${var.prefix}-nic"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  ip_configuration {
    name                          = "testconfiguration1"
    subnet_id                     = azurerm_subnet.internal.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_virtual_machine" "main" {
  name                  = "${var.prefix}-vm"
  location              = azurerm_resource_group.example.location
  resource_group_name   = azurerm_resource_group.example.name
  network_interface_ids = [azurerm_network_interface.main.id]
  vm_size               = "Standard_DS1_v2"

  # Uncomment this line to delete the OS disk automatically when deleting the VM
  # delete_os_disk_on_termination = true

  # Uncomment this line to delete the data disks automatically when deleting the VM
  # delete_data_disks_on_termination = true

  storage_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts"
    version   = "22.04.202301100"
  }
  storage_os_disk {
    name              = "myosdisk1"
    caching           = "ReadWrite"
    create_option     = "FromImage"
    managed_disk_type = "Standard_LRS"
  }
  os_profile {
    computer_name  = "hostname"
    admin_username = "larry"
    admin_password = "Password1234!"
  }
  os_profile_linux_config {
    disable_password_authentication = false
  }
  tags = {
    environment = "staging"
  }
}

然后我们执行初始化，会下载Azure的Terraform插件：

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/azurerm versions matching "3.38.0"...
- Installing hashicorp/azurerm v3.38.0...
- Installed hashicorp/azurerm v3.38.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

查看plan，看看会生成什么资源：

$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_network_interface.main will be created
  + resource "azurerm_network_interface" "main" {
      + applied_dns_servers           = (known after apply)
      + dns_servers                   = (known after apply)
      + enable_accelerated_networking = false
      + enable_ip_forwarding          = false
      + id                            = (known after apply)
      + internal_dns_name_label       = (known after apply)
      + internal_domain_name_suffix   = (known after apply)
      + location                      = "westeurope"
      + mac_address                   = (known after apply)
      + name                          = "pkslow-azure-nic"
      + private_ip_address            = (known after apply)
      + private_ip_addresses          = (known after apply)
      + resource_group_name           = "pkslow-azure-resources"
      + virtual_machine_id            = (known after apply)

      + ip_configuration {
          + gateway_load_balancer_frontend_ip_configuration_id = (known after apply)
          + name                                               = "testconfiguration1"
          + primary                                            = (known after apply)
          + private_ip_address                                 = (known after apply)
          + private_ip_address_allocation                      = "Dynamic"
          + private_ip_address_version                         = "IPv4"
          + subnet_id                                          = (known after apply)
        }
    }

  # azurerm_resource_group.example will be created
  + resource "azurerm_resource_group" "example" {
      + id       = (known after apply)
      + location = "westeurope"
      + name     = "pkslow-azure-resources"
    }

  # azurerm_subnet.internal will be created
  + resource "azurerm_subnet" "internal" {
      + address_prefixes                               = [
          + "10.0.2.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = (known after apply)
      + enforce_private_link_service_network_policies  = (known after apply)
      + id                                             = (known after apply)
      + name                                           = "internal"
      + private_endpoint_network_policies_enabled      = (known after apply)
      + private_link_service_network_policies_enabled  = (known after apply)
      + resource_group_name                            = "pkslow-azure-resources"
      + virtual_network_name                           = "pkslow-azure-network"
    }

  # azurerm_virtual_machine.main will be created
  + resource "azurerm_virtual_machine" "main" {
      + availability_set_id              = (known after apply)
      + delete_data_disks_on_termination = false
      + delete_os_disk_on_termination    = false
      + id                               = (known after apply)
      + license_type                     = (known after apply)
      + location                         = "westeurope"
      + name                             = "pkslow-azure-vm"
      + network_interface_ids            = (known after apply)
      + resource_group_name              = "pkslow-azure-resources"
      + tags                             = {
          + "environment" = "staging"
        }
      + vm_size                          = "Standard_DS1_v2"

      + identity {
          + identity_ids = (known after apply)
          + principal_id = (known after apply)
          + type         = (known after apply)
        }

      + os_profile {
          + admin_password = (sensitive value)
          + admin_username = "larry"
          + computer_name  = "hostname"
          + custom_data    = (known after apply)
        }

      + os_profile_linux_config {
          + disable_password_authentication = false
        }

      + storage_data_disk {
          + caching                   = (known after apply)
          + create_option             = (known after apply)
          + disk_size_gb              = (known after apply)
          + lun                       = (known after apply)
          + managed_disk_id           = (known after apply)
          + managed_disk_type         = (known after apply)
          + name                      = (known after apply)
          + vhd_uri                   = (known after apply)
          + write_accelerator_enabled = (known after apply)
        }

      + storage_image_reference {
          + offer     = "0001-com-ubuntu-server-jammy"
          + publisher = "Canonical"
          + sku       = "22_04-lts"
          + version   = "22.04.202301100"
        }

      + storage_os_disk {
          + caching                   = "ReadWrite"
          + create_option             = "FromImage"
          + disk_size_gb              = (known after apply)
          + managed_disk_id           = (known after apply)
          + managed_disk_type         = "Standard_LRS"
          + name                      = "myosdisk1"
          + os_type                   = (known after apply)
          + write_accelerator_enabled = false
        }
    }

  # azurerm_virtual_network.main will be created
  + resource "azurerm_virtual_network" "main" {
      + address_space       = [
          + "10.0.0.0/16",
        ]
      + dns_servers         = (known after apply)
      + guid                = (known after apply)
      + id                  = (known after apply)
      + location            = "westeurope"
      + name                = "pkslow-azure-network"
      + resource_group_name = "pkslow-azure-resources"
      + subnet              = (known after apply)
    }

Plan: 5 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

直接apply，创建对应的资源：

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_network_interface.main will be created
  + resource "azurerm_network_interface" "main" {
      + applied_dns_servers           = (known after apply)
      + dns_servers                   = (known after apply)
      + enable_accelerated_networking = false
      + enable_ip_forwarding          = false
      + id                            = (known after apply)
      + internal_dns_name_label       = (known after apply)
      + internal_domain_name_suffix   = (known after apply)
      + location                      = "westeurope"
      + mac_address                   = (known after apply)
      + name                          = "pkslow-azure-nic"
      + private_ip_address            = (known after apply)
      + private_ip_addresses          = (known after apply)
      + resource_group_name           = "pkslow-azure-resources"
      + virtual_machine_id            = (known after apply)

      + ip_configuration {
          + gateway_load_balancer_frontend_ip_configuration_id = (known after apply)
          + name                                               = "testconfiguration1"
          + primary                                            = (known after apply)
          + private_ip_address                                 = (known after apply)
          + private_ip_address_allocation                      = "Dynamic"
          + private_ip_address_version                         = "IPv4"
          + subnet_id                                          = (known after apply)
        }
    }

  # azurerm_resource_group.example will be created
  + resource "azurerm_resource_group" "example" {
      + id       = (known after apply)
      + location = "westeurope"
      + name     = "pkslow-azure-resources"
    }

  # azurerm_subnet.internal will be created
  + resource "azurerm_subnet" "internal" {
      + address_prefixes                               = [
          + "10.0.2.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = (known after apply)
      + enforce_private_link_service_network_policies  = (known after apply)
      + id                                             = (known after apply)
      + name                                           = "internal"
      + private_endpoint_network_policies_enabled      = (known after apply)
      + private_link_service_network_policies_enabled  = (known after apply)
      + resource_group_name                            = "pkslow-azure-resources"
      + virtual_network_name                           = "pkslow-azure-network"
    }

  # azurerm_virtual_machine.main will be created
  + resource "azurerm_virtual_machine" "main" {
      + availability_set_id              = (known after apply)
      + delete_data_disks_on_termination = false
      + delete_os_disk_on_termination    = false
      + id                               = (known after apply)
      + license_type                     = (known after apply)
      + location                         = "westeurope"
      + name                             = "pkslow-azure-vm"
      + network_interface_ids            = (known after apply)
      + resource_group_name              = "pkslow-azure-resources"
      + tags                             = {
          + "environment" = "staging"
        }
      + vm_size                          = "Standard_DS1_v2"

      + identity {
          + identity_ids = (known after apply)
          + principal_id = (known after apply)
          + type         = (known after apply)
        }

      + os_profile {
          + admin_password = (sensitive value)
          + admin_username = "larry"
          + computer_name  = "hostname"
          + custom_data    = (known after apply)
        }

      + os_profile_linux_config {
          + disable_password_authentication = false
        }

      + storage_data_disk {
          + caching                   = (known after apply)
          + create_option             = (known after apply)
          + disk_size_gb              = (known after apply)
          + lun                       = (known after apply)
          + managed_disk_id           = (known after apply)
          + managed_disk_type         = (known after apply)
          + name                      = (known after apply)
          + vhd_uri                   = (known after apply)
          + write_accelerator_enabled = (known after apply)
        }

      + storage_image_reference {
          + offer     = "0001-com-ubuntu-server-jammy"
          + publisher = "Canonical"
          + sku       = "22_04-lts"
          + version   = "22.04.202301100"
        }

      + storage_os_disk {
          + caching                   = "ReadWrite"
          + create_option             = "FromImage"
          + disk_size_gb              = (known after apply)
          + managed_disk_id           = (known after apply)
          + managed_disk_type         = "Standard_LRS"
          + name                      = "myosdisk1"
          + os_type                   = (known after apply)
          + write_accelerator_enabled = false
        }
    }

  # azurerm_virtual_network.main will be created
  + resource "azurerm_virtual_network" "main" {
      + address_space       = [
          + "10.0.0.0/16",
        ]
      + dns_servers         = (known after apply)
      + guid                = (known after apply)
      + id                  = (known after apply)
      + location            = "westeurope"
      + name                = "pkslow-azure-network"
      + resource_group_name = "pkslow-azure-resources"
      + subnet              = (known after apply)
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_resource_group.example: Creating...
azurerm_resource_group.example: Creation complete after 9s [id=/subscriptions/cd7921d5-9ba9-45db-bfba-1c397fcaaba3/resourceGroups/pkslow-azure-resources]
azurerm_virtual_network.main: Creating...
azurerm_virtual_network.main: Still creating... [10s elapsed]
azurerm_virtual_network.main: Creation complete after 17s [id=/subscriptions/cd7921d5-9ba9-45db-bfba-1c397fcaaba3/resourceGroups/pkslow-azure-resources/providers/Microsoft.Network/virtualNetworks/pkslow-azure-network]
azurerm_subnet.internal: Creating...
azurerm_subnet.internal: Still creating... [10s elapsed]
azurerm_subnet.internal: Creation complete after 11s [id=/subscriptions/cd7921d5-9ba9-45db-bfba-1c397fcaaba3/resourceGroups/pkslow-azure-resources/providers/Microsoft.Network/virtualNetworks/pkslow-azure-network/subnets/internal]
azurerm_network_interface.main: Creating...
azurerm_network_interface.main: Still creating... [10s elapsed]
azurerm_network_interface.main: Creation complete after 10s [id=/subscriptions/cd7921d5-9ba9-45db-bfba-1c397fcaaba3/resourceGroups/pkslow-azure-resources/providers/Microsoft.Network/networkInterfaces/pkslow-azure-nic]
azurerm_virtual_machine.main: Creating...
azurerm_virtual_machine.main: Still creating... [10s elapsed]
azurerm_virtual_machine.main: Still creating... [20s elapsed]
azurerm_virtual_machine.main: Still creating... [30s elapsed]
azurerm_virtual_machine.main: Still creating... [40s elapsed]
azurerm_virtual_machine.main: Still creating... [50s elapsed]
azurerm_virtual_machine.main: Still creating... [1m0s elapsed]
azurerm_virtual_machine.main: Creation complete after 1m0s [id=/subscriptions/cd7921d5-9ba9-45db-bfba-1c397fcaaba3/resourceGroups/pkslow-azure-resources/providers/Microsoft.Compute/virtualMachines/pkslow-azure-vm]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

查看所有资源，选择资源组pkslow-azure-resources下面的，已经成功创建：

使用完成后，通过下面命令删除：

terraform destroy

部署Azure Kubernetes集群

通过Auzre CLI部署

创建资源组

Azure资源组是用于部署和管理Azure资源的逻辑组。创建资源时，系统会提示你指定一个位置。该位置主要用于：

（1）资源组元数据的存储位置；

（2）在创建资源期间未指定另一个区域时，资源在Azure中的运行位置。

我们通过以下命令来创建资源组：

$ az group create --name pkslow-aks --location eastasia
{
  "id": "/subscriptions/cd7921d5-9ba9-45db-bfba-1c397fcaaba3/resourceGroups/pkslow-aks",
  "location": "eastasia",
  "managedBy": null,
  "name": "pkslow-aks",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

创建AKS

通过下面的命令创建AKS：

az aks create -g pkslow-aks -n pkslow --enable-managed-identity --node-count 1 --enable-addons monitoring --enable-msi-auth-for-monitoring  --generate-ssh-keys

创建完成后会输出很大的Json日志，我们直接来查看一下是否正确生成：

$ az aks list --output table
Name    Location    ResourceGroup    KubernetesVersion    CurrentKubernetesVersion    ProvisioningState    Fqdn
------  ----------  ---------------  -------------------  --------------------------  -------------------  --------------------------------------------------------
pkslow  eastasia    pkslow-aks       1.24.6               1.24.6                      Succeeded            pkslow-pkslow-aks-cd7921-725c7247.hcp.eastasia.azmk8s.io

连接到AKS

需要有kubectl命令，没有的就安装一下：

az aks install-cli

连接集群需要认证，要获取一下验证配置：

$ az aks get-credentials --resource-group pkslow-aks --name pkslow
Merged "pkslow" as current context in /Users/larry/.kube/config

成功后就可以连接并操作了：

$ kubectl get node
NAME                                STATUS   ROLES   AGE     VERSION
aks-nodepool1-29201873-vmss000000   Ready    agent   8m45s   v1.24.6

$ kubectl get ns
NAME              STATUS   AGE
default           Active   9m33s
kube-node-lease   Active   9m35s
kube-public       Active   9m35s
kube-system       Active   9m35s

$ kubectl get pod -n kube-system
NAME                                  READY   STATUS    RESTARTS   AGE
ama-logs-lhlkb                        3/3     Running   0          9m8s
ama-logs-rs-6cf9546595-rdmh9          2/2     Running   0          9m26s
azure-ip-masq-agent-nppvd             1/1     Running   0          9m8s
cloud-node-manager-bd4c2              1/1     Running   0          9m8s
coredns-59b6bf8b4f-lrzpp              1/1     Running   0          9m26s
coredns-59b6bf8b4f-zbbkm              1/1     Running   0          7m56s
coredns-autoscaler-5655d66f64-5946c   1/1     Running   0          9m26s
csi-azuredisk-node-9rpvd              3/3     Running   0          9m8s
csi-azurefile-node-hvxhc              3/3     Running   0          9m8s
konnectivity-agent-95ff8bbd-fwkds     1/1     Running   0          9m26s
konnectivity-agent-95ff8bbd-qg9vx     1/1     Running   0          9m26s
kube-proxy-c5crz                      1/1     Running   0          9m8s
metrics-server-7dd74d8758-ms8h9       2/2     Running   0          7m50s
metrics-server-7dd74d8758-nxq9t       2/2     Running   0          7m50s

部署测试应用

为了方便，我们直接使用官网的示例来测试一下。创建文件azure-vote.yaml，内容如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-back
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-back
  template:
    metadata:
      labels:
        app: azure-vote-back
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: azure-vote-back
          image: mcr.microsoft.com/oss/bitnami/redis:6.0.8
          env:
            - name: ALLOW_EMPTY_PASSWORD
              value: "yes"
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 250m
              memory: 256Mi
          ports:
            - containerPort: 6379
              name: redis
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-back
spec:
  ports:
    - port: 6379
  selector:
    app: azure-vote-back
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-vote-front
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-vote-front
  template:
    metadata:
      labels:
        app: azure-vote-front
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: azure-vote-front
          image: mcr.microsoft.com/azuredocs/azure-vote-front:v1
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 250m
              memory: 256Mi
          ports:
            - containerPort: 80
          env:
            - name: REDIS
              value: "azure-vote-back"
---
apiVersion: v1
kind: Service
metadata:
  name: azure-vote-front
spec:
  type: LoadBalancer
  ports:
    - port: 80
  selector:
    app: azure-vote-front

然后执行以下命令：

$ kubectl apply -f azure-vote.yaml
deployment.apps/azure-vote-back created
service/azure-vote-back created
deployment.apps/azure-vote-front created
service/azure-vote-front created

成功后查看对应资源：

$ kubectl get svc
NAME               TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
azure-vote-back    ClusterIP      10.0.156.161   <none>         6379/TCP       112s
azure-vote-front   LoadBalancer   10.0.29.217    20.239.124.1   80:30289/TCP   112s
kubernetes         ClusterIP      10.0.0.1       <none>         443/TCP        21m

$ kubectl get deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
azure-vote-back    1/1     1            1           2m1s
azure-vote-front   1/1     1            1           2m1s

$ kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
azure-vote-back-7cd69cc96f-gqm7r    1/1     Running   0          2m7s
azure-vote-front-7c95676c68-jtkqz   1/1     Running   0          2m7s

已经成功创建。

看front那有external IP，通过它直接在浏览器访问如下：

应用已经成功部署并访问了。

删除资源组

如果完成测试，不再使用，可以整个资源组一起删除：

az group delete --name pkslow-aks --yes --no-wait

通过Terraform部署

配置插件和版本

terraform {
  required_version = ">= 1.1.3"
  required_providers {

    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.38.0"
    }

    random = {
      source  = "hashicorp/random"
      version = "= 3.1.0"
    }
  }
}

变量设置

给Terraform设置一些要用到的变量：

variable "agent_count" {
  default = 1
}

# The following two variable declarations are placeholder references.
# Set the values for these variable in terraform.tfvars
variable "aks_service_principal_app_id" {
  default = ""
}

variable "aks_service_principal_client_secret" {
  default = ""
}

variable "cluster_name" {
  default = "pkslow-k8s"
}

variable "dns_prefix" {
  default = "pkslow"
}

# Refer to https://azure.microsoft.com/global-infrastructure/services/?products=monitor for available Log Analytics regions.
variable "log_analytics_workspace_location" {
  default = "eastus"
}

variable "log_analytics_workspace_name" {
  default = "testLogAnalyticsWorkspaceName"
}

# Refer to https://azure.microsoft.com/pricing/details/monitor/ for Log Analytics pricing
variable "log_analytics_workspace_sku" {
  default = "PerGB2018"
}

variable "resource_group_location" {
  default     = "eastus"
  description = "Location of the resource group."
}

variable "resource_group_name_prefix" {
  default     = "rg"
  description = "Prefix of the resource group name that's combined with a random ID so name is unique in your Azure subscription."
}

variable "ssh_public_key" {
  default = "~/.ssh/id_rsa.pub"
}

agent_count应该设置合理，这里设成1是因为我的账号是免费的，有限制。

输出结果

当Terraform执行完，会有一些结果，我们可以把一些值输出以便使用：

output "client_certificate" {
  value     = azurerm_kubernetes_cluster.k8s.kube_config[0].client_certificate
  sensitive = true
}

output "client_key" {
  value     = azurerm_kubernetes_cluster.k8s.kube_config[0].client_key
  sensitive = true
}

output "cluster_ca_certificate" {
  value     = azurerm_kubernetes_cluster.k8s.kube_config[0].cluster_ca_certificate
  sensitive = true
}

output "cluster_password" {
  value     = azurerm_kubernetes_cluster.k8s.kube_config[0].password
  sensitive = true
}

output "cluster_username" {
  value     = azurerm_kubernetes_cluster.k8s.kube_config[0].username
  sensitive = true
}

output "host" {
  value     = azurerm_kubernetes_cluster.k8s.kube_config[0].host
  sensitive = true
}

output "kube_config" {
  value     = azurerm_kubernetes_cluster.k8s.kube_config_raw
  sensitive = true
}

output "resource_group_name" {
  value = azurerm_resource_group.rg.name
}

main.tf创建AKS

通过azurerm_kubernetes_cluster创建AKS：

provider "azurerm" {
  features {}
}

# Generate random resource group name
resource "random_pet" "rg_name" {
  prefix = var.resource_group_name_prefix
}

resource "azurerm_resource_group" "rg" {
  location = var.resource_group_location
  name     = random_pet.rg_name.id
}

resource "random_id" "log_analytics_workspace_name_suffix" {
  byte_length = 8
}

resource "azurerm_log_analytics_workspace" "test" {
  location            = var.log_analytics_workspace_location
  # The WorkSpace name has to be unique across the whole of azure;
  # not just the current subscription/tenant.
  name                = "${var.log_analytics_workspace_name}-${random_id.log_analytics_workspace_name_suffix.dec}"
  resource_group_name = azurerm_resource_group.rg.name
  sku                 = var.log_analytics_workspace_sku
}

resource "azurerm_log_analytics_solution" "test" {
  location              = azurerm_log_analytics_workspace.test.location
  resource_group_name   = azurerm_resource_group.rg.name
  solution_name         = "ContainerInsights"
  workspace_name        = azurerm_log_analytics_workspace.test.name
  workspace_resource_id = azurerm_log_analytics_workspace.test.id

  plan {
    product   = "OMSGallery/ContainerInsights"
    publisher = "Microsoft"
  }
}

resource "azurerm_kubernetes_cluster" "k8s" {
  location            = azurerm_resource_group.rg.location
  name                = var.cluster_name
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = var.dns_prefix
  tags                = {
    Environment = "Development"
  }

  default_node_pool {
    name       = "agentpool"
    vm_size    = "Standard_D2_v2"
    node_count = var.agent_count
  }
  linux_profile {
    admin_username = "ubuntu"

    ssh_key {
      key_data = file(var.ssh_public_key)
    }
  }
  network_profile {
    network_plugin    = "kubenet"
    load_balancer_sku = "standard"
  }
  service_principal {
    client_id     = var.aks_service_principal_app_id
    client_secret = var.aks_service_principal_client_secret
  }
}

执行

准备好文件后，先初始化，下载插件：

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/random versions matching "3.1.0"...
- Finding hashicorp/azurerm versions matching "3.38.0"...
- Installing hashicorp/random v3.1.0...
- Installed hashicorp/random v3.1.0 (unauthenticated)
- Installing hashicorp/azurerm v3.38.0...
- Installed hashicorp/azurerm v3.38.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

查看Terraform计划，知道将要生成多少资源：

$ terraform plan -out main.tfplan -var="aks_service_principal_app_id=$ARM_CLIENT_ID" -var="aks_service_principal_client_secret=$ARM_CLIENT_SECRET"

没有问题则执行变更：

$ terraform apply main.tfplan
Outputs:

client_certificate = <sensitive>
client_key = <sensitive>
cluster_ca_certificate = <sensitive>
cluster_password = <sensitive>
cluster_username = <sensitive>
host = <sensitive>
kube_config = <sensitive>
resource_group_name = "rg-harmless-tomcat"

连接AKS

把kube_config输出，然后设置环境变量就可以通过kubectl连接了：

$ echo "$(terraform output kube_config)" > ./azurek8s

$ export KUBECONFIG=./azurek8s

$ kubectl get nodes
NAME                                STATUS   ROLES   AGE     VERSION
aks-agentpool-45159290-vmss000000   Ready    agent   9m20s   v1.24.6

如果有问题，可以查看azurek8s文件是否正常。

创建PostgreSQL

通过Azure CLI创建Single Server

创建资源组和数据库

先创建资源组：

az group create --name pkslow-sql --location eastasia --tag create-postgresql-server-and-firewall-rule

然后创建数据库：

$ az postgres server create \
> --name pkslow-pg \
> --resource-group pkslow-sql \
> --location eastasia \
> --admin-user pguser \
> --admin-password 'Pa$$word' \
> --sku-name GP_Gen5_2

Checking the existence of the resource group 'pkslow-sql'...
Resource group 'pkslow-sql' exists ? : True
Creating postgres Server 'pkslow-pg' in group 'pkslow-sql'...
Your server 'pkslow-pg' is using sku 'GP_Gen5_2' (Paid Tier). Please refer to https://aka.ms/postgres-pricing  for pricing details
Make a note of your password. If you forget, you would have to reset your password with 'az postgres server update -n pkslow-pg -g pkslow-sql -p <new-password>'.
{
  "additionalProperties": {},
  "administratorLogin": "pguser",
  "byokEnforcement": "Disabled",
  "connectionString": "postgres://pguser%40pkslow-pg:Pa$$word@pkslow-pg.postgres.database.azure.com/postgres?sslmode=require",
  "earliestRestoreDate": "2023-01-15T03:24:18.440000+00:00",
  "fullyQualifiedDomainName": "pkslow-pg.postgres.database.azure.com",
  "id": "/subscriptions/cd7921d5-9ba9-45db-bfba-1c397fcaaba3/resourceGroups/pkslow-sql/providers/Microsoft.DBforPostgreSQL/servers/pkslow-pg",
  "identity": null,
  "infrastructureEncryption": "Disabled",
  "location": "eastasia",
  "masterServerId": "",
  "minimalTlsVersion": "TLSEnforcementDisabled",
  "name": "pkslow-pg",
  "password": "Pa$$word",
  "privateEndpointConnections": [],
  "publicNetworkAccess": "Enabled",
  "replicaCapacity": 5,
  "replicationRole": "None",
  "resourceGroup": "pkslow-sql",
  "sku": {
    "additionalProperties": {},
    "capacity": 2,
    "family": "Gen5",
    "name": "GP_Gen5_2",
    "size": null,
    "tier": "GeneralPurpose"
  },
  "sslEnforcement": "Enabled",
  "storageProfile": {
    "additionalProperties": {},
    "backupRetentionDays": 7,
    "geoRedundantBackup": "Disabled",
    "storageAutogrow": "Enabled",
    "storageMb": 5120
  },
  "tags": null,
  "type": "Microsoft.DBforPostgreSQL/servers",
  "userVisibleState": "Ready",
  "version": "11"
}

创建成功后，会打印很多有用的信息，如连接信息。

也可以在以后查看：

az postgres server show --resource-group pkslow-sql --name pkslow-pg

禁用SSL

创建完成后还可以更新一些配置，如我们禁用SSL：

az postgres server update --resource-group pkslow-sql --name pkslow-pg --ssl-enforcement Disabled

生产环境不要禁用SSL。

添加防火墙

需要把客户端IP添加到Firewall，不然会连接失败。

az postgres server firewall-rule create \
--resource-group pkslow-sql \
--server pkslow-pg \
--name AllowIps \
--start-ip-address '0.0.0.0' \
--end-ip-address '255.255.255.255'

测试连接

配置连接如下，注意用户名不只是pguser：

删除资源

如果不需要再使用，就删除资源：

az group delete --name pkslow-sql

通过Terraform创建Flexible Server

插件与版本

terraform {
  required_version = ">= 1.1.3"
  required_providers {

    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.38.0"
    }
  }
}

provider "azurerm" {
  features {}
}

变量设置

variable "name_prefix" {
  default     = "pkslow-pg-fs"
  description = "Prefix of the resource name."
}

variable "location" {
  default     = "eastus"
  description = "Location of the resource."
}

main.tf创建

resource "random_pet" "rg-name" {
  prefix = var.name_prefix
}

resource "azurerm_resource_group" "default" {
  name     = random_pet.rg-name.id
  location = var.location
}

resource "azurerm_virtual_network" "default" {
  name                = "${var.name_prefix}-vnet"
  location            = azurerm_resource_group.default.location
  resource_group_name = azurerm_resource_group.default.name
  address_space       = ["10.0.0.0/16"]
}

resource "azurerm_network_security_group" "default" {
  name                = "${var.name_prefix}-nsg"
  location            = azurerm_resource_group.default.location
  resource_group_name = azurerm_resource_group.default.name

  security_rule {
    name                       = "test123"
    priority                   = 100
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "*"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }
}

resource "azurerm_subnet" "default" {
  name                 = "${var.name_prefix}-subnet"
  virtual_network_name = azurerm_virtual_network.default.name
  resource_group_name  = azurerm_resource_group.default.name
  address_prefixes     = ["10.0.2.0/24"]
  service_endpoints    = ["Microsoft.Storage"]

  delegation {
    name = "fs"

    service_delegation {
      name = "Microsoft.DBforPostgreSQL/flexibleServers"

      actions = [
        "Microsoft.Network/virtualNetworks/subnets/join/action",
      ]
    }
  }
}

resource "azurerm_subnet_network_security_group_association" "default" {
  subnet_id                 = azurerm_subnet.default.id
  network_security_group_id = azurerm_network_security_group.default.id
}

resource "azurerm_private_dns_zone" "default" {
  name                = "${var.name_prefix}-pdz.postgres.database.azure.com"
  resource_group_name = azurerm_resource_group.default.name

  depends_on = [azurerm_subnet_network_security_group_association.default]
}

resource "azurerm_private_dns_zone_virtual_network_link" "default" {
  name                  = "${var.name_prefix}-pdzvnetlink.com"
  private_dns_zone_name = azurerm_private_dns_zone.default.name
  virtual_network_id    = azurerm_virtual_network.default.id
  resource_group_name   = azurerm_resource_group.default.name
}

resource "azurerm_postgresql_flexible_server" "default" {
  name                   = "${var.name_prefix}-server"
  resource_group_name    = azurerm_resource_group.default.name
  location               = azurerm_resource_group.default.location
  version                = "13"
  delegated_subnet_id    = azurerm_subnet.default.id
  private_dns_zone_id    = azurerm_private_dns_zone.default.id
  administrator_login    = "pguser"
  administrator_password = "QAZwsx123"
  zone                   = "1"
  storage_mb             = 32768
  sku_name               = "GP_Standard_D2s_v3"
  backup_retention_days  = 7

  depends_on = [azurerm_private_dns_zone_virtual_network_link.default]
}

准备文件：pg-fs-db.tf

resource "azurerm_postgresql_flexible_server_database" "default" {
  name      = "${var.name_prefix}-db"
  server_id = azurerm_postgresql_flexible_server.default.id
  collation = "en_US.UTF8"
  charset   = "UTF8"
}

输出结果

output "resource_group_name" {
  value = azurerm_resource_group.default.name
}

output "azurerm_postgresql_flexible_server" {
  value = azurerm_postgresql_flexible_server.default.name
}

output "postgresql_flexible_server_database_name" {
  value = azurerm_postgresql_flexible_server_database.default.name
}

执行

准备好hcl文件后，执行如下：

$ terraform init

$ terraform plan -out main.tfplan

$ terraform apply main.tfplan
Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

Outputs:
azurerm_postgresql_flexible_server = "pkslow-pg-fs-server"
postgresql_flexible_server_database_name = "pkslow-pg-fs-db"
resource_group_name = "pkslow-pg-fs-delicate-honeybee"

创建成功后，可以查看：

$ az postgres flexible-server list --output table
Name                 Resource Group                  Location    Version    Storage Size(GiB)    Tier            SKU              State    HA State    Availability zone
-------------------  ------------------------------  ----------  ---------  -------------------  --------------  ---------------  -------  ----------  -------------------
pkslow-pg-fs-server  pkslow-pg-fs-delicate-honeybee  East US     13         32                   GeneralPurpose  Standard_D2s_v3  Ready    NotEnabled  1

当然，在Portal上看也是可以的：

删除

不需要了可以执行删除：

 terraform destroy

在Azure云存储上管理Terraform状态

默认Terraform的状态是保存在本地的，为了安全和协作，在生产环境中一般要保存在云上。

创建Azure Storage

我们创建Storage来存储Terraform状态。按下面一步步执行即可：

RESOURCE_GROUP_NAME=pkslow-tstate-rg
STORAGE_ACCOUNT_NAME=pkslowtfstate
CONTAINER_NAME=tfstate

# Create resource group
az group create --name $RESOURCE_GROUP_NAME --location "West Europe"

# Create storage account
az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob

# Get storage account key
ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query [0].value -o tsv)

# Create blob container
az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME --account-key $ACCOUNT_KEY

echo "storage_account_name: $STORAGE_ACCOUNT_NAME"
echo "container_name: $CONTAINER_NAME"
echo "access_key: $ACCOUNT_KEY"

Terraform backend

创建完Storage后，我们需要在Terraform中配置使用：

terraform {
  required_version = ">= 1.1.3"
  required_providers {

    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.38.0"
    }
    local = {
      source  = "hashicorp/local"
      version = "= 2.1.0"
    }
  }

  backend "azurerm" {
    resource_group_name  = "pkslow-tstate-rg"
    storage_account_name = "pkslowtfstate"
    container_name       = "tfstate"
    key                  = "pkslow.tfstate"
  }
}

provider "azurerm" {
  features {}
}

resource "local_file" "test-file" {
  content  = "https://www.pkslow.com"
  filename = "${path.root}/terraform-guides-by-pkslow.txt"
}

主要代码是这块：

backend "azurerm" {
resource_group_name  = "pkslow-tstate-rg"
storage_account_name = "pkslowtfstate"
container_name       = "tfstate"
key                  = "pkslow.tfstate"
}

这里前三个变量的值都是前面创建Storage的时候指定的。

执行Terraform

初始化：

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/local versions matching "2.1.0"...
- Finding hashicorp/azurerm versions matching "3.38.0"...
- Installing hashicorp/local v2.1.0...
- Installed hashicorp/local v2.1.0 (unauthenticated)
- Installing hashicorp/azurerm v3.38.0...
- Installed hashicorp/azurerm v3.38.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

看日志就会初始化backend。

执行apply：

$ terraform apply -auto-approve
Acquiring state lock. This may take a few moments...

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # local_file.test-file will be created
  + resource "local_file" "test-file" {
      + content              = "https://www.pkslow.com"
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "./terraform-guides-by-pkslow.txt"
      + id                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.
local_file.test-file: Creating...
local_file.test-file: Creation complete after 0s [id=6db7ad1bbf57df0c859cd5fc62ff5408515b5fc1]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

然后我们去查看Azure Storage，就可以发现已经生成一个Terraform状态文件：

如果不再使用，记得删除资源。