neutron 的 quota design

发现，　ｃｉｎｄｅｒ，　ｎｏｖａ　制实现了，　CountableResource。　只有ｎｕｅｔｒｏｎ实现了　TrackedResource　和　　CountableResource。　

I read up the   quota of neutron.

http://docs.openstack.org/developer/neutron/devref/quota.html

Let me list the design:

1. Most resources
exposed by the Neutron API are subject to quota limits.

ｎｉｍｂｌｅ　需要定义　哪些　ｒｅｓｏｕｒｃｅｓ　需要　ｑｕｏｔａ　

２．　Default values
for quota limits are specified in neutron.conf
ｃｏｎｆ　可配置。
目前，　ｉｎｓｔａｎｃｅ，　ｌｉｍｉｔｅ的默认值是多少？　

Limits are stored in the Neutron
database;  默认值存ＤＢ
Admin users can
override those defaults values on a per-project
basis.　Ａｄｍｉｎ用户，可以修改默认值。

Configuration-based
quota management, where every project gets the same quota limit
specified in the configuration file, has been deprecated as of
the Liberty release.

ｎｅｕｔｒｏｎ已经废弃了基于ｃｏｎｆｉｇｕｒａｔｉｏｎ　的ｑｕｏｔａ管理，这个对我们没有影响。　

３.　Quota limits are
enforced at the API layer, before the request is dispatched to
the plugin.

ｎｉｍｂｌｅ　的ｑｕｏｔａ　也确保在ＡＰＩ　ｌａｙｅｒ。

4.　不支持 hierarchical multitenancy

Please note that Neutron does not
support both specification of quota limits per user and quota
management for hierarchical multitenancy (as a matter of fact
Neutron does not support hierarchical multitenancy at all).

这个是什么意思？

Also, quota limits are currently not enforced on RPC
interfaces listening on the AMQP bus.

RPC的调用， quota limit 不起作用？
应该是指 RPC的调用 中用到的资源吧？

5. 通过 Quota engine注册测resources 都会受 quota 管理。

Plugin and ML2 drivers are not supposed to enforce quotas
for resources they manage. However, the subnet_allocation [1] extension is an exception
and will be discussed below.

The quota management and enforcement mechanisms discussed
here apply to every resource which has been registered with the
Quota engine, regardless of whether such resource belongs to the
core Neutron API or one of its extensions.

6． quota组件的组成
有两个部分， API 和quota Engine。 Nimble　的设计 是不是需要
quota Engine？

There are two main components in the Neutron quota system:

• The Quota API extension;
• The Quota Engine.

Both components rely on a quota driver. The neutron
codebase currently defines two quota drivers:

• neutron.db.quota.driver.DbQuotaDriver
• neutron.quota.ConfDriver

The latter driver is however deprecated.

The Quota API
extension handles quota management, whereas the Quota Engine
component handles quota enforcement.

ＡＰＩ　负责ｑｕｏｔａ的管理，　Ｅｎｇｉｎｅ　负责ｑｕｏｔａ的执行。

This API
extension is loaded like any other extension.

For this reason
plugins must explicitly support it by including “quotas” in the
support_extension_aliases attribute.

言外之意，　ｑｕｏｔａｓ不是必须的。

For a
reservation to be successful, the total amount of resources
requested, plus the total amount of resources reserved, plus the
total amount of resources already stored in the database should
not exceed the project’s quota limit.

quota Engine　看来是有必要的。　管理起来有些负责。

ｒｅｑｕｅｓｔ　＋　ｒｅｓｅｒｖｅｄ　+
DB amount <= limit

７．　删除意味着回复默认值

Please note that
the current behaviour when deleting a project quota is to reset
quota limits for that project to configuration defaults.

The API
extension does not validate the project identifier with the
identity service.　不校验　ｐｒｏｊｅｃｔ　ＩＤ

=======================================================================

http://docs.openstack.org/developer/neutron/devref/quota.html#quota-management

1.  quota-management (quota的管理）

The quota management component is fairly straightforward.

However, unlike the vast majority of Neutron extensions,
it uses it own controller class [3]. This class does not implement
the POST operation. List, get, update, and delete
operations are implemented by the usual index, show, update and
delete methods. These method simply call into the quota driver
for either fetching project quotas or updating them.

不支持创建，　所以删除　就是回复默认值。

ｎｉｍｂｌｅ　是否类似？

The _update_attributes method is called only once in the
controller lifetime. This method dynamically updates Neutron’s
resource attribute map [4] so
that an attribute is added for every resource managed by the
quota engine.

Request authorisation is performed in this controller,
and only ‘admin’ users are
allowed to modify quotas for projects.

目前 只有Admin 有授权。

As the neutron policy engine is not used, it is not possible to configure which users should
be allowed to manage quotas using policy.json.

authorization 机制目前不支持其他的用户 管理
quotas。

The driver operations dealing with quota management are:

• delete_tenant_quota, which
  simply removes all entries from the ‘quotas’ table for a
  given project identifier;
• update_quota_limit, which
  adds or updates an entry in the ‘quotas’ project for a
  given project identifier and a given resource name;
• _get_quotas, which fetches
  limits for a set of resource and a given project
  identifier
• _get_all_quotas, which
  behaves like _get_quotas, but for all projects.

http://docs.openstack.org/developer/neutron/devref/quota.html#resource-usage-info
2. quota 中 resource 的使用信息。
Neutron 有2种， resource usage info的跟踪。

Neutron has two ways of tracking resource usage info:

• CountableResource, where
  resource usage is calculated every time quotas limits are
  enforced by counting rows in the resource table and
  reservations for that resource.

就是一种 同步机制？， 每次 在check quotas 有没有超限的时候。

• TrackedResource, which
  instead relies on a specific table
  tracking usage data, and performs explicitly counting only
  when the data in this table are not in sync with actual
  used and reserved resources.

就是一种异步机制？， 数据不一致的时候，
执行一下。　需要在ｄｂ中　额外加一个　ｔａｂｌｅ。

Another difference between CountableResource and
TrackedResource is that the former invokes a plugin method to
count resources. CountableResource should be therefore employed
for plugins which do not leverage the Neutron database. The
actual class that the Neutron quota engine will use is determined by the track_quota_usage variable
in the quota configuration section.
If True, TrackedResource instances will be created, otherwise
the quota engine will use CountableResource instances. Resource
creation is performed by the create_resource_instance factory
method in the neutron.quota.resource module.

这两种
方法可配置　 quota configuration section.

From a performance
perspective, having a table tracking resource usage has some
advantages, albeit not fundamental. Indeed the time required for
executing queries to explicitly count objects will increase with
the number of records in the table. On the other hand, using
TrackedResource will fetch a single record, but has the drawback of having to execute an UPDATE
statement once the operation is completed. Nevertheless,
CountableResource instances do not simply perform a SELECT query
on the relevant table for a resource, but invoke a plugin
method, which might execute several statements and sometimes
even interacts with the backend before returning. Resource usage
tracking also becomes important for operational correctness when
coupled with the concept of resource reservation, discussed in
another section of this chapter.

性能上　TrackedResource　更好。　

缺点　：　
having to execute an UPDATE statement

Tracking quota usage is not as
simple as updating a counter every time resources are
created or deleted. Indeed a quota-limited resource in Neutron
can be created in several ways. While a RESTful API request is
the most common one, resources can be created by RPC handlers
listing on the AMQP bus, such as those which create DHCP ports,
or by plugin operations, such as those which create router
ports.

每次资源的创建（如创建一个ｐｏｒｔｓ）／删除，　ｑｕｏｔａ　ｕｓａｇｅ　跟踪比较麻烦。

估计ｎｉｍｂｌｅ　也类似。　

To this aim, TrackedResource instances are initialised
with a reference to the model class for the resource for which
they track usage data. During object initialisation, SqlAlchemy event handlers are
installed for this class. The event handler is executed after a
record is inserted or deleted. As result usage data for that
resource and will be marked as ‘dirty’ once the operation
completes, so that the next time usage data is requested, it
will be synchronised counting resource usage from the database.
Even if this solution has some drawbacks, listed in the
‘exceptions and caveats’ section, it is more reliable than
solutions such as:

都引入了　
SqlAlchemy event handlers，　看来是比较麻烦。

• Updating the usage counters
  with the new ‘correct’ value every time an operation
  completes.
• Having a periodic task
  synchronising quota usage data with actual data in the
  Neutron DB.

Finally, regardless of whether CountableResource or
TrackedResource is used, the quota engine always invokes its
count() method to retrieve resource usage. Therefore, from the
perspective of the Quota engine there is absolutely no
difference between CountableResource and TrackedResource.

采用 CountableResource 还是
TrackedResource，　ｑｕｏｔａｓ　ｅｎｇｉｎｅ　没有区别，　都是调用同一个函数。

=======================================================================

http://docs.openstack.org/developer/neutron/devref/quota.html#quota-enforcement

看来，　ｑｕｏｔａ　ｅｎｇｉｎｅ　设计的还是不错，　考虑到了很多　ｅｄｇｅ　ｃａｓｅ。

1.  quota-enforcement (quota的执行）

Before dispatching a request to
the plugin, the Neutron ‘base’ controller [5] attempts
to make a reservation for requested
resource(s). Reservations are made by calling the
make_reservation method in neutron.quota.QuotaEngine. The
process of making a reservation is fairly straightforward:

dispatch
之前，先reserve

• Get current resource
  usages. This is achieved by invoking the count method on
  every requested resource, and then retrieving the amount
  of reserved resources.
• Fetch current quota limits
  for requested resources, by invoking the
  _get_tenant_quotas method.
• Fetch expired reservations for
  selected resources. This amount will be subtracted from resource usage.
  As in most cases there won’t be any expired reservation,
  this approach actually requires less DB operations than
  doing a sum of non-expired, reserved resources for each
  request.

还有过期的reservation， 需要从usage中扣掉。

• For each resource
  calculate its headroom,
  and verify the requested amount of resource is less than
  the headroom.

headroom， 哈哈， 老外们的词汇丰富啊。
直接使用 “可用的”， available，  更直接。

• If the above is true for
  all resource, the reservation is saved in the DB,
  otherwise an OverQuotaLimit exception is raised.

The quota
engine is able to make a reservation for multiple resources.
However, it is worth noting that because of the current
structure of the Neutron API layer, there will not be any
practical case in which a reservation for multiple resources
is made. For this reason performance optimisation avoiding
repeating queries for every resource are not part of the
current implementation.

quota
engine 设计的还是很好，支持多resource的reservation。

不过 neutron
API 没有使用。

Nimble需要考虑　多　ｒｅｓｏｕｒｃｅ　ｒｅｓｅｒｖａｔｉｏｎ　吗？

In order to ensure correct
operations, a row-level lock is
acquired in the transaction which creates the reservation. The
lock is acquired when reading usage data.
In case of write-set certification
failures, which can occur in active/active clusters
such as MySQL galera, the decorator
neutron.db.api.retry_db_errors will retry
the transaction if a DBDeadLock exception is raised.

创建
reservation的时候， 有 lock 用来避免race。　

双活　ＨＡ的　write-set
certification failures　都考虑了。　　有retry机制　

While
non-locking approaches are possible, it has been
found out that, since a non-locking algorithms increases the
chances of collision, the cost of handling a DBDeadlock is
still lower than the cost of retrying the operation when a
collision is detected. A study in this direction was conducted
for IP allocation operations, but the same principles apply
here as well [6]. Nevertheless, moving
away for DB-level locks is something that must happen
for quota enforcement in the future.

看看没有　锁　也是可行的，　然而那是后面的事情了。　

Committing and cancelling a
reservation is as simple as deleting the reservation itself.

Committing　reservation　流程：

When a reservation is committed,
the resources which were committed are now stored in the
database, so the reservation itself should be deleted.

cancelling　reservation　流程：

The Neutron quota engine simply
removes the record when cancelling a reservation (ie: the
request failed to complete), and also marks quota usage info
as dirty when the reservation is
committed (ie: the request completed correctly). Reservations
are committed or cancelled by respectively calling the
commit_reservation and cancel_reservation methods in
neutron.quota.QuotaEngine.

Reservations are not perennial. Eternal reservation would
eventually exhaust projects’ quotas because they would never
be removed when an API worker crashes whilst in the middle of
an operation. Reservation expiration is
currently set to 120 seconds, and is not
configurable, not yet at least. Expired reservations are not
counted when calculating resource usage. While creating a
reservation, if any expired reservation is found, all expired
reservation for that project and resource will be removed from
the database, thus avoiding build-up of expired reservations.

reservation
也有生命啊，　真可怜，跟人一样。　

默认　１２０ｓ　到期，　目前不可配置。　

nov quota api

这个是nova的quota api 说明
http://developer.openstack.org/api-ref/compute/?expanded=show-server-details-detail,show-a-quota-detail

{
    "quota_set": {
        "cores": 20,
        "fixed_ips": -1,
        "floating_ips": 10,
        "id": "fake_tenant",
        "injected_file_content_bytes": 10240,
        "injected_file_path_bytes": 255,
        "injected_files": 5,
        "instances": 10,
        "key_pairs": 100,
        "metadata_items": 128,
        "ram": 51200,
        "security_group_rules": 20,
        "security_groups": 10,
        "server_groups": 10,
        "server_group_members": 10
    }
}

inject file 相关的，都是cloud init 或者 config driver 相关的。
不知道， 裸机在deploy阶段， 可不可以 inject file？

fixed_ips， floating_ips  这个裸机有吗？

key_pairs, 我知道MAAS作为裸机管理，是可以注入， key的。 nimble 以后会增加这个功能吗？

ｍｅｔａｄａｔａ　security_group
server_groups， nimble 需要吗？

=======================================================================

感觉ｎｉｍｂｌｅ中，　ｐｏｒｔ是需要做ｑｕｏｔａ的。

ｉｒｏｎｉｃ　没有知道ｑｕｏｔａ
http://developer.openstack.org/api-ref/baremetal/

ｃｉｎｄｅｒ的　是有的。
http://developer.openstack.org/api-ref/block-storage/v3/?expanded=show-quotas-for-a-user-detail

Name	In	Type	Description
injected_file_content_bytes	body	integer	The number of bytes of content that are allowed for each injected file.
metadata_items	body	integer	The number of metadata items that are allowed for each instance.
reserved (Optional)	body	integer	Reserved volume size. Visible only if you set the usage=true query parameter.
in_use (Optional)	body	string	The in use data size. Visible only if you set the usage=true query parameter.
ram	body	integer	The amount of instance RAM in megabytes that are allowed for each tenant.
floating_ips	body	integer	The number of floating IP addresses that are allowed for each tenant.
key_pairs	body	integer	The number of key pairs that are allowed for each user.
injected_file_path_bytes	body	integer	The number of bytes that are allowed for each injected file path.
instances	body	integer	The number of instances that are allowed for each tenant.
security_group_rules (Optional)	body	integer	The number of rules that are allowed for each security group.
injected_files	body	integer	The number of injected files that are allowed for each tenant.
quota_set	body	object	A quota_set object.
cores	body	integer	The number of instance cores that are allowed for each tenant.
fixed_ips	body	integer	The number of fixed IP addresses that are allowed for each tenant. Must be equal to or greater than the number of allowed instances.
id	body	string	The UUID of the volume transfer.
security_groups	body	integer	The number of security groups that are allowed for each tenant.

 

 

 

=======================================================================

http://docs.openstack.org/developer/neutron/devref/quota.html#setting-up-resource-tracking-for-a-plugin

Setting up
Resource Tracking for a Plugin

By default plugins do not
leverage resource tracking. Having the plugin explicitly
declare which resources should be tracked is a precise design
choice aimed at limiting as much as possible the chance of
introducing errors in existing plugins.

For this reason a plugin must
declare which resource it intends to track. This can be
achieved using the tracked_resources decorator available in
the neutron.quota.resource_registry module. The decorator
should ideally be applied to the plugin’s __init__ method.

The decorator accepts in input a
list of keyword arguments. The name of the argument must be a
resource name, and the value of the argument must be a DB
model class. For example:

::

@resource_registry.tracked_resources(network=models_v2.Network,

port=models_v2.Port,
subnet=models_v2.Subnet,
subnetpool=models_v2.SubnetPool)

Will ensure network, port, subnet
and subnetpool resources are tracked. In theory, it is
possible to use this decorator multiple times, and not
exclusively to __init__ methods. However, this would
eventually lead to code readability and maintainability
problems, so developers are strongly encourage to apply this
decorator exclusively to the plugin’s __init__ method (or any
other method which is called by the plugin only once during
its initialization).

就是必须显示的声明那些resource 需要 track，
调用quota.resource_registry.tracked_resources 就可以了。
可以decorator 多次， 但是不建议这么做。

http://docs.openstack.org/developer/neutron/devref/quota.html#notes-for-implementors-of-rpc-interfaces-and-restful-controllers

Notes
for Implementors of RPC Interfaces and RESTful Controllers

Neutron unfortunately does not have a layer which is
called before dispatching the operation from the plugin which
can be leveraged both from RESTful and RPC over AMQP APIs. In
particular the RPC handlers call straight into the plugin,
without doing any request authorisation or quota enforcement.

Therefore RPC handlers must explicitly indicate if they
are going to call the plugin to create or delete any sort of
resources. This is achieved in a simple way, by ensuring
modified resources are marked as dirty after the RPC handler
execution terminates. To this aim developers can use the
mark_resources_dirty decorator available in the module
neutron.quota.resource_registry.

The decorator would scan the whole list of registered
resources, and store the dirty status for their usage trackers
in the database for those resources for which items have been
created or destroyed during the plugin operation.

RPC 调用， 不做任何 授权 和 配额。
所有 如果是 创建或者删除 资源的 RPC， 需要特别 指出。
用 neutron.quota.resource_registry.mark_resources_dirty 装饰一下就可以了。
这个装饰器 会 扫描 注册的resource， 标记发生变化的资源为 dirty 状态。       

=======================================================================

quota engine 还存在 局限性 :’(

http://docs.openstack.org/developer/neutron/devref/quota.html#exceptions-and-caveats

Exceptions and Caveats¶

Please be aware of the following limitations of the quota enforcement engine:

• Subnet allocation from subnet
  pools, in particularly shared pools, is also subject to quota limit
  checks. However this checks are not enforced by the quota engine, but
  trough a mechanism implemented in the neutron.ipam.subnetalloc
  module. This is because the Quota engine is not able to satisfy the
  requirements for quotas on subnet allocation.

         
哈哈，这是neutron得有的资源， nimble目前不存在这种资源。

          quota engine 做不了 subnet allocation的 quota 执行。

• The quota engine also provides a
  limit_check routine which enforces quota checks without creating
  reservations. This way of doing quota enforcement is extremely
  unreliable and superseded by the reservation mechanism.
  It has not been removed to ensure off-tree plugins and extensions which
  leverage are not broken.

         提供了只check 不创建reservations的 limit_check
routine， 应该是不建议使用了。

• SqlAlchemy events might not be
  the most reliable way for detecting changes in resource usage. Since the
  event mechanism monitors the data model class, it is paramount for a
  correct quota enforcement, that resources are
  always created and deleted using object relational mappings. For
  instance, deleting a resource with a query.delete call, will not trigger
  the event. SQLAlchemy events should be considered as a temporary
  measure adopted as Neutron lacks persistent API objects.

         
原来 SqlAlchemy events
不可靠。只是一个临时方案。
删除的时候，不会触发event。   :’(

         
Nimble 需要注意一下。

• As
  CountableResource instance do not track usage data, when making a
  reservation no write-intent lock is acquired. Therefore the quota engine
  with CountableResource
  is not concurrency-safe.

 CountableResource
不保证 并发安全

• The mechanism for specifying
  for which resources enable usage tracking relies on the fact that the
  plugin is loaded before quota-limited resources are registered. For this
  reason it is not possible to validate whether
  a resource actually exists or not when enabling tracking for it.
  Developers should pay particular attention into ensuring resource names
  are correctly specified.

 plugin
在 quota limited resource 注册之前加载，才能使能 resources usage tracking。 没有手段来校验 在使能 track的时候， 资源 是否存在了。

           这是给码农挖空，犯错误啊。   :’(

 
 nimble 应该不存在这个问题。  让neutron搞plugin， 搞这么复杂。

 

• The code assumes usage trackers
  are a trusted source of truth: if they report a usage counter and the
  dirty bit is not set, that counter is correct. If it’s dirty than surely
  that counter is out of sync. This is not
  very robust, as there might be issues upon restart when toggling the
  use_tracked_resources configuration variable, as stale counters might be
  trusted upon for making reservations. Also, the same situation might
  occur if a server crashes after the API operation
  is completed but before the reservation is committed, as the actual
  resource usage is changed but the corresponding usage tracker is not
  marked as dirty.

usage
trackers 会不robust。 

1.
在启动时， 切换 use_tracked_resources
configuration 变量， reservation  可能使用了
陈旧的 usage counter

2. 在 API 操作已完成， 但reservation 提交之前， servier crash了， 此时，真正的 resource
usage已经改变，但是相应的 usage tracker没有标记为dirty