CMDB服务器管理系统【s5day88】：采集资产-文件配置(一)

django中间件工作原理

整体流程：

在接受一个Http请求之前的准备

1. 启动一个支持WSGI网关协议的服务器监听端口等待外界的Http请求，比如Django自带的开发者服务器或者uWSGI服务器。
2. 服务器根据WSGI协议指定相应的Handler来处理Http请求，并且初始化该Handler，在Django框架中由框架自身负责实现这一个Handler。
3. 此时服务器已处于监听状态，可以接受外界的Http请求

当一个http请求到达服务器的时候

1. 服务器根据WSGI协议从Http请求中提取出必要的参数组成一个字典（environ）并传入Handler中进行处理。
2. 在Handler中对已经符合WSGI协议标准规定的http请求进行分析，比如加载Django提供的中间件，路由分配，调用路由匹配的视图等。
3. 返回一个可以被浏览器解析的符合Http协议的HttpResponse。

工作流程解析

1、在默认项目的wsgi.py文件中，application是由一个get_wsgi_application的函数返回的。

auto_server\auto_server\wsgi.py

def get_wsgi_application():
    """
    The public interface to Django's WSGI support. Should return a WSGI
    callable.

    Allows us to avoid making django.core.handlers.WSGIHandler public API, in
    case the internal WSGI implementation changes or moves in the future.
    """
    django.setup(set_prefix=False)
    return WSGIHandler()

总而言之，WSGIHandler在初始化的时候做了两件事情：

1. 初始化所有符合Django文档定义的中间件的钩子，比如process_view, process_request等。
2. 将self._middleware_chain属性赋值为经过convert_exception_to_response函数装饰的self._legacy_get_response。

2、初始化WSGIHandler

django\core\wsgi.py

class WSGIHandler(base.BaseHandler):
    request_class = WSGIRequest

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.load_middleware()

    def __call__(self, environ, start_response):
        set_script_prefix(get_script_name(environ))
        signals.request_started.send(sender=self.__class__, environ=environ)
        request = self.request_class(environ)
        response = self.get_response(request)

        response._handler_class = self.__class__

        status = '%d %s' % (response.status_code, response.reason_phrase)
        response_headers = list(response.items())
        for c in response.cookies.values():
            response_headers.append(('Set-Cookie', c.output(header='')))
        start_response(status, response_headers)
        if getattr(response, 'file_to_stream', None) is not None and environ.get('wsgi.file_wrapper'):
            response = environ['wsgi.file_wrapper'](response.file_to_stream)
        return response

def get_path_info(environ):
    """Return the HTTP request's PATH_INFO as a string."""
    path_info = get_bytes_from_wsgi(environ, 'PATH_INFO', '/')

    return repercent_broken_unicode(path_info).decode()

def get_script_name(environ):
    """
    Return the equivalent of the HTTP request's SCRIPT_NAME environment
    variable. If Apache mod_rewrite is used, return what would have been
    the script name prior to any rewriting (so it's the script name as seen
    from the client's perspective), unless the FORCE_SCRIPT_NAME setting is
    set (to anything).
    """
    if settings.FORCE_SCRIPT_NAME is not None:
        return settings.FORCE_SCRIPT_NAME

    # If Apache's mod_rewrite had a whack at the URL, Apache set either
    # SCRIPT_URL or REDIRECT_URL to the full resource URL before applying any
    # rewrites. Unfortunately not every Web server (lighttpd!) passes this
    # information through all the time, so FORCE_SCRIPT_NAME, above, is still
    # needed.
    script_url = get_bytes_from_wsgi(environ, 'SCRIPT_URL', '')
    if not script_url:
        script_url = get_bytes_from_wsgi(environ, 'REDIRECT_URL', '')

    if script_url:
        if b'//' in script_url:
            # mod_wsgi squashes multiple successive slashes in PATH_INFO,
            # do the same with script_url before manipulating paths (#17133).
            script_url = _slashes_re.sub(b'/', script_url)
        path_info = get_bytes_from_wsgi(environ, 'PATH_INFO', '')
        script_name = script_url[:-len(path_info)] if path_info else script_url
    else:
        script_name = get_bytes_from_wsgi(environ, 'SCRIPT_NAME', '')

    return script_name.decode()

def get_bytes_from_wsgi(environ, key, default):
    """
    Get a value from the WSGI environ dictionary as bytes.

    key and default should be strings.
    """
    value = environ.get(key, default)
    # Non-ASCII values in the WSGI environ are arbitrarily decoded with
    # ISO-8859-1. This is wrong for Django websites where UTF-8 is the default.
    # Re-encode to recover the original bytestring.
    return value.encode('iso-8859-1')

def get_str_from_wsgi(environ, key, default):
    """
    Get a value from the WSGI environ dictionary as str.

    key and default should be str objects.
    """
    value = get_bytes_from_wsgi(environ, key, default)
    return value.decode(errors='replace')

总而言之，WSGIHandler在初始化的时候做了两件事情：

1. 初始化所有符合Django文档定义的中间件的钩子，比如process_view, process_request等。
2. 将self._middleware_chain属性赋值为经过convert_exception_to_response函数装饰的self._legacy_get_response。

3、当WSGIHandler遇到Http请求

根据WSGI协议规定，application可以为一个函数，一个类，或者一个类的的实例

在get_wsgi_application函数中，可以看到application被指定为WSGIHandler类的实例，因此根据WSGI协议WSGIHanler类需要定义__call__方法。

django\core\handlers\wsgi.py

class WSGIHandler(base.BaseHandler):
    request_class = WSGIRequest

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.load_middleware()

    def __call__(self, environ, start_response):
        set_script_prefix(get_script_name(environ))
        signals.request_started.send(sender=self.__class__, environ=environ)
        request = self.request_class(environ)
        response = self.get_response(request)

        response._handler_class = self.__class__

        status = '%d %s' % (response.status_code, response.reason_phrase)
        response_headers = list(response.items())
        for c in response.cookies.values():
            response_headers.append(('Set-Cookie', c.output(header='')))
        start_response(status, response_headers)
        if getattr(response, 'file_to_stream', None) is not None and environ.get('wsgi.file_wrapper'):
            response = environ['wsgi.file_wrapper'](response.file_to_stream)
        return response

总结一下WSGIHandler做的主要事情：调用了BaseHandler中的self.get_response方法。

笔者在这里稍微提及一下魔术方法__call__的用法，它的作用是让类的实例能像函数一样被调用，就像重载了（）运算符

4、__call__方法

    def __call__(self, environ, start_response):
        set_script_prefix(get_script_name(environ))
        signals.request_started.send(sender=self.__class__, environ=environ)
        request = self.request_class(environ)
        response = self.get_response(request)

        response._handler_class = self.__class__

        status = '%d %s' % (response.status_code, response.reason_phrase)
        response_headers = list(response.items())
        for c in response.cookies.values():
            response_headers.append(('Set-Cookie', c.output(header='')))
        start_response(status, response_headers)
        if getattr(response, 'file_to_stream', None) is not None and environ.get('wsgi.file_wrapper'):
            response = environ['wsgi.file_wrapper'](response.file_to_stream)
        return response

举个例子

class Hello(object):

    def __init__(self):
        print("Class Hello instance init.")

    def __call__(self, *args, **kwargs):
        print('Hello instance call')

hello = Hello()
hello()

# output
Class Hello instance init.
Hello instance call

5、分析BaseHandler中的self.get_response方法

接下来分析BaseHandler中的self.get_response方法，在上文的BaseHandler的源代码中提及过，在get_response中调用了self._legacy_get_response方法，笔者从方法的名字推测这应该是Django的一个作为向前兼容的方法。

django\core\handlers\base.py

class BaseHandler(object):

    def get_response(self, request):
        """Return an HttpResponse object for the given HttpRequest."""
        # Setup default url resolver for this thread
        set_urlconf(settings.ROOT_URLCONF)

        # 在初始化load_middleware的时候self._middleware_chain属性被指定为handler，即经过convert_exception_to_response装饰的self._legacy_get_response，在这里执行了该方法
        response = self._middleware_chain(request)

        # This block is only needed for legacy MIDDLEWARE_CLASSES; if
        # MIDDLEWARE is used, self._response_middleware will be empty.
        try:
            # Apply response middleware, regardless of the response
            for middleware_method in self._response_middleware:
                response = middleware_method(request, response)
                # Complain if the response middleware returned None (a common error).
                if response is None:
                    raise ValueError(
                        "%s.process_response didn't return an "
                        "HttpResponse object. It returned None instead."
                        % (middleware_method.__self__.__class__.__name__))
        except Exception:  # Any exception should be gathered and handled
            signals.got_request_exception.send(sender=self.__class__, request=request)
            response = self.handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())

        response._closable_objects.append(request)

        # If the exception handler returns a TemplateResponse that has not
        # been rendered, force it to be rendered.
        if not getattr(response, 'is_rendered', True) and callable(getattr(response, 'render', None)):
            response = response.render()

        if response.status_code == 404:
            logger.warning(
                'Not Found: %s', request.path,
                extra={'status_code': 404, 'request': request},
            )

        return response

    def _get_response(self, request):
        """
        Resolve and call the view, then apply view, exception, and
        template_response middleware. This method is everything that happens
        inside the request/response middleware.
        """
        response = None

        if hasattr(request, 'urlconf'):
            urlconf = request.urlconf
            set_urlconf(urlconf)
            resolver = get_resolver(urlconf)
        else:
            resolver = get_resolver()

        # 从request中获取url，然后经过路由解析，匹配到对应的view
        resolver_match = resolver.resolve(request.path_info)
        callback, callback_args, callback_kwargs = resolver_match
        request.resolver_match = resolver_match

        # 加载process_view钩子
        # Apply view middleware
        for middleware_method in self._view_middleware:
            response = middleware_method(request, callback, callback_args, callback_kwargs)
            if response:
                break

        # 执行开发者自己定义的业务逻辑，即view
        if response is None:
            wrapped_callback = self.make_view_atomic(callback)
            try:
                response = wrapped_callback(request, *callback_args, **callback_kwargs)
            except Exception as e:
                response = self.process_exception_by_middleware(e, request)

        # Complain if the view returned None (a common error).
        if response is None:
            if isinstance(callback, types.FunctionType):    # FBV
                view_name = callback.__name__
            else:                                           # CBV
                view_name = callback.__class__.__name__ + '.__call__'

            raise ValueError(
                "The view %s.%s didn't return an HttpResponse object. It "
                "returned None instead." % (callback.__module__, view_name)
            )

        # If the response supports deferred rendering, apply template
        # response middleware and then render the response
        elif hasattr(response, 'render') and callable(response.render):
            for middleware_method in self._template_response_middleware:
                response = middleware_method(request, response)
                # Complain if the template response middleware returned None (a common error).
                if response is None:
                    raise ValueError(
                        "%s.process_template_response didn't return an "
                        "HttpResponse object. It returned None instead."
                        % (middleware_method.__self__.__class__.__name__)
                    )

            try:
                response = response.render()
            except Exception as e:
                response = self.process_exception_by_middleware(e, request)

        return response

    def _legacy_get_response(self, request):
        """
        Apply process_request() middleware and call the main _get_response(),
        if needed. Used only for legacy MIDDLEWARE_CLASSES.
        """
        response = None
        # Apply request middleware
        for middleware_method in self._request_middleware:
            response = middleware_method(request)
            if response:
                break

        if response is None:
            response = self._get_response(request)
        return response

BaseHandler三个方法之间的关系

self.get_response调用了self._legacy_get_response，self._legacy_get_response在加载了所有process_request钩子之后，调用了self._get_response。

仔细分析一下这三个方法都干了什么事情：

self.get_response

1. 调用了self._legacy_get_response方法
2. 得到self._legacy_get_response方法返回的结果之后加载process_response钩子

self._get_response

1. 路由解析
2. 加载process_view钩子
3. 执行view（开发者自行定义的业务逻辑）
4. 加载process_template_response钩子

self._legacy_get_response

1. 加载process_request钩子
2. 调用了self._get_response方法

最后，HttpResponse被传送回WSHIHandler的__call__方法中，并按照HTTP协议返回给浏览器。

高度可扩展，可插拔式插件，参考Django源码中的中间件

流程图

django\utils\module_loading.py

def import_string(dotted_path):
    """
    Import a dotted module path and return the attribute/class designated by the
    last name in the path. Raise ImportError if the import failed.
    """
    try:
        module_path, class_name = dotted_path.rsplit('.', 1)
    except ValueError as err:
        raise ImportError("%s doesn't look like a module path" % dotted_path) from err

    module = import_module(module_path)

    try:
        return getattr(module, class_name)
    except AttributeError as err:
        raise ImportError('Module "%s" does not define a "%s" attribute/class' % (
            module_path, class_name)
        ) from err

小结：

请求来了，先到达wsgi(因为django没有socket用的是wsgi)

1、manage.py 创建完WSGIHandler()这个对象后什么都不做了停了，

1. 只要一起启动，就会加载配置文件加载到内存
2. 对象后面加()是执行call方法  def __call__
3. 类后面加()是执行构造方法

2、停了等待什么？等待用户发来请求
3、只要有一个用户发来请求，整个call方法就会开始执行
4、返回给用户的 return response
5、你如果把源码的call方法删除，它肯定就运行不小了

配置文件

1、auto_server\auto_server\viwes.py

    from django.conf import settings

2、django\conf\__init__.py

   settings = LazySettings() #是某个类的对象

3、django\conf\__init__.py 中的class LazySettings(LazyObject)类

    def _setup(self, name=None):
         settings_module = os.environ.get(ENVIRONMENT_VARIABLE) #这里读了一下配置文件  

    ......    

django启动auto_server\manage.py

   if __name__ == "__main__":
        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "auto_server.settings")
        # 启动的时候os.environ.setdefault赋值 key：DJANGO_SETTINGS_MODULE  vlales：auto_server.settings是我写的配置文件的路径

4、django\conf\__init__.py 中的class LazySettings(LazyObject)类

class LazySettings(LazyObject):

    def _setup(self, name=None):
        settings_module = os.environ.get(ENVIRONMENT_VARIABLE)
       ......

        self._wrapped = Settings(settings_module)
         #实例化了一个对象

5、  django\conf\global_settings.py是个什么？

 class Settings(BaseSettings):
    def __init__(self, settings_module):
        # update this dict from global settings (but only for ALL_CAPS settings)
        for setting in dir(global_settings):
		#dir找到它里面所有的变量，global_settings是个什么鬼？是全局配置变量
            if setting.isupper():
                setattr(self, setting, getattr(global_settings, setting))

        # store the settings module in case someone later cares
        self.SETTINGS_MODULE = settings_module

        mod = importlib.import_module(self.SETTINGS_MODULE)
		#导入了这个路径

6、你的配置如果写成小写，就是因为这它是不是没有读

 class Settings(BaseSettings):
    def __init__(self, settings_module):
        # update this dict from global settings (but only for ALL_CAPS settings)
        for setting in dir(global_settings):
		#dir找到它里面所有的变量，global_settings是个什么鬼？是全局配置变量
            if setting.isupper():
                setattr(self, setting, getattr(global_settings, setting))

        # store the settings module in case someone later cares
        self.SETTINGS_MODULE = settings_module

        mod = importlib.import_module(self.SETTINGS_MODULE)

		#导入了这个路径

        tuple_settings = (
            "ALLOWED_INCLUDE_ROOTS",
            "INSTALLED_APPS",
            "TEMPLATE_DIRS",
            "LOCALE_PATHS",
        )
        self._explicit_settings = set()
        for setting in dir(mod):
            if setting.isupper():
			#你的配置如果写成小写，就是因为这没有通过
                setting_value = getattr(mod, setting)

                if (setting in tuple_settings and
                        isinstance(setting_value, six.string_types)):
                    raise ImproperlyConfigured("The %s setting must be a tuple. "
                            "Please fix your settings." % setting)
                setattr(self, setting, setting_value)
                self._explicit_settings.add(setting)

        if not self.SECRET_KEY:
            raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")

        if ('django.contrib.auth.middleware.AuthenticationMiddleware' in self.MIDDLEWARE_CLASSES and
                'django.contrib.auth.middleware.SessionAuthenticationMiddleware' not in self.MIDDLEWARE_CLASSES):
            warnings.warn(
                "Session verification will become mandatory in Django 1.10. "
                "Please add 'django.contrib.auth.middleware.SessionAuthenticationMiddleware' "
                "to your MIDDLEWARE_CLASSES setting when you are ready to opt-in after "
                "reading the upgrade considerations in the 1.8 release notes.",
                RemovedInDjango110Warning
            )

        if hasattr(time, 'tzset') and self.TIME_ZONE:
            # When we can, attempt to validate the timezone. If we can't find
            # this file, no check happens and it's harmless.
            zoneinfo_root = '/usr/share/zoneinfo'
            if (os.path.exists(zoneinfo_root) and not
                    os.path.exists(os.path.join(zoneinfo_root, *(self.TIME_ZONE.split('/'))))):
                raise ValueError("Incorrect timezone setting: %s" % self.TIME_ZONE)
            # Move the time zone info into os.environ. See ticket #2315 for why
            # we don't do this unconditionally (breaks Windows).
            os.environ['TZ'] = self.TIME_ZONE
            time.tzset()

    def is_overridden(self, setting):
        return setting in self._explicit_settings

1. 读默认的配置
2. 读我们导入的配置
3. 默认的即使有了，我后来的还能给他覆盖，所以说用户定义的优先级更高

配置文件：为了让用户使用方便，将默认配置文件，放在内部；只让用户做常用配置

__init__.py

import os
import importlib
from . import global_settings

class Settings(object):
    """
    global_settings,配置获取
    settings.py，配置获取
    """
    def __init__(self):

        for item in dir(global_settings):
            if item.isupper():
                k = item
                v = getattr(global_settings,item)
'''
这就是getattr的本质
我给你一个py文件，是不是一个对象
我要去对象里面拿它那的元素怎么拿？
'''
                setattr(self,k,v)

        setting_path = os.environ.get('AUTO_CLIENT_SETTINGS')
        md_settings = importlib.import_module(setting_path)
        for item in dir(md_settings):
            if item.isupper():
                k = item
                v = getattr(md_settings,item)
                setattr(self,k,v)

settings = Settings()

1. 设置环境变量:os.environ['AUTO_CLIENT_SETTINGS'] = "conf.settings"
2. 默认配置+用户配置
3. importlib 和 getattr  setattr

global_settings.py

TEST = True

NAME = "GAOXU"

test.py

import sys
import os
import importlib
import requests

BASEDIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.append(BASEDIR)

from lib.config import settings

拿到所有的变量

__init__.py

__init__.py

from . import global_settings

class Settings(object):
	"""
	global_settings 获取
	settings 获取

	"""
    def __init__(self):
        for items in dir(global_settings):
                #items 方法和属性
                print(items)

settings = Settings()

test.py

import sys
import os
import importlib
import requests

BASEDIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.append(BASEDIR)

from lib.config import settings

global_settings.py

TEST = True

NAME = "GAOXU"

截图

只拿大写

__init__.py

from . import global_settings

class Settings(object):

    def __init__(self):
        for item in dir(global_settings):
            if item.isupper():
                #items 方法和属性
                print(item,getattr(global_settings,item))

settings = Settings()

截图

打印API

用户设置优先级高

文件形式实现：单例模式

src.a1.py

class Foo:
    pass

obj = Foo()

src.a2.py

from src.a1 import obj
print(obj)

src.a2.py

#单例模式 不管怎么玩，用的都是同一个对象
from src.a1 import obj
print(obj)
from src.a1 import obj
print(obj)