kestrel网络编程--开发Fiddler

1 文章目的

本文讲解基于kestrel开发类似Fiddler应用的过程，让读者了解kestrel网络编程里面的kestrel中间件和http应用中间件。由于最终目的不是输出完整功能的产品，所以这里只实现Fiddler最核心的http请求和响应内容查看的功能。本文章是KestrelApp项目里面的一个demo的讲解，希望对您有用。

2 开发顺序

代理协议kestrel中间件
tls协议侦测kestrel中间件
隧道和http协议侦测kestrel中间件
请求响应分析http中间件
反向代理http中间件
编排中间件创建服务器和应用

3 传输层与kestrel中间件

所谓传输层，其目的是为了让应用协议数据安全、可靠、快速等传输而存在的一种协议，其特征是把应用协议的报文做为自己的负载，常见的tcp、udp、quic、tls等都可以理解为传输层协议。

比如http协议，常见有如下的传输方式：

http over tcp
http over tls over tcp
http over quic over udp

3.1 Fiddler的传输层

Fiddler要处理以下三种http传输情况：

http over tcp：直接http请求首页
http over proxy over tcp：代理http流量
http over tls over proxy over tcp：代理https流量

3.2 Kestrel的中间件

kestrel目前的传输层基于tcp或quic两种，同时内置了tls中间件，需要调用ListenOptions.UseHttps()来使用tls中间件。kestrel的中间件的表现形式为：Func<ConnectionDelegate, ConnectionDelegate>，为了使用读者能够简单理解中间件，我在KestrelFramework里定义了kestrel中间件的变种接口，大家基于此接口来实现更多的中间件就方便很多：

/// <summary>

/// Kestrel的中间件接口

/// </summary>

public interface IKestrelMiddleware

{

    /// <summary>

    /// 执行

    /// </summary>

    /// <param name="next"></param>

    /// <param name="context"></param>

    /// <returns></returns>

    Task InvokeAsync(ConnectionDelegate next, ConnectionContext context);

}

4 代理协议kestrel中间件

Filddler最基础的功能是它是一个http代理服务器，我们需要为kestrel编写代理中间件，用于处理代理传输层。http代理协议分两种：普通的http代理和Connect隧道代理。两种的报文者是遵循http1.0或1.1的文本格式，我们可以使用kestrel自带的HttpParser<>来解析这些复杂的http文本协议。

4.1 代理特征

在中间件编程模式中，Feature是一个很重要的中间件沟通桥梁，它往往是某个中间件工作之后，留下的财产，让之后的中间件来获取并受益。我们的代理中间件，也设计了IProxyFeature，告诉之后的中间件一些代理特征。

/// <summary>

/// 代理Feature

/// </summary>

public interface IProxyFeature

{

    /// <summary>

    /// 代理主机

    /// </summary>

    HostString ProxyHost { get; }

    /// <summary>

    /// 代理协议

    /// </summary>

    ProxyProtocol ProxyProtocol { get; }

}

/// <summary>

/// 代理协议

/// </summary>

public enum ProxyProtocol

{

    /// <summary>

    /// 无代理

    /// </summary>

    None,

    /// <summary>

    /// http代理

    /// </summary>

    HttpProxy,

    /// <summary>

    /// 隧道代理

    /// </summary>

    TunnelProxy

}

4.2 代理中间件的实现

/// <summary>

/// 代理中间件

/// </summary>

sealed class KestrelProxyMiddleware : IKestrelMiddleware

{

    private static readonly HttpParser<HttpRequestHandler> httpParser = new();

    private static readonly byte[] http200 = Encoding.ASCII.GetBytes("HTTP/1.1 200 Connection Established\r\n\r\n");

    private static readonly byte[] http400 = Encoding.ASCII.GetBytes("HTTP/1.1 400 Bad Request\r\n\r\n");

    /// <summary>

    /// 解析代理

    /// </summary>

    /// <param name="next"></param>

    /// <param name="context"></param>

    /// <returns></returns>

    public async Task InvokeAsync(ConnectionDelegate next, ConnectionContext context)

    {

        var input = context.Transport.Input;

        var output = context.Transport.Output;

        var request = new HttpRequestHandler();

        while (context.ConnectionClosed.IsCancellationRequested == false)

        {

            var result = await input.ReadAsync();

            if (result.IsCanceled)

            {

                break;

            }

            try

            {

                if (ParseRequest(result, request, out var consumed))

                {

                    if (request.ProxyProtocol == ProxyProtocol.TunnelProxy)

                    {

                        input.AdvanceTo(consumed);

                        await output.WriteAsync(http200);

                    }

                    else

                    {

                        input.AdvanceTo(result.Buffer.Start);

                    }

                    context.Features.Set<IProxyFeature>(request);

                    await next(context);

                    break;

                }

                else

                {

                    input.AdvanceTo(result.Buffer.Start, result.Buffer.End);

                }

                if (result.IsCompleted)

                {

                    break;

                }

            }

            catch (Exception)

            {

                await output.WriteAsync(http400);

                break;

            }

        }

    }

    /// <summary>

    /// 解析http请求

    /// </summary>

    /// <param name="result"></param>

    /// <param name="request"></param>

    /// <param name="consumed"></param>

    /// <returns></returns>

    private static bool ParseRequest(ReadResult result, HttpRequestHandler request, out SequencePosition consumed)

    {

        var reader = new SequenceReader<byte>(result.Buffer);

        if (httpParser.ParseRequestLine(request, ref reader) &&

            httpParser.ParseHeaders(request, ref reader))

        {

            consumed = reader.Position;

            return true;

        }

        else

        {

            consumed = default;

            return false;

        }

    }

    /// <summary>

    /// 代理请求处理器

    /// </summary>

    private class HttpRequestHandler : IHttpRequestLineHandler, IHttpHeadersHandler, IProxyFeature

    {

        private HttpMethod method;

        public HostString ProxyHost { get; private set; }

        public ProxyProtocol ProxyProtocol

        {

            get

            {

                if (ProxyHost.HasValue == false)

                {

                    return ProxyProtocol.None;

                }

                if (method == HttpMethod.Connect)

                {

                    return ProxyProtocol.TunnelProxy;

                }

                return ProxyProtocol.HttpProxy;

            }

        }

        void IHttpRequestLineHandler.OnStartLine(HttpVersionAndMethod versionAndMethod, TargetOffsetPathLength targetPath, Span<byte> startLine)

        {

            method = versionAndMethod.Method;

            var host = Encoding.ASCII.GetString(startLine.Slice(targetPath.Offset, targetPath.Length));

            if (versionAndMethod.Method == HttpMethod.Connect)

            {

                ProxyHost = HostString.FromUriComponent(host);

            }

            else if (Uri.TryCreate(host, UriKind.Absolute, out var uri))

            {

                ProxyHost = HostString.FromUriComponent(uri);

            }

        }

        void IHttpHeadersHandler.OnHeader(ReadOnlySpan<byte> name, ReadOnlySpan<byte> value)

        {

        }

        void IHttpHeadersHandler.OnHeadersComplete(bool endStream)

        {

        }

        void IHttpHeadersHandler.OnStaticIndexedHeader(int index)

        {

        }

        void IHttpHeadersHandler.OnStaticIndexedHeader(int index, ReadOnlySpan<byte> value)

        {

        }

    }

}

5 tls协议侦测kestrel中间件

Fiddler只监听了一个端口，要同时支持非加密和加密两种流量，如果不调用调用ListenOptions.UseHttps()，我们的程序就不支持https的分析；如果直接调用ListenOptions.UseHttps()，会让我们的程序不支持非加密的http的分析，这就要求我们有条件的根据客户端发来的流量分析是否需要开启。

我已经在KestrelFramework内置了TlsDetection中间件，这个中间件可以根据客户端的实际流量类型来选择是否使用tls。在Fiddler中，我们还需要根据客户端的tls握手中的sni使用ca证书来动态生成服务器证书用于tls加密传输。

/// <summary>

/// 证书服务

/// </summary>

sealed class CertService

{

    private const string CACERT_PATH = "cacert";

    private readonly IMemoryCache serverCertCache;

    private readonly IEnumerable<ICaCertInstaller> certInstallers;

    private readonly ILogger<CertService> logger;

    private X509Certificate2? caCert;

    /// <summary>

    /// 获取证书文件路径

    /// </summary>

    public string CaCerFilePath { get; } = OperatingSystem.IsLinux() ? $"{CACERT_PATH}/fiddler.crt" : $"{CACERT_PATH}/fiddler.cer";

    /// <summary>

    /// 获取私钥文件路径

    /// </summary>

    public string CaKeyFilePath { get; } = $"{CACERT_PATH}/fiddler.key";

    /// <summary>

    /// 证书服务

    /// </summary>

    /// <param name="serverCertCache"></param>

    /// <param name="certInstallers"></param>

    /// <param name="logger"></param>

    public CertService(

        IMemoryCache serverCertCache,

        IEnumerable<ICaCertInstaller> certInstallers,

        ILogger<CertService> logger)

    {

        this.serverCertCache = serverCertCache;

        this.certInstallers = certInstallers;

        this.logger = logger;

        Directory.CreateDirectory(CACERT_PATH);

    }

    /// <summary>

    /// 生成CA证书

    /// </summary>

    public bool CreateCaCertIfNotExists()

    {

        if (File.Exists(this.CaCerFilePath) && File.Exists(this.CaKeyFilePath))

        {

            return false;

        }

        File.Delete(this.CaCerFilePath);

        File.Delete(this.CaKeyFilePath);

        var notBefore = DateTimeOffset.Now.AddDays(-1);

        var notAfter = DateTimeOffset.Now.AddYears(10);

        var subjectName = new X500DistinguishedName($"CN={nameof(Fiddler)}");

        this.caCert = CertGenerator.CreateCACertificate(subjectName, notBefore, notAfter);

        var privateKeyPem = this.caCert.GetRSAPrivateKey()?.ExportRSAPrivateKeyPem();

        File.WriteAllText(this.CaKeyFilePath, new string(privateKeyPem), Encoding.ASCII);

        var certPem = this.caCert.ExportCertificatePem();

        File.WriteAllText(this.CaCerFilePath, new string(certPem), Encoding.ASCII);

        return true;

    }

    /// <summary>

    /// 安装和信任CA证书

    /// </summary>

    public void InstallAndTrustCaCert()

    {

        var installer = this.certInstallers.FirstOrDefault(item => item.IsSupported());

        if (installer != null)

        {

            installer.Install(this.CaCerFilePath);

        }

        else

        {

            this.logger.LogWarning($"请根据你的系统平台手动安装和信任CA证书{this.CaCerFilePath}");

        }

    }

    /// <summary>

    /// 获取颁发给指定域名的证书

    /// </summary>

    /// <param name="domain"></param>

    /// <returns></returns>

    public X509Certificate2 GetOrCreateServerCert(string? domain)

    {

        if (this.caCert == null)

        {

            using var rsa = RSA.Create();

            rsa.ImportFromPem(File.ReadAllText(this.CaKeyFilePath));

            this.caCert = new X509Certificate2(this.CaCerFilePath).CopyWithPrivateKey(rsa);

        }

        var key = $"{nameof(CertService)}:{domain}";

        var endCert = this.serverCertCache.GetOrCreate(key, GetOrCreateCert);

        return endCert!;

        // 生成域名的1年证书

        X509Certificate2 GetOrCreateCert(ICacheEntry entry)

        {

            var notBefore = DateTimeOffset.Now.AddDays(-1);

            var notAfter = DateTimeOffset.Now.AddYears(1);

            entry.SetAbsoluteExpiration(notAfter);

            var extraDomains = GetExtraDomains();

            var subjectName = new X500DistinguishedName($"CN={domain}");

            var endCert = CertGenerator.CreateEndCertificate(this.caCert, subjectName, extraDomains, notBefore, notAfter);

            // 重新初始化证书，以兼容win平台不能使用内存证书

            return new X509Certificate2(endCert.Export(X509ContentType.Pfx));

        }

    }

    /// <summary>

    /// 获取域名

    /// </summary>

    /// <param name="domain"></param>

    /// <returns></returns>

    private static IEnumerable<string> GetExtraDomains()

    {

        yield return Environment.MachineName;

        yield return IPAddress.Loopback.ToString();

        yield return IPAddress.IPv6Loopback.ToString();

    }

}

6 隧道和http协议侦测kestrel中间件

经过KestrelProxyMiddleware后的流量，在tls解密(如果可能)之后，一般情况下都是http流量了，但如果你在qq设置代理到我们这个伪Fildder之后，会发现部分流量流量不是http流量，原因是http隧道也是一个通用传输层，可以传输任意tcp或tcp之上的流量。所以我们需要新的中间件来检测当前流量，如果不是http流量就回退到隧道代理的流程，即我们不跟踪不分析这部分非http流量。

6.1 http流量侦测

/// <summary>

/// 流量侦测器

/// </summary>

private static class FlowDetector

{

    private static readonly byte[] crlf = Encoding.ASCII.GetBytes("\r\n");

    private static readonly byte[] http10 = Encoding.ASCII.GetBytes(" HTTP/1.0");

    private static readonly byte[] http11 = Encoding.ASCII.GetBytes(" HTTP/1.1");

    private static readonly byte[] http20 = Encoding.ASCII.GetBytes(" HTTP/2.0");

    /// <summary>

    /// 传输内容是否为http

    /// </summary>

    /// <param name="context"></param>

    /// <returns></returns>

    public static async ValueTask<bool> IsHttpAsync(ConnectionContext context)

    {

        var input = context.Transport.Input;

        var result = await input.ReadAtLeastAsync(1);

        var isHttp = IsHttp(result);

        input.AdvanceTo(result.Buffer.Start);

        return isHttp;

    }

    private static bool IsHttp(ReadResult result)

    {

        var reader = new SequenceReader<byte>(result.Buffer);

        if (reader.TryReadToAny(out ReadOnlySpan<byte> line, crlf))

        {

            return line.EndsWith(http11) || line.EndsWith(http20) || line.EndsWith(http10);

        }

        return false;

    }

}

6.2 隧道回退中间件

/// <summary>

/// 隧道传输中间件

/// </summary>

sealed class KestrelTunnelMiddleware : IKestrelMiddleware

{

    private readonly ILogger<KestrelTunnelMiddleware> logger;

    /// <summary>

    /// 隧道传输中间件

    /// </summary>

    /// <param name="logger"></param>

    public KestrelTunnelMiddleware(ILogger<KestrelTunnelMiddleware> logger)

    {

        this.logger = logger;

    }

    /// <summary>

    /// 执行中间你件

    /// </summary>

    /// <param name="next"></param>

    /// <param name="context"></param>

    /// <returns></returns>

    public async Task InvokeAsync(ConnectionDelegate next, ConnectionContext context)

    {

        var feature = context.Features.Get<IProxyFeature>();

        if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)

        {

            this.logger.LogInformation($"侦测到http直接请求");

            await next(context);

        }

        else if (feature.ProxyProtocol == ProxyProtocol.HttpProxy)

        {

            this.logger.LogInformation($"侦测到普通http代理流量");

            await next(context);

        }

        else if (await FlowDetector.IsHttpAsync(context))

        {

            this.logger.LogInformation($"侦测到隧道传输http流量");

            await next(context);

        }

        else

        {

            this.logger.LogInformation($"跳过隧道传输非http流量{feature.ProxyHost}的拦截");

            await TunnelAsync(context, feature);

        }

    }

    /// <summary>

    /// 隧道传输其它协议的数据

    /// </summary>

    /// <param name="context"></param>

    /// <param name="feature"></param>

    /// <returns></returns>

    private async ValueTask TunnelAsync(ConnectionContext context, IProxyFeature feature)

    {

        var port = feature.ProxyHost.Port;

        if (port == null)

        {

            return;

        }

        try

        {

            var host = feature.ProxyHost.Host;

            using var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);

            await socket.ConnectAsync(host, port.Value, context.ConnectionClosed);

            Stream stream = new NetworkStream(socket, ownsSocket: false);

            // 如果有tls中间件，则反回来加密隧道

            if (context.Features.Get<ITlsConnectionFeature>() != null)

            {

                var sslStream = new SslStream(stream, leaveInnerStreamOpen: true);

                await sslStream.AuthenticateAsClientAsync(feature.ProxyHost.Host);

                stream = sslStream;

            }

            var task1 = stream.CopyToAsync(context.Transport.Output);

            var task2 = context.Transport.Input.CopyToAsync(stream);

            await Task.WhenAny(task1, task2);

        }

        catch (Exception ex)

        {

            this.logger.LogError(ex, $"连接到{feature.ProxyHost}异常");

        }

    }

}

7 请求响应分析http中间件

这部分属于asp.netcore应用层内容，关键点是制作可多次读取的http请求body流和http响应body流，因为每个分析器实例都可以会重头读取一次请求内容和响应内容。

7.1 http分析器

为了方便各种分析器的独立实现，我们定义http分析器的接口

/// <summary>

/// http分析器

/// 支持多个实例

/// </summary>

public interface IHttpAnalyzer

{

    /// <summary>

    /// 分析http

    /// </summary>

    /// <param name="context"></param>

    /// <returns></returns>

    ValueTask AnalyzeAsync(HttpContext context);

}

这是输到日志的http分析器

public class LoggingHttpAnalyzer : IHttpAnalyzer

{

    private readonly ILogger<LoggingHttpAnalyzer> logger;

    public LoggingHttpAnalyzer(ILogger<LoggingHttpAnalyzer> logger)

    {

        this.logger = logger;

    }

    public async ValueTask AnalyzeAsync(HttpContext context)

    {

        var builder = new StringBuilder();

        var writer = new StringWriter(builder);

        writer.WriteLine("[REQUEST]");

        await context.SerializeRequestAsync(writer);

        writer.WriteLine("[RESPONSE]");

        await context.SerializeResponseAsync(writer);

        this.logger.LogInformation(builder.ToString());

    }

}

7.2 分析http中间件

我们把请求body流和响应body流保存到临时文件，在所有分析器工作之后再删除。

/// <summary>

/// http分析中间件

/// </summary>

sealed class HttpAnalyzeMiddleware

{

    private readonly RequestDelegate next;

    private readonly IEnumerable<IHttpAnalyzer> analyzers;

    /// <summary>

    /// http分析中间件

    /// </summary>

    /// <param name="next"></param>

    /// <param name="analyzers"></param>

    public HttpAnalyzeMiddleware(

        RequestDelegate next,

        IEnumerable<IHttpAnalyzer> analyzers)

    {

        this.next = next;

        this.analyzers = analyzers;

    }

    /// <summary>

    /// 分析代理的http流量

    /// </summary>

    /// <param name="context"></param>

    /// <returns></returns>

    public async Task InvokeAsync(HttpContext context)

    {

        var feature = context.Features.Get<IProxyFeature>();

        if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)

        {

            await next(context);

            return;

        }

        context.Request.EnableBuffering();

        var oldBody = context.Response.Body;

        using var response = new FileResponse();

        try

        {

            // 替换response的body

            context.Response.Body = response.Body;

            // 请求下个中间件

            await next(context);

            // 处理分析

            await this.AnalyzeAsync(context);

        }

        finally

        {

            response.Body.Position = 0L;

            await response.Body.CopyToAsync(oldBody);

            context.Response.Body = oldBody;

        }

    }

    private async ValueTask AnalyzeAsync(HttpContext context)

    {

        foreach (var item in this.analyzers)

        {

            context.Request.Body.Position = 0L;

            context.Response.Body.Position = 0L;

            await item.AnalyzeAsync(context);

        }

    }

    private class FileResponse : IDisposable

    {

        private readonly string filePath = Path.GetTempFileName();

        public Stream Body { get; }

        public FileResponse()

        {

            this.Body = new FileStream(filePath, FileMode.Open, FileAccess.ReadWrite);

        }

        public void Dispose()

        {

            this.Body.Dispose();

            File.Delete(filePath);

        }

    }

}

8 反向代理http中间件

我们需要把请求转发到真实的目标服务器，这时我们的应用程序是一个http客户端角色，这个过程与nginx的反向代理是一致的。具体的实现上，我们直接使用yarp库来完成即可。

/// <summary>

/// http代理执行中间件

/// </summary>

sealed class HttpForwardMiddleware

{

    private readonly RequestDelegate next;

    private readonly IHttpForwarder httpForwarder;

    private readonly HttpMessageInvoker httpClient = new(CreateSocketsHttpHandler());

    /// <summary>

    /// http代理执行中间件

    /// </summary>

    /// <param name="next"></param>

    /// <param name="httpForwarder"></param>

    public HttpForwardMiddleware(

        RequestDelegate next,

        IHttpForwarder httpForwarder)

    {

        this.next = next;

        this.httpForwarder = httpForwarder;

    }

    /// <summary>

    /// 转发http流量

    /// </summary>

    /// <param name="context"></param>

    /// <returns></returns>

    public async Task InvokeAsync(HttpContext context)

    {

        var feature = context.Features.Get<IProxyFeature>();

        if (feature == null || feature.ProxyProtocol == ProxyProtocol.None)

        {

            await next(context);

        }

        else

        {

            var scheme = context.Request.Scheme;

            var destinationPrefix = $"{scheme}://{feature.ProxyHost}";

            await httpForwarder.SendAsync(context, destinationPrefix, httpClient, ForwarderRequestConfig.Empty, HttpTransformer.Empty);

        }

    }

    private static SocketsHttpHandler CreateSocketsHttpHandler()

    {

        return new SocketsHttpHandler

        {

            Proxy = null,

            UseProxy = false,

            UseCookies = false,

            AllowAutoRedirect = false,

            AutomaticDecompression = DecompressionMethods.None,

        };

    }

}

9 编排中间件创建服务器和应用

9.1 kestrel中间件编排

这里要特别注意顺序，传输层套娃。

/// <summary>

///  ListenOptions扩展

/// </summary>

public static partial class ListenOptionsExtensions

{

    /// <summary>

    /// 使用Fiddler的kestrel中间件

    /// </summary>

    /// <param name="listen"></param>

    public static ListenOptions UseFiddler(this ListenOptions listen)

    {

        // 代理协议中间件

        listen.Use<KestrelProxyMiddleware>();

        // tls侦测中间件

        listen.UseTlsDetection(tls =>

        {

            var certService = listen.ApplicationServices.GetRequiredService<CertService>();

            certService.CreateCaCertIfNotExists();

            certService.InstallAndTrustCaCert();

            tls.ServerCertificateSelector = (context, domain) => certService.GetOrCreateServerCert(domain);

        });

        // 隧道代理处理中间件

        listen.Use<KestrelTunnelMiddleware>();

        return listen;

    }

}

9.2 http中间件的编排

public static class ApplicationBuilderExtensions

{

    /// <summary>

    /// 使用Fiddler的http中间件

    /// </summary>

    /// <param name="app"></param>

    public static void UseFiddler(this IApplicationBuilder app)

    {

        app.UseMiddleware<HttpAnalyzeMiddleware>();

        app.UseMiddleware<HttpForwardMiddleware>();

    }

}

9.3 创建应用

我们可以在传统的MVC里创建伪fiddler的首页、下载证书等http交互页面。

public static void Main(string[] args)

{

    var builder = WebApplication.CreateBuilder(args);

    builder.Services

        .AddFiddler()

        .AddControllers();

    builder.WebHost.ConfigureKestrel((context, kestrel) =>

    {

        var section = context.Configuration.GetSection("Kestrel");

        kestrel.Configure(section).Endpoint("Fiddler", endpoint => endpoint.ListenOptions.UseFiddler());

    });

    var app = builder.Build();

    app.UseRouting();

    app.UseFiddler();

    app.MapControllerRoute(

        name: "default",

        pattern: "{controller=Home}/{action=Index}/{id?}");

    app.Run();

}

10 留给读者

如果让您来开发个伪Fiddler，除了本文的方法，您会使用什么方式来开发呢？