Hessian Free Optimization——外国网友分享的“共轭梯度”的推导

外国网友分享的“共轭梯度”的推导：

https://andrew.gibiansky.com/blog/machine-learning/hessian-free-optimization/

=====================================

个人对共轭梯度法的理解是，这个算法是用来求解凸二次函数中二次项系数矩阵为正定的情况下最小值的，由于该种情况下函数的导数形式刚好为二次项系数矩阵的一次线性方程组的形式，因此共轭梯度法也可以看做是用来求系数矩阵为正定矩阵的一次线性方程组的求解算法。

从这个角度来看共轭梯度法可以参看：

python版本的“共轭梯度法”算法代码

Ax = b 的迭代解法 —— 共轭梯度 （算法步骤）

 共轭梯度法的标准算法流程：

外国网友给出了一种其他形式的推导，并给出了其算法流程：

https://andrew.gibiansky.com/blog/machine-learning/hessian-free-optimization/

PS: 标准流程便于计算，外国网友分享的计算流程便于理解，在计算本质上是一致和等价的。

=============================================

在外国网友的分享中有一个很重要的技术，就是hessian free，对于该技术本博之前也分享过，具体见：

系数矩阵为Hessian矩阵时使用“Pearlmutter trick”或“有限差分法”近似的共轭梯度解法 —— Hession-free的共轭梯度法

外国网友分享的Hessian Free技术其实是有限差分法，关于更多的hessian free技术可以参看：

https://www.cs.toronto.edu/~jmartens/docs/HF_book_chapter.pdf

给出外国网友分享的hessian free计算流程：（有限差分法）

该方法最大的优点是节省内存，因为不需要在内存中对Hessian矩阵展开，因为Hessian矩阵的size为n*n，而不展开的情况下size为n，这样的话对于一些Deep Learning下的神经网络参数也可以使用常见硬件配置的计算机进行计算；

该方法不仅使计算成为可能，同时在牺牲一定计算精度的情况下减少单次计算的时间，因为该种hessian free的计算方法放弃了二次求导而使用两次一次求导进行近似，这些需要注意的是对于非正定矩阵进行求解，即使是使用共轭梯度法也只是进行近似求解，所以该方法在这里对hessian矩阵的近似求解也是可以理解的。

关于共轭梯度法对于非正定矩阵的近似求解参见资料：

共轭梯度法对“正定矩阵”的求解与对“非正定矩阵”的求解的对比

 

 

本文给出hessian free共轭梯度法求解的精度与标准共轭梯度使用hessian矩阵方式下的求解精度对比：

import numpy as np
from scipy.linalg import orth
import torch

# torch版本的共轭梯度算法，device为CPU
def conjugate_gradient(f_Ax, b: torch.FloatTensor, cg_iters=10, residual_tol=1e-10):
    # assert isinstance(A, np.ndarray)
    # assert isinstance(b, np.ndarray)

    # r = np.copy(b)
    # p = np.copy(b)
    b = torch.squeeze(b)
    r = torch.clone(b)
    p = torch.clone(b)

    x = torch.zeros_like(b)
    rdotr = torch.dot(r, r)

    for i in range(cg_iters):
        # z = np.dot(A, p)
        z = f_Ax(p=p)
        # print(z)
        p_z = torch.dot(p, z)
        if p_z == 0:  # hessian free方法时会出现 z为零向量, 原因为p向量过小导致x的delta变化被忽略
            break
        v = rdotr / p_z
        # print(v)
        x += v * p
        # print(x)
        r -= v * z
        # print(r)
        newrdotr = torch.dot(r, r)
        # print(newrdotr)
        mu = newrdotr / rdotr
        # print(mu)
        p = r + mu * p
        # print(p)

        rdotr = newrdotr
        # print(i, rdotr, np.sqrt(np.mean(np.square(np.dot(A, x)-b))))
        print(i, rdotr.detach().numpy())
        if rdotr < residual_tol:
            break
    return x

#####################################################
class Data_Gen():
    def __init__(self, N, positive_definite=True):
        self.N = N

        # 不变的矩阵
        B = np.random.rand(N, 1)
        # 数据集，数据B矩阵
        self.B = torch.tensor(B, dtype=torch.float32)

        if positive_definite==True:
            # 正定矩阵的生成, A_
            X = np.diag(np.abs(np.random.rand(N))+1)
            U = orth(np.random.rand(N, N))
            A = np.dot(np.dot(U.T, X), U)
        else:
            # 非正定矩阵的生成，A
            M = np.random.rand(N, N)
            A = np.dot(M.T, M) + 0.01*np.eye(N)
        self.A = torch.tensor(A, dtype=torch.float32)

    def data_gen(self, x: torch.FloatTensor):
        """ 数据生成 """
        # 训练数据，x_data, x
        # x = torch.randn(N, requires_grad=True)
        N = x.size()[0]
        assert N==self.N, "向量长度不匹配"

        # 构建训练数据，非正定数据矩阵生成的y_data
        y=self.object_fun(self.A, self.B, x)
        return y

    def object_fun(self, A: torch.FloatTensor, B: torch.FloatTensor, x: torch.FloatTensor):
        if x.dim() == 1:
            x = torch.unsqueeze(x, 1)
        ans = torch.mm(x.T, torch.mm(A, x)) + torch.mm(x.T, B)
        # print( torch.mm(A, x) )
        # print( torch.mm(x.T, torch.mm(A, x)) )
        # print( torch.mm(x.T, B) )
        return torch.squeeze(ans)

#####################################################
########### 假设不知道矩阵A,B的具体数值形式
class F_Ax():
    def __init__(self, data_gen, free=False, model_based=True):
        self.free = free
        self.model_based = model_based

        self.data_gen = data_gen

        # x=0, f'
        x = torch.zeros(self.data_gen.N, requires_grad=True)
        y = self.data_gen.data_gen(x)
        self.b = -1.0*torch.autograd.grad(y, x, create_graph=False, retain_graph=False)[0]

        self.x = torch.randn(self.data_gen.N, requires_grad=True)
        self.y = self.data_gen.data_gen(self.x)

        if not free:
            if not model_based:
                self.first_grad = torch.autograd.grad(self.y, self.x, create_graph=True, retain_graph=True)[0]
            else:
                self.first_grad = None
        else:
            self.first_grad = torch.autograd.grad(self.y, self.x, create_graph=False, retain_graph=False)[0]

    def __call__(self, p):
        if not self.free:
            return self.f_Ax(p)
        else:
            return self.f_Ax_free(p)

    def f_Ax(self, p):
        if self.model_based:
            return torch.squeeze( torch.mm(2*self.data_gen.A, torch.unsqueeze(p, 1)) ).detach()
        else:
            tmp = torch.dot(self.first_grad, p)
            hessian_b = torch.autograd.grad(tmp, self.x, create_graph=False, retain_graph=True)[0]
            return hessian_b.detach()

    def f_Ax_free(self, p):
        delta = 0.0001
        x_delta = self.x + delta*p
        y_delta = self.data_gen.data_gen(x_delta)
        first_grad_delta = torch.autograd.grad(y_delta, x_delta, create_graph=False, retain_graph=False)[0]

        """
        print("p", p)
        print("x", self.x)
        print("x_delta", x_delta)
        print("y_delta", y_delta)
        print(first_grad_delta, self.first_grad)
        """
        hessian_b = (first_grad_delta - self.first_grad)/delta
        return hessian_b.detach()

if __name__ == '__main__':
    # 数据生成
    N = 500

    data_gen = Data_Gen(N)

    print("*"*30)
    # 共轭法解正定矩阵, 标准hessian矩阵求解
    f_Ax = F_Ax(data_gen, free=False, model_based=True)
    # A*x=b, x=0, r=b-A*x
    # f=xT*A*x+b*x, f'=2A*x+b=0, 2A*x=-b, r=-b-2A*x, x=0, r=-b
    obj_cg = conjugate_gradient(f_Ax, f_Ax.b, cg_iters=N)
    y_cg = data_gen.data_gen(torch.Tensor(obj_cg)).detach().numpy()
    print("cg_method / model_based:     ")
    # print("cg        x: ", obj_cg.detach().numpy())
    print("cg        y: ", y_cg)

    print("*"*30)
    # 共轭法解正定矩阵, 标准hessian矩阵求解
    f_Ax = F_Ax(data_gen, free=False, model_based=False)
    # A*x=b, x=0
    # f=xT*A*x+b*x, f'=2A*x+b=0, x=0, -f'=-b
    # obj_cg = conjugate_gradient(f_Ax, -1.0*f_Ax.data_gen.B.detach(), cg_iters=N)
    obj_cg = conjugate_gradient(f_Ax, f_Ax.b, cg_iters=N)
    y_cg = data_gen.data_gen(torch.Tensor(obj_cg)).detach().numpy()
    print("cg_method / no_model:     ")
    # print("cg        x: ", obj_cg.detach().numpy())
    print("cg        y: ", y_cg)

    print("*"*30)
    # 共轭法解正定矩阵, hessian free矩阵求解
    f_Ax = F_Ax(data_gen, free=True)
    obj_cg = conjugate_gradient(f_Ax, f_Ax.b, cg_iters=N)
    y_cg = data_gen.data_gen(torch.Tensor(obj_cg)).detach().numpy()
    print("cg_method / hessian free:     ")
    # print("cg        x: ", obj_cg.detach().numpy())
    print("cg        y: ", y_cg)

    print("*"*30)
    print("求逆矩阵方法:")
    obj_inv = np.squeeze( np.dot(np.linalg.inv(2*data_gen.A), -data_gen.B) )
    y_inv = data_gen.data_gen(torch.Tensor(obj_inv)).detach().numpy()
    # print("inv       x: ", obj_inv)
    print("inv       y: ", y_inv)

当变量维度为500时，N=500，运行结果如下：

******************************
0 6.151779
1 0.1959353
2 0.0062688473
3 0.00018499946
4 5.845999e-06
5 1.6695711e-07
6 4.435799e-09
7 1.4255e-10
8 3.8852853e-12
cg_method / model_based:
cg        y:  -31.02312
******************************
0 6.151779
1 0.19593522
2 0.006268844
3 0.00018499937
4 5.8459937e-06
5 1.6695697e-07
6 4.4357944e-09
7 1.4254992e-10
8 3.8852836e-12
cg_method / no_model:
cg        y:  -31.023113
******************************
0 6.1594234
1 0.20159078
2 0.008541295
3 0.0011287229
4 0.00068465277
5 0.00050850643
6 0.0004090694
7 0.00030704378
8 0.0002285685
9 0.00020425764
10 0.00019200041
11 0.00018489834
12 0.00018958737
13 0.00019306257
14 0.00020015262
15 0.00020086684
16 0.00019615004
17 0.00018864215
18 0.00018794968
19 0.00018813496
20 0.00018472853
21 0.0001813241
22 0.00017957346
23 0.0001774597
24 0.00017438357
25 0.00017113979
26 0.00016771276
27 0.0001648154
28 0.00016261148
29 0.0001646318
30 0.00016398162
31 0.00016432792
32 0.00016693369
33 0.00016853935
34 0.00017149383
35 0.00017309244
36 0.00017486836
37 0.00017641405
38 0.00017831716
39 0.00018032563
40 0.00018177755
41 0.00018389904
42 0.00018667165
43 0.00018768277
44 0.00018915892
45 0.00018997813
46 0.00019037939
47 0.00019211981
48 0.0001927464
49 0.00019338286
50 0.0001943383
51 0.00019452666
52 0.00019514289
53 0.00019481007
54 0.00019525306
55 0.00019617433
56 0.00019688645
57 0.0001975983
58 0.00019843293
59 0.00019926215
60 0.00020034573
61 0.00020137688
62 0.00020215649
63 0.00020300435
64 0.00020388639
65 0.00020509919
66 0.00020629674
67 0.0002069474
68 0.00020748249
69 0.00020758687
70 0.00020794242
71 0.00020868226
72 0.00020837913
73 0.00020850706
74 0.00020821663
75 0.00020762766
76 0.00020737154
77 0.00020726079
78 0.00020692014
79 0.00020747946
80 0.00020765557
81 0.00020788841
82 0.0002085683
83 0.00020897665
84 0.0002098172
85 0.00021086376
86 0.00021200915
87 0.00021282284
88 0.00021389453
89 0.00021519943
90 0.00021696577
91 0.00021785928
92 0.0002191308
93 0.00022065573
94 0.00022225926
95 0.00022364585
96 0.00022458918
97 0.00022542216
98 0.0002262149
99 0.00022661831
100 0.00022701544
101 0.00022735083
102 0.00022750269
103 0.00022724541
104 0.00022713674
105 0.00022665656
106 0.00022589233
107 0.00022527316
108 0.00022477424
109 0.0002243446
110 0.00022396893
111 0.00022360984
112 0.00022310254
113 0.00022258243
114 0.00022253316
115 0.00022248052
116 0.00022306429
117 0.00022385173
118 0.0002247648
119 0.00022542346
120 0.0002262611
121 0.00022674265
122 0.00022716755
123 0.0002277162
124 0.00022807892
125 0.00022870538
126 0.00022907302
127 0.00022950309
128 0.00022931909
129 0.00022937861
130 0.00022934753
131 0.00022937884
132 0.00022908638
133 0.00022879281
134 0.00022841981
135 0.00022788359
136 0.00022733235
137 0.00022680746
138 0.00022609474
139 0.00022585233
140 0.00022543827
141 0.00022525541
142 0.00022515916
143 0.00022509988
144 0.00022517858
145 0.00022535844
146 0.0002253043
147 0.00022527447
148 0.0002253766
149 0.00022560984
150 0.00022640392
151 0.00022702795
152 0.00022778369
153 0.00022822978
154 0.00022852901
155 0.00022916793
156 0.00022977224
157 0.0002305794
158 0.00023089546
159 0.00023147257
160 0.00023170623
161 0.00023212912
162 0.00023236318
163 0.00023269719
164 0.00023297901
165 0.00023317966
166 0.00023331103
167 0.0002336496
168 0.0002338668
169 0.000233995
170 0.00023398726
171 0.00023416503
172 0.00023406725
173 0.00023389464
174 0.00023356493
175 0.00023326468
176 0.00023257232
177 0.00023218279
178 0.00023180687
179 0.00023131882
180 0.00023098273
181 0.00023061676
182 0.00023021865
183 0.00022970044
184 0.0002292027
185 0.00022876884
186 0.00022848867
187 0.00022810115
188 0.00022782276
189 0.00022744815
190 0.0002271328
191 0.00022669628
192 0.00022637927
193 0.00022632389
194 0.00022638107
195 0.00022631523
196 0.0002264069
197 0.0002265512
198 0.00022678994
199 0.00022720282
200 0.00022759449
201 0.00022788282
202 0.00022839486
203 0.00022865746
204 0.00022866378
205 0.0002287827
206 0.000228774
207 0.00022895617
208 0.00022895684
209 0.00022892051
210 0.00022892517
211 0.0002290561
212 0.00022909214
213 0.00022911649
214 0.00022881923
215 0.0002285918
216 0.00022841917
217 0.000228261
218 0.00022806498
219 0.00022781544
220 0.0002276793
221 0.00022742781
222 0.00022716969
223 0.00022697583
224 0.00022696472
225 0.00022695411
226 0.00022695356
227 0.00022687836
228 0.00022700147
229 0.00022710013
230 0.00022718502
231 0.00022741966
232 0.00022780802
233 0.00022799062
234 0.0002282761
235 0.0002286889
236 0.00022908242
237 0.00022946045
238 0.000229987
239 0.00023043984
240 0.00023087315
241 0.00023123738
242 0.00023162122
243 0.00023205292
244 0.00023240264
245 0.00023265317
246 0.00023275155
247 0.00023276746
248 0.00023289626
249 0.0002328227
250 0.00023284531
251 0.00023289092
252 0.00023288638
253 0.00023292993
254 0.00023277476
255 0.00023273577
256 0.0002325469
257 0.00023237546
258 0.00023226121
259 0.00023223204
260 0.00023238528
261 0.00023231853
262 0.00023216866
263 0.00023196667
264 0.00023165485
265 0.00023142993
266 0.00023110787
267 0.00023055883
268 0.0002301863
269 0.0002298143
270 0.00022944534
271 0.00022903144
272 0.00022872834
273 0.00022840456
274 0.00022804341
275 0.00022768846
276 0.00022722551
277 0.0002268515
278 0.00022632812
279 0.00022571546
280 0.00022509633
281 0.00022459165
282 0.00022399866
283 0.00022338965
284 0.00022284754
285 0.00022233489
286 0.00022185949
287 0.0002215319
288 0.0002212327
289 0.00022082646
290 0.00022054365
291 0.00022037246
292 0.00022011013
293 0.00021993449
294 0.00021978612
295 0.00021961654
296 0.00021943254
297 0.00021921389
298 0.00021913109
299 0.00021905085
300 0.00021882472
301 0.00021863113
302 0.00021839591
303 0.000218173
304 0.0002179669
305 0.00021783324
306 0.00021764103
307 0.0002174809
308 0.00021737436
309 0.00021735577
310 0.00021724831
311 0.00021728688
312 0.00021717863
313 0.00021710606
314 0.00021712459
315 0.00021708765
316 0.00021707293
317 0.0002170371
318 0.00021704563
319 0.00021713655
320 0.00021723637
321 0.00021734701
322 0.00021742153
323 0.00021763307
324 0.00021779806
325 0.00021785415
326 0.00021802122
327 0.00021812455
328 0.00021822259
329 0.000218284
330 0.0002184184
331 0.00021854613
332 0.00021864491
333 0.00021863607
334 0.00021875897
335 0.00021889256
336 0.0002190354
337 0.0002191391
338 0.00021934311
339 0.00021944038
340 0.00021962303
341 0.0002197267
342 0.00021990185
343 0.0002201355
344 0.00022040689
345 0.00022053893
346 0.00022065497
347 0.00022085774
348 0.00022095154
349 0.00022111065
350 0.00022134071
351 0.000221422
352 0.00022161622
353 0.00022178215
354 0.00022199095
355 0.00022210032
356 0.00022221604
357 0.00022222067
358 0.00022226431
359 0.00022228237
360 0.00022236252
361 0.00022244663
362 0.00022249873
363 0.00022254925
364 0.00022256386
365 0.0002226119
366 0.00022273415
367 0.00022286046
368 0.0002229963
369 0.00022319847
370 0.00022340492
371 0.00022348411
372 0.00022364657
373 0.00022371435
374 0.00022369849
375 0.00022371346
376 0.00022368776
377 0.00022350838
378 0.00022344757
379 0.00022339853
380 0.00022334023
381 0.00022315964
382 0.00022306487
383 0.00022290435
384 0.00022267574
385 0.00022244093
386 0.00022225216
387 0.00022197029
388 0.00022160012
389 0.00022122069
390 0.00022077147
391 0.00022033765
392 0.00021997867
393 0.00021951078
394 0.00021902549
395 0.00021855644
396 0.00021808066
397 0.00021768955
398 0.00021724944
399 0.00021672025
400 0.0002161864
401 0.00021571867
402 0.00021525045
403 0.00021470065
404 0.00021407509
405 0.00021344144
406 0.00021270712
407 0.00021199684
408 0.00021130784
409 0.00021086683
410 0.00021030851
411 0.00020983213
412 0.0002093105
413 0.00020882132
414 0.00020827653
415 0.00020781314
416 0.00020738214
417 0.00020698794
418 0.00020665792
419 0.00020626359
420 0.00020604793
421 0.00020578166
422 0.00020556655
423 0.00020533871
424 0.00020520785
425 0.00020509795
426 0.00020502004
427 0.00020514321
428 0.00020523198
429 0.0002053542
430 0.00020552002
431 0.0002057385
432 0.0002059575
433 0.00020618163
434 0.00020629048
435 0.00020642139
436 0.0002065987
437 0.0002066293
438 0.00020672739
439 0.00020681194
440 0.00020691814
441 0.00020700257
442 0.0002071764
443 0.0002072244
444 0.00020728902
445 0.00020741299
446 0.00020754982
447 0.00020762914
448 0.00020773818
449 0.0002078797
450 0.000208029
451 0.00020821225
452 0.00020833641
453 0.00020844635
454 0.00020863336
455 0.00020878832
456 0.00020892094
457 0.00020911955
458 0.00020935797
459 0.00020949241
460 0.00020960439
461 0.0002096837
462 0.00020979834
463 0.00020988252
464 0.00020993745
465 0.00020993358
466 0.00020996267
467 0.0002099606
468 0.00020997575
469 0.00021000754
470 0.00021001714
471 0.00020990241
472 0.00020982145
473 0.00020975285
474 0.00020967293
475 0.00020966539
476 0.00020957882
477 0.00020952235
478 0.00020946964
479 0.00020944391
480 0.00020944673
481 0.00020942779
482 0.00020946003
483 0.00020952446
484 0.00020957571
485 0.0002096438
486 0.0002097213
487 0.00020974988
488 0.00020970467
489 0.00020982334
490 0.00020990153
491 0.00020989559
492 0.00020998485
493 0.00021000508
494 0.00020998938
495 0.00021000604
496 0.00021002302
497 0.00021007683
498 0.0002101883
499 0.00021017529
cg_method / hessian free:
cg        y:  -31.011902
******************************
求逆矩阵方法:
inv       y:  -31.023113

当变量维度为10时，N=10，运行结果如下：

******************************
0 0.016065279
1 0.00059065747
2 1.0640101e-05
3 9.062602e-08
4 3.834896e-09
5 9.851212e-12
cg_method / model_based:     
cg        y:  -0.3055587
******************************
0 0.016065277
1 0.00059065747
2 1.0640104e-05
3 9.0626116e-08
4 3.834903e-09
5 9.851191e-12
cg_method / no_model:     
cg        y:  -0.3055587
******************************
0 0.016052932
1 0.0006164954
2 1.6439779e-05
3 5.5340865e-07
4 7.641636e-07
5 5.641166e-07
6 3.3871862e-07
7 2.0135943e-07
8 2.2088993e-07
9 1.9546216e-07
cg_method / hessian free:     
cg        y:  -0.30555704
******************************
求逆矩阵方法:
inv       y:  -0.30555865

可以看到使用hessian
free方法求解的精度要差于标准共轭梯度，标准共轭梯度求得的是精确解，而使用hessian
free方法求得的是近似解。从上面的运行结果中我们可以看到，使用hessian
free方法后共轭梯度迭代10次左右就收敛了，这个表现和不使用hessian free方法时是相差不多的，而且从近似精度上来看使用hessian
free方法也是表现不错的。

上面的运行代码使用的是正定矩阵，如果我们使用非正定矩阵表现是怎么样的，给出下面代码：

import numpy as np
from scipy.linalg import orth
import torch

# torch版本的共轭梯度算法，device为CPU
def conjugate_gradient(f_Ax, b: torch.FloatTensor, cg_iters=10, residual_tol=1e-10):
    # assert isinstance(A, np.ndarray)
    # assert isinstance(b, np.ndarray)

    # r = np.copy(b)
    # p = np.copy(b)
    b = torch.squeeze(b)
    r = torch.clone(b)
    p = torch.clone(b)

    x = torch.zeros_like(b)
    rdotr = torch.dot(r, r)

    for i in range(cg_iters):
        # z = np.dot(A, p)
        z = f_Ax(p=p)
        # print(z)
        p_z = torch.dot(p, z)
        if p_z == 0:  # hessian free方法时会出现 z为零向量, 原因为p向量过小导致x的delta变化被忽略
            break
        v = rdotr / p_z
        # print(v)
        x += v * p
        # print(x)
        r -= v * z
        # print(r)
        newrdotr = torch.dot(r, r)
        # print(newrdotr)
        mu = newrdotr / rdotr
        # print(mu)
        p = r + mu * p
        # print(p)

        rdotr = newrdotr
        # print(i, rdotr, np.sqrt(np.mean(np.square(np.dot(A, x)-b))))
        print(i, rdotr.detach().numpy())
        if rdotr < residual_tol:
            break
    return x

#####################################################
class Data_Gen():
    def __init__(self, N, positive_definite=True):
        self.N = N

        # 不变的矩阵
        B = np.random.rand(N, 1)
        # 数据集，数据B矩阵
        self.B = torch.tensor(B, dtype=torch.float32)

        if positive_definite==True:
            # 正定矩阵的生成, A_
            X = np.diag(np.abs(np.random.rand(N))+1)
            U = orth(np.random.rand(N, N))
            A = np.dot(np.dot(U.T, X), U)
        else:
            # 非正定矩阵的生成，A
            M = np.random.rand(N, N)
            A = np.dot(M.T, M) + 0.01*np.eye(N)
        self.A = torch.tensor(A, dtype=torch.float32)

    def data_gen(self, x: torch.FloatTensor):
        """ 数据生成 """
        # 训练数据，x_data, x
        # x = torch.randn(N, requires_grad=True)
        N = x.size()[0]
        assert N==self.N, "向量长度不匹配"

        # 构建训练数据，非正定数据矩阵生成的y_data
        y=self.object_fun(self.A, self.B, x)
        return y

    def object_fun(self, A: torch.FloatTensor, B: torch.FloatTensor, x: torch.FloatTensor):
        if x.dim() == 1:
            x = torch.unsqueeze(x, 1)
        ans = torch.mm(x.T, torch.mm(A, x)) + torch.mm(x.T, B)
        # print( torch.mm(A, x) )
        # print( torch.mm(x.T, torch.mm(A, x)) )
        # print( torch.mm(x.T, B) )
        return torch.squeeze(ans)

#####################################################
########### 假设不知道矩阵A,B的具体数值形式
class F_Ax():
    def __init__(self, data_gen, free=False, model_based=True):
        self.free = free
        self.model_based = model_based

        self.data_gen = data_gen

        # x=0, f'
        x = torch.zeros(self.data_gen.N, requires_grad=True)
        y = self.data_gen.data_gen(x)
        self.b = -1.0*torch.autograd.grad(y, x, create_graph=False, retain_graph=False)[0]

        self.x = torch.randn(self.data_gen.N, requires_grad=True)
        self.y = self.data_gen.data_gen(self.x)

        if not free:
            if not model_based:
                self.first_grad = torch.autograd.grad(self.y, self.x, create_graph=True, retain_graph=True)[0]
            else:
                self.first_grad = None
        else:
            self.first_grad = torch.autograd.grad(self.y, self.x, create_graph=False, retain_graph=False)[0]

    def __call__(self, p):
        if not self.free:
            return self.f_Ax(p)
        else:
            return self.f_Ax_free(p)

    def f_Ax(self, p):
        if self.model_based:
            return torch.squeeze( torch.mm(2*self.data_gen.A, torch.unsqueeze(p, 1)) ).detach()
        else:
            tmp = torch.dot(self.first_grad, p)
            hessian_b = torch.autograd.grad(tmp, self.x, create_graph=False, retain_graph=True)[0]
            return hessian_b.detach()

    def f_Ax_free(self, p):
        delta = 0.0001
        x_delta = self.x + delta*p
        y_delta = self.data_gen.data_gen(x_delta)
        first_grad_delta = torch.autograd.grad(y_delta, x_delta, create_graph=False, retain_graph=False)[0]

        """
        print("p", p)
        print("x", self.x)
        print("x_delta", x_delta)
        print("y_delta", y_delta)
        print(first_grad_delta, self.first_grad)
        """
        hessian_b = (first_grad_delta - self.first_grad)/delta
        return hessian_b.detach()

if __name__ == '__main__':
    # 数据生成
    N = 500

    data_gen = Data_Gen(N, positive_definite=False)

    print("*"*30)
    # 共轭法解正定矩阵, 标准hessian矩阵求解
    f_Ax = F_Ax(data_gen, free=False, model_based=True)
    # A*x=b, x=0, r=b-A*x
    # f=xT*A*x+b*x, f'=2A*x+b=0, 2A*x=-b, r=-b-2A*x, x=0, r=-b
    obj_cg = conjugate_gradient(f_Ax, f_Ax.b, cg_iters=N)
    y_cg = data_gen.data_gen(torch.Tensor(obj_cg)).detach().numpy()
    print("cg_method / model_based:     ")
    # print("cg        x: ", obj_cg.detach().numpy())
    print("cg        y: ", y_cg)

    print("*"*30)
    # 共轭法解正定矩阵, 标准hessian矩阵求解
    f_Ax = F_Ax(data_gen, free=False, model_based=False)
    # A*x=b, x=0
    # f=xT*A*x+b*x, f'=2A*x+b=0, x=0, -f'=-b
    # obj_cg = conjugate_gradient(f_Ax, -1.0*f_Ax.data_gen.B.detach(), cg_iters=N)
    obj_cg = conjugate_gradient(f_Ax, f_Ax.b, cg_iters=N)
    y_cg = data_gen.data_gen(torch.Tensor(obj_cg)).detach().numpy()
    print("cg_method / no_model:     ")
    # print("cg        x: ", obj_cg.detach().numpy())
    print("cg        y: ", y_cg)

    print("*"*30)
    # 共轭法解正定矩阵, hessian free矩阵求解
    f_Ax = F_Ax(data_gen, free=True)
    obj_cg = conjugate_gradient(f_Ax, f_Ax.b, cg_iters=N)
    y_cg = data_gen.data_gen(torch.Tensor(obj_cg)).detach().numpy()
    print("cg_method / hessian free:     ")
    # print("cg        x: ", obj_cg.detach().numpy())
    print("cg        y: ", y_cg)

    print("*"*30)
    print("求逆矩阵方法:")
    obj_inv = np.squeeze( np.dot(np.linalg.inv(2*data_gen.A), -data_gen.B) )
    y_inv = data_gen.data_gen(torch.Tensor(obj_inv)).detach().numpy()
    # print("inv       x: ", obj_inv)
    print("inv       y: ", y_inv)

=======================================================

根据下面的运算表现可以知道，对于非正定矩阵，使用标准的共轭梯度法依然可以获得较高近似度的解，但是如果变量维度过高，那么使用hessian free方法所获得的解会与真实解有较大距离。

----------------------------------------------------

当变量维度为500时，N=500，运行结果如下：

******************************
0 57.371216
1 42.43196
2 49.113327
3 9233.965
4 44.62644
5 42.043587
6 41.741745
7 12164.323
8 37.046265
9 37.87662
10 11593.292
11 34.09523
12 34.291733
13 6389.4863
14 36.66685
15 38.191536
16 9121.264
17 46.320816
18 40.81516
19 15581.823
20 43.27904
21 43.656227
22 5370.096
23 40.599594
24 41.988503
25 607.96985
26 36.300903
27 50.90444
28 125.86273
29 32.25437
30 127.32823
31 50.140793
32 37.194588
33 714.8141
34 37.47375
35 35.19345
36 6714.643
37 36.96501
38 39.88392
39 12462.299
40 34.7416
41 36.54345
42 211.86272
43 30.307438
44 486.33954
45 31.100868
46 27.131727
47 10970.497
48 27.282417
49 36.308456
50 224.7307
51 34.707664
52 1024.9998
53 35.103043
54 31.385048
55 9184.724
56 27.037994
57 31.179802
58 299.6322
59 26.421516
60 150.70734
61 28.612602
62 26.645443
63 5669.1367
64 28.377247
65 30.440868
66 206.117
67 34.562317
68 31.231052
69 36.751144
70 498.77646
71 31.555225
72 31.528282
73 1732.9908
74 31.635553
75 37.480064
76 72.124825
77 26.511412
78 399.75317
79 28.96152
80 25.901936
81 5670.7637
82 24.753803
83 30.679188
84 7146.023
85 35.409176
86 36.58112
87 900.1215
88 37.09934
89 74.98508
90 67.57076
91 34.80877
92 1096.9839
93 35.54541
94 32.89406
95 13053.612
96 33.1331
97 174.61107
98 37.53117
99 29.394775
100 2593.868
101 22.712702
102 866.7932
103 22.352226
104 20.735903
105 364.9077
106 20.453346
107 2670.6245
108 18.690022
109 24.300114
110 98.09103
111 17.302582
112 5844.871
113 16.669
114 32.17601
115 39.915466
116 18.897827
117 6653.461
118 17.182903
119 67.12341
120 17.205257
121 12.608309
122 1295.8981
123 12.367279
124 608.64954
125 12.227095
126 12.833444
127 152.50052
128 13.718049
129 2183.2085
130 13.025301
131 17.412449
132 39.50153
133 10.405979
134 4340.6846
135 10.605988
136 89.489624
137 10.823059
138 10.380878
139 612.3274
140 10.6352825
141 847.55005
142 11.884264
143 13.262777
144 74.67858
145 12.454906
146 4326.6924
147 11.381798
148 83.836296
149 14.0325985
150 13.647509
151 438.73932
152 14.914915
153 3862.6006
154 16.258652
155 36.66799
156 34.494347
157 16.455229
158 4014.5796
159 16.042141
160 426.7125
161 13.140771
162 12.13816
163 82.04869
164 10.020496
165 3491.6216
166 10.006816
167 66.46086
168 9.455457
169 8.987829
170 197.23499
171 8.170771
172 2740.0393
173 7.2035484
174 35.889797
175 11.034092
176 7.71358
177 463.32703
178 6.107677
179 1076.6904
180 6.2761593
181 10.350861
182 12.974571
183 4.990115
184 1088.4124
185 4.530218
186 227.14127
187 5.36735
188 6.623748
189 29.466856
190 4.844181
191 2057.6934
192 4.7237177
193 72.389114
194 4.6482935
195 4.6448154
196 44.084465
197 3.9048011
198 1553.9905
199 4.2231574
200 35.759598
201 5.620123
202 4.7377234
203 124.70429
204 4.0475044
205 1496.6033
206 4.4871387
207 17.787554
208 5.863317
209 4.2702637
210 250.15576
211 3.5596018
212 818.5853
213 3.5654426
214 10.833195
215 6.253158
216 3.5712457
217 388.4282
218 3.1228046
219 454.59515
220 2.8857603
221 7.7618494
222 5.138753
223 3.362376
224 336.83905
225 2.47856
226 210.65569
227 2.3652039
228 3.6693823
229 6.2841935
230 2.1106977
231 483.67023
232 1.7347314
233 92.92268
234 1.9320685
235 1.8249925
236 6.924918
237 1.3937067
238 459.78577
239 1.2635496
240 26.794218
241 1.1684456
242 1.2088621
243 9.34343
244 0.96714586
245 330.99146
246 0.6894243
247 8.986591
248 0.6322347
249 0.67571366
250 6.7460027
251 0.66867614
252 257.2688
253 0.61759627
254 7.3272877
255 0.630314
256 0.5871036
257 6.16669
258 0.5482025
259 211.99046
260 0.5725684
261 6.094777
262 0.621442
263 0.6649758
264 10.237093
265 0.4972903
266 179.312
267 0.41816264
268 4.2326736
269 0.5116682
270 0.5660416
271 8.762831
272 0.55657125
273 218.64615
274 0.6004535
275 5.0130496
276 0.7167789
277 0.6326715
278 17.963165
279 0.5116113
280 197.83405
281 0.58591396
282 3.3662472
283 0.71794224
284 0.5303303
285 15.147354
286 0.49290556
287 146.09183
288 0.5294705
289 1.8536817
290 0.7688271
291 0.4796582
292 36.97225
293 0.433975
294 61.64202
295 0.38384578
296 0.7233697
297 0.6152241
298 0.36418617
299 57.319656
300 0.34480774
301 23.166193
302 0.3640464
303 0.51601815
304 1.6914914
305 0.35271323
306 114.82585
307 0.34735107
308 10.814686
309 0.34138978
310 0.37718928
311 2.598002
312 0.28617314
313 133.65218
314 0.3155054
315 4.270967
316 0.41884428
317 0.33317864
318 5.638303
319 0.35180175
320 115.36754
321 0.36156896
322 0.8339474
323 0.4011857
324 0.24910875
325 30.5554
326 0.23861426
327 13.958509
328 0.26277548
329 0.2928189
330 1.0171593
331 0.2191773
332 69.136955
333 0.191293
334 2.168476
335 0.2134876
336 0.17124265
337 4.563777
338 0.16150463
339 34.192673
340 0.1546121
341 0.35904467
342 0.26456624
343 0.13815959
344 18.052288
345 0.13946208
346 7.907863
347 0.14082703
348 0.16767536
349 0.4373564
350 0.1436337
351 43.84839
352 0.17090033
353 3.8326807
354 0.19157273
355 0.17065394
356 2.2333546
357 0.16909403
358 58.95788
359 0.18377164
360 0.5844439
361 0.21017438
362 0.17259704
363 14.715322
364 0.15398389
365 9.29929
366 0.12532444
367 0.20103242
368 0.37337878
369 0.1377285
370 34.224236
371 0.09743266
372 2.3440795
373 0.09079773
374 0.117995344
375 0.8999311
376 0.11457223
377 37.952515
378 0.08485417
379 0.5390755
380 0.08956204
381 0.082301795
382 3.7527392
383 0.07473669
384 6.7779655
385 0.064136535
386 0.0877499
387 0.16982937
388 0.055582482
389 16.453403
390 0.046009652
391 0.60342
392 0.047630556
393 0.050603732
394 1.8001161
395 0.041113306
396 9.416584
397 0.04037293
398 0.11374679
399 0.092441946
400 0.051466033
401 10.792936
402 0.040342323
403 1.2618771
404 0.0354492
405 0.041137896
406 0.45535958
407 0.03216341
408 9.917621
409 0.026359893
410 0.079639435
411 0.045430947
412 0.02401809
413 3.547625
414 0.020955358
415 1.435612
416 0.021705065
417 0.030111827
418 0.12524128
419 0.024692316
420 9.536298
421 0.022540215
422 0.16479623
423 0.02428196
424 0.018922588
425 0.6186556
426 0.015874157
427 2.4874916
428 0.014601104
429 0.023875695
430 0.027871002
431 0.012520948
432 2.3194923
433 0.011306131
434 0.9893346
435 0.01502161
436 0.015478085
437 0.07553321
438 0.010098346
439 3.9474933
440 0.010396894
441 0.12778395
442 0.010620081
443 0.009923747
444 0.27152643
445 0.009015486
446 1.2834467
447 0.006521443
448 0.014216543
449 0.01549989
450 0.008812496
451 2.061974
452 0.008241267
453 0.16674978
454 0.009408826
455 0.008896973
456 0.16668975
457 0.008814862
458 3.6771533
459 0.010615008
460 0.027656462
461 0.016421294
462 0.008200899
463 1.2704537
464 0.0069025364
465 0.4641803
466 0.007239921
467 0.010220282
468 0.028847128
469 0.006827265
470 2.1826324
471 0.006788359
472 0.11764765
473 0.0063122157
474 0.006407187
475 0.06274532
476 0.0060631335
477 1.9126221
478 0.0046681827
479 0.029613854
480 0.005836845
481 0.005395731
482 0.2760051
483 0.0056133666
484 0.93803513
485 0.006185229
486 0.008173183
487 0.01548765
488 0.0054143434
489 1.3272967
490 0.0058620363
491 0.08215591
492 0.0048236037
493 0.0056010927
494 0.09436391
495 0.0044144248
496 1.2184632
497 0.0034422847
498 0.014386739
499 0.00566003
cg_method / model_based:
cg        y:  -19.127024
******************************
0 57.371223
1 42.43197
2 51.22411
3 1059.0156
4 44.620792
5 42.7191
6 2450.0435
7 41.66584
8 37.067318
9 14167.739
10 37.834076
11 34.14937
12 9889.629
13 34.301445
14 36.667355
15 6388.578
16 38.19957
17 47.73807
18 1341.5471
19 40.774994
20 53.849915
21 218.5963
22 43.36254
23 131.51312
24 58.617256
25 39.327965
26 608.57434
27 38.553722
28 36.298244
29 4143.3623
30 32.4692
31 36.06945
32 13770.34
33 37.202545
34 38.60314
35 459.2127
36 35.1983
37 211.01956
38 44.539062
39 39.82486
40 7714.2705
41 34.843925
42 31.233292
43 11402.922
44 30.328476
45 30.092632
46 1026.2545
47 27.112253
48 70.600334
49 44.359917
50 31.311064
51 2887.0698
52 35.109306
53 34.07573
54 10755.587
55 31.418066
56 61.91002
57 48.50474
58 28.602535
59 8733.256
60 26.727545
61 29.813221
62 151.15991
63 27.895721
64 2536.811
65 29.505714
66 33.86497
67 1152.2627
68 30.412575
69 674.7414
70 32.576195
71 33.00013
72 3744.8718
73 33.46144
74 136.72961
75 36.03151
76 29.31876
77 12001.27
78 24.74346
79 36.12587
80 94.303856
81 25.964739
82 3065.541
83 25.296997
84 24.91758
85 997.1979
86 30.442436
87 514.84753
88 37.74727
89 34.940453
90 9565.8955
91 36.482643
92 40.298676
93 294.4956
94 34.997524
95 262.96497
96 39.831425
97 33.26039
98 7290.383
99 33.439213
100 32.52797
101 650.2897
102 29.167843
103 480.74213
104 23.932507
105 22.283728
106 4767.671
107 20.369081
108 76.45894
109 29.415718
110 19.138018
111 7378.057
112 21.253744
113 22.353985
114 93.91083
115 16.740574
116 1040.4496
117 16.514744
118 18.66809
119 1477.9126
120 16.99815
121 88.39746
122 16.154816
123 13.887348
124 4560.5215
125 13.237496
126 13.284217
127 1515.1433
128 11.175037
129 12.266822
130 2692.561
131 11.765138
132 11.71522
133 2559.3901
134 10.25153
135 28.411436
136 16.443775
137 9.148363
138 3414.8984
139 9.915588
140 12.782868
141 65.91678
142 12.108868
143 1004.2977
144 12.13443
145 13.125297
146 541.6792
147 11.642474
148 142.88364
149 13.363283
150 13.51931
151 3902.8345
152 16.214699
153 44.5099
154 23.668518
155 17.214182
156 6572.271
157 15.968173
158 65.87512
159 21.403297
160 14.139772
161 2157.4717
162 11.578924
163 147.13983
164 11.64844
165 9.129951
166 423.961
167 8.154768
168 1110.613
169 9.206502
170 12.226636
171 41.975037
172 7.635691
173 2875.999
174 7.1684914
175 41.888092
176 8.701086
177 6.28242
178 374.52856
179 6.360502
180 399.24094
181 5.5658407
182 5.757576
183 29.82484
184 4.5015936
185 2040.8219
186 5.241243
187 26.083698
188 6.881133
189 4.8935947
190 626.76965
191 4.740071
192 215.47691
193 4.467394
194 4.729203
195 39.619446
196 3.9975448
197 1166.8987
198 4.372852
199 7.3086357
200 14.235489
201 4.6080246
202 1878.2769
203 4.179776
204 23.997147
205 5.7605085
206 4.2785506
207 535.33997
208 4.11594
209 157.92331
210 3.8309636
211 4.1166477
212 53.31669
213 3.875993
214 723.5919
215 3.3694944
216 5.24083
217 7.7533884
218 3.4986176
219 968.1735
220 3.2635212
221 28.313034
222 3.2224684
223 2.4893765
224 74.42895
225 2.3914623
226 337.36102
227 2.2918487
228 2.890963
229 6.5813847
230 1.8973963
231 637.9188
232 2.091194
233 6.226073
234 1.7485001
235 1.3139606
236 144.09715
237 1.2431841
238 57.37037
239 1.1632528
240 1.1666673
241 13.014399
242 0.970096
243 228.7634
244 0.6901866
245 1.2667835
246 1.0966264
247 0.61213475
248 153.3657
249 0.6583272
250 11.368039
251 0.64358044
252 0.6082193
253 14.094596
254 0.54672074
255 107.57715
256 0.5626725
257 0.8399772
258 1.7523139
259 0.5581782
260 217.64392
261 0.6206129
262 2.6776602
263 0.61760825
264 0.43658066
265 46.434143
266 0.47426462
267 18.921227
268 0.54809093
269 0.59893405
270 8.11241
271 0.60979855
272 120.5137
273 0.635476
274 0.7557299
275 2.2042289
276 0.51040334
277 222.49942
278 0.6384516
279 1.2727977
280 1.3557501
281 0.56413186
282 209.3699
283 0.5004739
284 1.2576283
285 0.80910313
286 0.54927063
287 175.19147
288 0.4893755
289 2.4949377
290 0.519655
291 0.4360546
292 41.598553
293 0.3562215
294 12.590177
295 0.3252389
296 0.3951174
297 4.6180706
298 0.37512308
299 68.71782
300 0.36447987
301 0.5922743
302 1.0403686
303 0.36344397
304 140.48743
305 0.3592863
306 1.2824438
307 0.37558603
308 0.2950529
309 44.397026
310 0.33056277
311 4.707982
312 0.31934032
313 0.3094751
314 11.046141
315 0.350829
316 20.777336
317 0.36166185
318 0.28142592
319 4.611032
320 0.25351968
321 22.411901
322 0.25147188
323 0.2871975
324 2.8330402
325 0.24786332
326 43.867897
327 0.22837985
328 0.2101911
329 1.1711346
330 0.20192714
331 42.69105
332 0.16435277
333 0.18527699
334 1.2119169
335 0.15329835
336 19.90199
337 0.15102267
338 0.14530149
339 2.3170688
340 0.13884434
341 6.6566944
342 0.1425708
343 0.1435384
344 6.9724693
345 0.15591002
346 1.7096281
347 0.16872336
348 0.17515674
349 30.083843
350 0.15448192
351 1.0541046
352 0.20007984
353 0.178236
354 55.19296
355 0.15688151
356 0.37687534
357 0.29131877
358 0.14209029
359 58.37217
360 0.12264069
361 0.25812936
362 0.293176
363 0.14391035
364 32.678745
365 0.096594244
366 0.11855102
367 0.41176468
368 0.10432087
369 16.619432
370 0.11109008
371 0.10059624
372 0.83900416
373 0.08671349
374 9.959917
375 0.07682974
376 0.079354584
377 0.99964803
378 0.063817084
379 5.0534987
380 0.06415294
381 0.057670753
382 1.2942812
383 0.04751928
384 2.5522933
385 0.046066426
386 0.04693884
387 1.6273199
388 0.046240993
389 1.6543308
390 0.04517761
391 0.048616864
392 1.9257953
393 0.053839616
394 1.4603746
395 0.050288036
396 0.034181096
397 1.5692112
398 0.029108971
399 1.9596858
400 0.03181171
401 0.0320099
402 0.8902955
403 0.0264588
404 1.1817664
405 0.024589386
406 0.0231646
407 0.762828
408 0.025768116
409 0.65508056
410 0.024979718
411 0.02585225
412 1.6701188
413 0.022459162
414 0.5620828
415 0.020824142
416 0.021982482
417 1.2402222
418 0.016976696
419 0.2673221
420 0.014680858
421 0.014917485
422 1.5871772
423 0.013545474
424 0.11113019
425 0.01443293
426 0.010861938
427 1.8431759
428 0.012330004
429 0.09966482
430 0.013426252
431 0.0098179085
432 2.471009
433 0.010849502
434 0.03923423
435 0.010917233
436 0.008395879
437 2.3435345
438 0.0063952086
439 0.032042127
440 0.010311498
441 0.008191502
442 3.00426
443 0.007845624
444 0.020931391
445 0.016659744
446 0.00865793
447 3.4188638
448 0.009570595
449 0.014005154
450 0.05696491
451 0.009767905
452 1.1955546
453 0.0087552555
454 0.0077483915
455 0.10483812
456 0.006531326
457 0.72049165
458 0.008480515
459 0.0077986484
460 0.2714106
461 0.0067993524
462 0.12768242
463 0.006809288
464 0.006137648
465 0.9953808
466 0.005416801
467 0.032670315
468 0.006280697
469 0.004954569
470 1.8083233
471 0.004595643
472 0.02283982
473 0.0077102575
474 0.006105186
475 1.8987417
476 0.0050776685
477 0.014028737
478 0.01088143
479 0.0049807713
480 2.0969217
481 0.005056028
482 0.007053661
483 0.01666212
484 0.004363433
485 1.4516281
486 0.003897029
487 0.0049343565
488 0.016045112
489 0.0041677123
490 0.9138687
491 0.004612009
492 0.00388224
493 0.04329413
494 0.004387029
495 0.17979228
496 0.00438555
497 0.0034554184
498 0.2610308
499 0.0038759531
cg_method / no_model:
cg        y:  -19.11312
******************************
0 57.376663
1 73.35377
2 6715.5303
3 196.60863
4 10463.488
5 1850.5032
6 255.74355
7 21195.912
8 2482.1382
9 245.21658
10 7072.1846
11 11301.018
12 278.0222
13 1416.334
14 39999.246
15 597.0488
16 384.25214
17 27057.965
18 5610.5903
19 296.88202
20 2215.94
21 51316.17
22 1125.6624
23 345.4526
24 11310.23
25 25675.72
26 540.6161
27 650.01544
28 35959.812
29 14101.133
30 471.27133
31 1016.9487
32 48319.08
33 12192.153
34 474.30212
35 1116.1072
36 49382.785
37 14267.12
38 502.08118
39 847.58606
40 34752.758
41 22656.707
42 658.12
43 574.29675
44 18082.867
45 45946.664
46 1263.3448
47 399.6031
48 6263.9688
49 72828.11
50 3489.0737
51 365.54187
52 1906.9915
53 59749.477
54 14481.404
55 572.6305
56 649.91785
57 18555.324
58 60956.76
59 2154.2766
60 414.248
61 3351.358
62 78818.47
63 14011.946
64 615.0995
65 685.6967
66 17267.375
67 74328.945
68 3164.1646
69 427.1488
70 2160.309
71 60330.08
72 29553.766
73 1086.7655
74 540.79407
75 8121.282
76 110598.23
77 11533.21
78 667.63873
79 984.92096
80 24185.592
81 90805.414
82 4294.743
83 481.3775
84 1684.4487
85 44817.85
86 50800.723
87 1961.3127
88 433.14975
89 2963.7668
90 66907.75
91 27294.514
92 1078.8186
93 431.55347
94 4732.127
95 80394.7
96 16374.905
97 737.6947
98 472.73346
99 6948.98
100 84856.984
101 10791.465
102 569.78217
103 523.3032
104 9142.673
105 82915.31
106 7878.7783
107 478.7177
108 575.0979
109 11263.51
110 77095.19
111 6214.6597
112 416.53625
113 560.0972
114 11023.775
115 69066.03
116 5465.542
117 385.091
118 540.08325
119 10621.299
120 74099.484
121 6484.115
122 437.94366
123 554.88635
124 10303.977
125 75659.36
126 7062.7305
127 440.34872
128 466.47424
129 7915.871
130 71502.484
131 8533.188
132 480.81848
133 382.1964
134 5440.9785
135 65154.21
136 11559.709
137 588.14404
138 324.9634
139 3614.962
140 56828.85
141 17808.58
142 829.05164
143 273.93085
144 2018.2926
145 38963.6
146 26505.125
147 1260.7683
148 252.96803
149 1143.7854
150 24200.867
151 42011.82
152 2344.2295
153 259.7281
154 572.40173
155 11128.002
156 52285.195
157 4492.1123
158 310.4341
159 307.01105
160 4393.872
161 48586.15
162 9972.85
163 521.99396
164 213.60318
165 1694.4315
166 30926.635
167 25407.045
168 1347.8069
169 215.04573
170 672.18024
171 13083.71
172 48829.87
173 4139.2275
174 309.73532
175 305.58325
176 4049.3936
177 49194.953
178 14708.26
179 786.98663
180 219.70412
181 1147.6498
182 21763.57
183 45720.4
184 3290.1465
185 307.08936
186 406.2583
187 5808.877
188 60888.29
189 16086.005
190 906.8917
191 259.74963
192 1301.6854
193 23759.031
194 62942.03
195 5270.968
196 440.37964
197 435.53818
198 5181.593
199 66375.41
200 30247.23
201 1749.1865
202 321.16782
203 971.94244
204 16520.04
205 85756.375
206 11240.35
207 731.1933
208 352.22092
209 2561.808
210 41307.746
211 54671.605
212 3846.024
213 383.08438
214 476.20392
215 5990.562
216 67744.31
217 27297.852
218 1640.6133
219 303.17914
220 864.08093
221 13836.714
222 78935.07
223 12305.824
224 787.52246
225 291.4917
226 1621.5024
227 26550.162
228 65077.41
229 5997.2783
230 470.9826
231 329.63138
232 2898.4868
233 41910.664
234 43738.11
235 3113.2876
236 333.59854
237 422.33173
238 5033.84
239 55295.45
240 25201.514
241 1603.2032
242 241.367
243 491.1797
244 6862.091
245 60455.355
246 18070.018
247 1153.6283
248 253.67148
249 805.07623
250 12371.676
251 70943.75
252 12352.316
253 817.08575
254 267.8726
255 1257.8848
256 20152.086
257 81179.234
258 10790.785
259 769.86597
260 336.7336
261 1988.449
262 30852.453
263 82894.61
264 8721.643
265 682.34607
266 397.20654
267 2883.2725
268 42473.336
269 73594.734
270 6445.583
271 549.08716
272 397.98996
273 3335.4473
274 47416.453
275 66168.266
276 5496.518
277 527.7832
278 481.37747
279 4569.01
280 61241.324
281 66188.86
282 5205.8975
283 540.4238
284 558.7879
285 5593.9927
286 70544.02
287 64418.516
288 4895.2393
289 515.71936
290 541.6138
291 5411.299
292 67385.17
293 59912.21
294 4611.1562
295 500.89136
296 540.3789
297 5405.837
298 67763.78
299 66245.875
300 5284.868
301 563.73145
302 574.9565
303 5521.926
304 69717.62
305 74316.33
306 6146.2197
307 600.2194
308 498.4441
309 4191.929
310 56170.883
311 86944.64
312 8215.608
313 746.5606
314 505.38684
315 3681.1418
316 50632.5
317 105959.32
318 11344.574
319 916.2368
320 440.96872
321 2488.7153
322 34819.082
323 103947.086
324 13667.444
325 1029.9752
326 361.3182
327 1538.6422
328 21670.02
329 112160.06
330 23535.652
331 1732.2299
332 424.403
333 1239.0875
334 16274.906
335 120169.22
336 41074.426
337 3035.4941
338 490.7925
339 849.79724
340 9511.397
341 98068.625
342 70233.445
343 5791.084
344 657.2721
345 630.7613
346 5333.42
347 67651.805
348 120165.88
349 13051.777
350 1126.3926
351 547.6492
352 2911.2537
353 38987.133
354 144898.55
355 24761.666
356 1899.9546
357 491.27228
358 1423.5178
359 17749.305
360 135315.28
361 54928.664
362 4274.7393
363 591.08887
364 758.08716
365 7190.5166
366 81489.875
367 99786.23
368 9861.801
369 923.9706
370 523.18365
371 3009.0483
372 39008.152
373 133672.84
374 22372.055
375 1745.8506
376 439.19168
377 1195.6724
378 14348.283
379 114514.516
380 54733.47
381 4493.2734
382 574.61115
383 610.29395
384 5096.452
385 60834.098
386 110153.38
387 13074.756
388 1128.2812
389 447.0552
390 1865.6533
391 23540.102
392 127116.58
393 34282.99
394 2703.6814
395 466.23257
396 773.53296
397 7822.8955
398 81990.14
399 87940.336
400 8803.873
401 846.85474
402 455.87982
403 2400.8945
404 30110.021
405 126247.94
406 27141.734
407 2185.308
408 441.21484
409 872.167
410 9171.723
411 88991.63
412 80316.336
413 7859.502
414 795.0237
415 463.93726
416 2539.6147
417 31313.574
418 130950.97
419 29313.305
420 2407.3628
421 459.35968
422 813.9294
423 8039.302
424 79927.25
425 88411.86
426 9511.4375
427 902.3125
428 397.04926
429 1672.818
430 19803.021
431 111648.31
432 37499.777
433 3227.5967
434 466.8697
435 521.71497
436 4084.8901
437 45650.367
438 102771.52
439 15455.322
440 1335.8799
441 345.02094
442 839.0605
443 8809.316
444 76136.12
445 61500.29
446 6289.3213
447 669.9717
448 378.2815
449 1883.643
450 22112.906
451 115450.78
452 37076.824
453 3253.8618
454 465.80774
455 486.67285
456 3572.1323
457 39351.82
458 99390.83
459 16828.117
460 1485.2473
461 343.2306
462 701.40314
463 6790.3384
464 64115.598
465 78289.484
466 9468.905
467 916.4252
468 338.11804
469 1123.8933
470 12125.027
471 89355.34
472 56045.188
473 5660.9326
474 619.6034
475 334.9729
476 1529.8489
477 16936.238
478 90349.56
479 33307.664
480 3128.1223
481 427.6416
482 373.07434
483 2348.4634
484 26019.477
485 103555.3
486 27408.742
487 2514.5254
488 398.78467
489 445.54285
490 3184.394
491 33993.74
492 101365.83
493 21147.047
494 1951.5984
495 369.39636
496 532.1833
497 4247.836
498 42992.96
499 96404.68
cg_method / hessian free:
cg        y:  116.68558
******************************
求逆矩阵方法:
inv       y:  -19.138546

当变量维度为10时，N=10，运行结果如下：

******************************
0 0.98815763
1 0.48605096
2 0.35811183
3 0.37906504
4 0.28529668
5 0.73566395
6 0.11153275
7 0.04483844
8 0.0036493908
9 0.012259159
cg_method / model_based:     
cg        y:  -1.0424398
******************************
0 0.9881576
1 0.48605105
2 0.35811186
3 0.37904692
4 0.2621634
5 0.95217586
6 0.11153686
7 0.04483869
8 0.0036848062
9 0.12887856
cg_method / no_model:     
cg        y:  -1.039814
******************************
0 0.9874852
1 0.4882445
2 0.4150008
3 6.0078125
4 0.39969411
5 1.3679078
6 0.25824004
7 0.57612777
8 0.12597126
9 0.07849085
cg_method / hessian free:     
cg        y:  -0.98358154
******************************
求逆矩阵方法:
inv       y:  -1.0508757

=================================================

由于使用hessian free的共轭梯度算法求解非正定矩阵所得到的解与真实解有较大距离，因此如果由于内存限制等原因导致必须使用hessian free共轭梯度法来进行求解，那么也最好结合其他的求解方法来进行联合使用，以此来减少hessian free方法对非正定矩阵求解所存在的较大误差。

=================================================

参考：

【转载】共轭梯度法（视频讲解） 数值分析6(3共轭梯度法) ——苏州大学

=================================================