New default hyperparameters and miscellaneous bug fixes for Kriging. #374
Conversation
Codecov Report
@@ Coverage Diff @@
## master #374 +/- ##
==========================================
- Coverage 78.99% 78.85% -0.14%
==========================================
Files 16 16
Lines 2266 2280 +14
==========================================
+ Hits 1790 1798 +8
- Misses 476 482 +6
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
|
I have added nugget regularization to address the problem where the correlation matrix becomes singular when points are sufficiently close together. The way it currently works is to add a small "nugget" value to the main diagonal of the correlation matrix to keep the condition number less than 1e8. This has the same effect as a tiny noise variance, and prevents the matrix from becoming singular at the cost of slightly relaxing the interpolation condition. |
test/optimization.jl
Outdated
| @@ -21,27 +18,45 @@ a = 2 | |||
| b = 6 | |||
|
|
|||
| #Using Kriging | |||
| my_k_SRBF1 = Kriging(x, y, lb, ub) | |||
| surrogate_optimize(objective_function, SRBF(), a, b, my_k_SRBF1, UniformSample()) | |||
| begin | |||
There was a problem hiding this comment.
what's the purpose of the blocks here? It's fine but a little odd.
src/Kriging.jl
Outdated
| # Estimate nugget based on maximum allowed condition number | ||
| # This regularizes R to allow for points being close to eachother without R becoming | ||
| # singular, at the cost of slightly relaxing the interpolation condition | ||
| λ = eigen(R).values | ||
|
|
||
| λmax = λ[end] | ||
| λmin = λ[1] | ||
|
|
||
| κmax = 1e8 | ||
| λdiff = λmax - κmax * λmin | ||
| if λdiff ≥ 0 | ||
| nugget = λdiff / (κmax - 1) | ||
| else | ||
| nugget = 0.0 | ||
| end |
There was a problem hiding this comment.
Comment on the source for this?
|
That was for ease of running on my end, no other reason. I can remove them
…On Mon, Jul 11, 2022, 8:03 PM Christopher Rackauckas < ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In test/optimization.jl
<#374 (comment)>:
> @@ -21,27 +18,45 @@ a = 2
b = 6
#Using Kriging
-my_k_SRBF1 = Kriging(x, y, lb, ub)
-surrogate_optimize(objective_function, SRBF(), a, b, my_k_SRBF1, UniformSample())
+begin
what's the purpose of the blocks here? It's fine but a little odd.
—
Reply to this email directly, view it on GitHub
<#374 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AOP3QGXZMRWD475WT35V4SDVTSY27ANCNFSM53IZXSXQ>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
yeah probably best to remove the blocks then. |
|
amazing, thanks! |
Addresses #369 and #368. This PR defines new default hyperparameters for the
Krigingsurrogate. Along the way, I found that the implementation of the 1D kriging surrogate was incorrect and did not work properly for theta not equal to 1, so I've fixed that as well.To summarize, the default hyperparameters for
Krigingused to bep = ones(d),theta = ones(d), wheredis the problem dimension. These are very bad choices. I have changed them toThe latter comes from A Practical Guide to Gaussian Processes and the interpretation of$\theta_i = 1/2l_i^2$ , where $l_i$ is a characteristic correlation length scale for variable $i$ . Scaling the length scale with the standard deviation of the samples makes the hyperparameter initialization independent of the box size, which is a very desirable property. With this initialization, here's some comparisons to the old defaults:
Sphere function
Old
New
L1-norm function
Old
New
Branin function
Old
New
Rosenbrock
Old
New
Clearly, the new hyperparameters are a significant improvement. During implementation, I found that the 1D kriging surrogate only worked properly when theta = 1, so I have fixed that as well:
Old (with old default hyperparams)
Old (with new default hyperparams)
New (with new default hyperparams)
Still to do is update the documentation for kriging to make use of the better defaults, as the current documentation doesn't make kriging look all that effective