Init

HEBO/.gitignore

@@ -0,0 +1,18 @@
+# vim 
+*.swp
+ref/
+debug.log
+
+# python
+*.pyc
+__pycache__/
+.ipynb_checkpoints/
+.pytest_cache/
+.coverage*
+.idea/
+
+build/
+dist/
+*.egg-info/
+htmlcov/
+worksapce/

HEBO/LICENSE

@@ -0,0 +1,9 @@
+MIT License
+
+Copyright (C) 2019. Huawei Technologies Co., Ltd. All rights reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

HEBO/README.md

@@ -0,0 +1,105 @@
+![](hebo.png)
+```
+pip install HEBO
+```
+# README
+
+Bayesian optimsation library developped by Huawei Noahs Ark Decision Making and Reasoning (DMnR) lab. The <strong> winning submission </strong> to the [NeurIPS 2020 Black-Box Optimisation Challenge](https://bbochallenge.com/leaderboard). 
+
+Summary             |  Ablation
+:-------------------------:|:-------------------------:
+[Results]( https://github.com/huawei-noah/noah-research/blob/master/HEBO/summary_plot2.pdf) | [Results](https://github.com/huawei-noah/noah-research/blob/master/HEBO/summary_ablation2.pdf)
+
+# Contributors 
+
+<strong> Alexander I. Cowen-Rivers, Wenlong Lyu, Zhi Wang, Antoine Grosnit, Rasul Tutunov, Hao Jianye, Jun Wang, Haitham Bou Ammar. </strong>
+
+## Installation
+
+```bash
+python setup.py develop
+```
+
+## Demo
+
+```python
+import pandas as pd
+import numpy  as np
+from hebo.design_space.design_space import DesignSpace
+from hebo.optimizers.hebo import HEBO
+
+def obj(params : pd.DataFrame) -> np.ndarray:
+    return ((params.values - 0.37)**2).sum(axis = 1).reshape(-1, 1)
+
+space = DesignSpace().parse([{'name' : 'x', 'type' : 'num', 'lb' : -3, 'ub' : 3}])
+opt   = HEBO(space)
+for i in range(5):
+    rec = opt.suggest(n_suggestions = 4)
+    opt.observe(rec, obj(rec))
+    print('After %d iterations, best obj is %.2f' % (i, opt.y.min()))
+```
+
+## Auto Tuning via Sklearn Estimator
+
+```python
+from sklearn.datasets import load_boston
+from sklearn.ensemble import RandomForestRegressor
+from sklearn.metrics import r2_score, mean_squared_error
+
+from hebo.sklearn_tuner import sklearn_tuner
+
+space_cfg = [
+    {'name' : 'max_depth', 'type' : 'int', 'lb' : 1, 'ub' : 20},
+    {'name' : 'min_samples_leaf', 'type' : 'num', 'lb' : 1e-4, 'ub' : 0.5},
+    {'name' : 'max_features', 'type' : 'cat', 'categories' : ['auto', 'sqrt', 'log2']},
+    {'name' : 'bootstrap', 'type' : 'bool'},
+    {'name' : 'min_impurity_decrease', 'type' : 'pow', 'lb' : 1e-4, 'ub' : 1.0},
+    ]
+X, y   = load_boston(return_X_y = True)
+result = sklearn_tuner(RandomForestRegressor, space_cfg, X, y, metric = r2_score, max_iter = 16)
+```
+
+## Documentation
+
+```bash
+cd doc
+make html
+```
+
+You can view the compiled documentation in `doc/build/html/index.html`.
+
+## Test
+
+```bash
+pytest -v test/ --cov ./bo --cov-report term-missing --cov-config ./test/.coveragerc
+```
+
+## Reproduce Experimental Results
+
+- See `archived_submissions/hebo`, which is the exact submission that won the NeurIPS2020 Black-Box Optimsation Challenge.
+- Use `run_local.sh` in [bbo_challenge_starter_kit](https://github.com/rdturnermtl/bbo_challenge_starter_kit/) to reproduce `bayesmark` experiments, you can just drop `archived_submissions/hebo` to the `example_submissions` directory.
+- The `MACEBO` in `bo.optimizers.mace` is the same optimiser, with same hyperparameters but a modified interface (bayesmark dependency removed).
+
+
+## Features
+
+- Continuous, integer and categorical design parameters.
+- Constrained and multi-objective optimsation.
+- Contextual optimsation.
+- Multiple surrogate models including GP, RF and BNN.
+- Modular and flexible Bayesian Optimisation building blocks.
+
+
+## Cite Us
+
+Cowen-Rivers, Alexander I., et al. "An Empirical Study of Assumptions in Bayesian Optimisation." arXiv preprint arXiv:2012.03826 (2021).
+
+## BibTex
+```
+@article{cowen2020empirical,
+  title={An Empirical Study of Assumptions in Bayesian Optimisation},
+  author={Cowen-Rivers, Alexander I and Lyu, Wenlong and Tutunov, Rasul and Wang, Zhi and Grosnit, Antoine and Griffiths, Ryan Rhys and Jianye, Hao and Wang, Jun and Ammar, Haitham Bou},
+  journal={arXiv preprint arXiv:2012.03826},
+  year={2020}
+}
+```

HEBO/TODO.md

@@ -0,0 +1,43 @@
+# TODO
+
+- Documentation
+- Options for MACE
+    - Power transformation
+    - Noise exploration
+    - Verbose
+- EI, UCB, PI, MES, TS
+- MO model wrapper
+- Constrained and multi-objective opt
+- General single/multi-objective constrained optimizer
+- Gaussian process:
+    - Support different x/y scalers
+    - Support different kernels (Mat32, Mat52, RBF)
+    - Support different categorical processing methods
+- Build wheel, setup requirement
+
+# Done
+
+- Upgrade to pymoo-0.4.2
+- Optimizer API
+- Design space redesign
+- Contextual BO API in `suggest`
+- Options for MACE:
+    - Model
+    - Num random
+
+# Features to support
+
+- Basic Bayesian optimisation
+- Mainstream acquisition functions
+- Constrained Bayesian optimisation (weighted EI, Thompson sampling, entropy search) 
+- Multi-objective Bayesian optimisation
+- Multi-fidelity Bayesian optimisation
+- Safe Bayesian optimisation
+- High-dimensional Bayesian optimisation
+- Batch optimisation
+- Meta learning (learn from past optimisation tasks or relavent data)
+- Prior embedding
+- Support other probablistic models:
+    - BNN via variational inference
+    - BNN via SGDMCMC
+    - Deep ensemble

HEBO/archived_submissions/hebo/.gitignore

@@ -0,0 +1,10 @@
+# vim 
+*.swp
+ref/
+
+# python
+*.pyc
+__pycache__/
+.ipynb_checkpoints/
+.pytest_cache/
+.coverage

HEBO/archived_submissions/hebo/README.md

@@ -0,0 +1,24 @@
+# README
+
+Bayesian optimisation library
+
+## Features to support
+
+- Basic Bayesian optimisation
+- Mainstream acquisition functions
+- Constrained Bayesian optimisation (weighted EI, Thompson sampling, entropy search) 
+- Multi-objective Bayesian optimisation
+- Multi-fidelity Bayesian optimisation
+- Safe Bayesian optimisation
+- High-dimensional Bayesian optimisation
+- Batch optimisation
+- Meta learning (learn from past optimisation tasks or relavent data)
+- Prior embedding
+- Support other probablistic models:
+    - BNN via variational inference
+    - BNN via SGDMCMC
+    - Deep ensemble
+
+## TODO
+
+Everything (+_+)

HEBO/archived_submissions/hebo/bo/acquisitions/acq.py

@@ -0,0 +1,169 @@
+# Copyright (C) 2020. Huawei Technologies Co., Ltd. All rights reserved.
+
+# This program is free software; you can redistribute it and/or modify it under
+# the terms of the MIT license.
+
+# This program is distributed in the hope that it will be useful, but WITHOUT ANY
+# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+# PARTICULAR PURPOSE. See the MIT License for more details.
+
+import torch
+import numpy as np
+from torch import Tensor
+from torch.distributions import Normal
+from abc import ABC, abstractmethod
+from ..models.base_model import BaseModel
+
+class Acquisition(ABC):
+    def __init__(self, model, **conf):
+        self.model = model
+
+    @property
+    @abstractmethod
+    def num_obj(self):
+        pass
+
+    @property
+    @abstractmethod
+    def num_constr(self):
+        pass
+
+    @abstractmethod
+    def eval(self, x : Tensor,  xe : Tensor) -> Tensor:
+        """
+        Shape of output tensor: (x.shape[0], self.num_obj + self.num_constr)
+        """
+        pass
+
+    def __call__(self, x : Tensor,  xe : Tensor):
+        return self.eval(x, xe)
+
+class SingleObjectiveAcq(Acquisition):
+    def __init__(self, model : BaseModel, **conf):
+        super().__init__(model, **conf)
+
+    @property
+    def num_obj(self):
+        return 1
+
+    @property
+    def num_constr(self):
+        return 0
+
+class LCB(SingleObjectiveAcq):
+    def __init__(self, model : BaseModel, kappa = 3.0, **conf):
+        super().__init__(model, **conf)
+        self.kappa      = kappa
+        self.minimize   = 1.0  if conf.get('minimize', True) else -1.0 # minimize: LCB, maximize: UCB
+        assert(model.num_out == 1)
+
+    def eval(self, x : Tensor, xe : Tensor) -> Tensor:
+        py, ps2 = self.model.predict(x, xe)
+        return py - self.minimize * self.kappa * ps2.sqrt()
+
+class Mean(SingleObjectiveAcq):
+    def __init__(self, model : BaseModel, **conf):
+        super().__init__(model, **conf)
+        assert(model.num_out == 1)
+
+    def eval(self, x : Tensor, xe : Tensor) -> Tensor:
+        py, _ = self.model.predict(x, xe)
+        return py
+
+class Sigma(SingleObjectiveAcq):
+    def __init__(self, model : BaseModel, **conf):
+        super().__init__(model, **conf)
+        assert(model.num_out == 1)
+
+    def eval(self, x : Tensor, xe : Tensor) -> Tensor:
+        _, ps2 = self.model.predict(x, xe)
+        return ps2.sqrt()
+
+class EI(SingleObjectiveAcq):
+    pass
+
+class logEI(SingleObjectiveAcq):
+    pass
+
+class WEI(SingleObjectiveAcq):
+    pass
+
+class Log_WEI(SingleObjectiveAcq):
+    pass
+
+class MES(SingleObjectiveAcq):
+    pass
+
+class MOMeanSigmaLCB(Acquisition):
+    def __init__(self, model, best_y, **conf):
+        super().__init__(model, **conf)
+        self.best_y = best_y
+        self.kappa  = conf.get('kappa', 2.0)
+        assert(self.model.num_out == 1)
+
+    @property
+    def num_obj(self):
+        return 2
+
+    @property
+    def num_constr(self):
+        return 1
+
+    def eval(self, x: Tensor, xe : Tensor) -> Tensor:
+        """
+        minimize (py, -1 * ps)
+        s.t.     LCB  < best_y
+        """
+        with torch.no_grad():
+            out        = torch.zeros(x.shape[0], self.num_obj + self.num_constr)
+            py, ps2    = self.model.predict(x, xe)
+            noise      = np.sqrt(self.model.noise)
+            py        += noise * torch.randn(py.shape)
+            ps         = ps2.sqrt()
+            lcb        = py - self.kappa * ps
+            out[:, 0]  = py.squeeze()
+            out[:, 1]  = -1 * ps.squeeze()
+            out[:, 2]  = lcb.squeeze() - self.best_y # lcb - best_y < 0
+            return out
+
+class MACE(Acquisition):
+    def __init__(self, model, best_y, **conf):
+        super().__init__(model, **conf)
+        self.kappa = conf.get('kappa', 2.0)
+        self.eps   = conf.get('eps', 1e-4)
+        self.tau   = best_y
+
+    @property
+    def num_constr(self):
+        return 0
+
+    @property
+    def num_obj(self):
+        return 3
+
+    def eval(self, x : torch.FloatTensor, xe : torch.LongTensor) -> torch.FloatTensor:
+        """
+        minimize (-1 * EI,  -1 * PI, lcb)
+        """
+        with torch.no_grad():
+            py, ps2   = self.model.predict(x, xe)
+            noise     = np.sqrt(2.0) * self.model.noise.sqrt()
+            ps        = ps2.sqrt()
+            lcb       = (py + noise * torch.randn(py.shape)) - self.kappa * ps
+            normed    = ((self.tau - self.eps - py - noise * torch.randn(py.shape)) / ps)
+            dist      = Normal(0., 1.)
+            log_phi   = dist.log_prob(normed)
+            Phi       = dist.cdf(normed)
+            PI        = Phi
+            EI        = ps * (Phi * normed +  log_phi.exp())
+            logEIapp  = ps.log() - 0.5 * normed**2 - (normed**2 - 1).log()
+            logPIapp  = -0.5 * normed**2 - torch.log(-1 * normed) - torch.log(torch.sqrt(torch.tensor(2 * np.pi)))
+
+            use_app             = normed.reshape(-1) < -6
+            out                 = torch.zeros(x.shape[0], 3)
+            out[:, 0]           = lcb.reshape(-1)
+            out[:, 1][use_app]  = -1 * logEIapp[use_app].reshape(-1)
+            out[:, 2][use_app]  = -1 * logPIapp[use_app].reshape(-1)
+            out[:, 1][~use_app] = -1 * EI[~use_app].log().reshape(-1)
+            out[:, 2][~use_app] = -1 * PI[~use_app].log().reshape(-1)
+            return out