# FedHPO-Bench
A benchmark suite for studying federated hyperparameter optimization. FedHPO-Bench incorporates comprehensive FL tasks, enables efficient function evaluations, and eases continuing extensions. We also conduct extensive experiments based on FedHPO-B to benchmark a few HPO methods.
## Quick Start
We highly recommend running FedHPO-Bench with conda.
### Step 0. Dependency
* FedHPO-Bench is built on a stable [FederatedScope](https://github.com/alibaba/FederatedScope), please see [Installation](https://github.com/alibaba/FederatedScope#step-1-installation) for install FederatedScope.
```bash
git clone https://github.com/alibaba/FederatedScope.git
cd FederatedScope
git checkout a653102c6b8d5421d2874077594df0b2e29401a1
pip install -e .
```
* (Optianal) In order to reproduce the results in our paper, please consider installing the following packages via:
```bash
# hpbandster, smac3, optuna via conda
conda install hpbandster smac3 optuna -c conda-forge
# dehb via git
conda install dask distributed -c conda-forge
git clone https://github.com/automl/DEHB.git
cd DEHB
git checkout b8dcba7b38bf6e7fc8ce3e84ea567b66132e0eb5
export PYTHONPATH=~/DEHB:$PYTHONPATH
```
### Step 1. Installation
We recommend installing FedHPOBench directly using git by:
```bash
git clone https://github.com/alibaba/FederatedScope.git
cd FederatedScope/benchmark/FedHPOBench
export PYTHONPATH=~/FedHPOBench:$PYTHONPATH
```
### Step 2. Prepare data files
**Note**: If you only want to use FedHPO-Bench with raw mode, you can skip to **Step3**.
All data files are available on AliyunOSS, you need to download the data files and place them in the `~/data/tabular_data/` or `~/data/surrogate_model/` before using FedHPO-Bench.
The naming pattern of the url of data files obeys the rule:
```
https://federatedscope.oss-cn-beijing.aliyuncs.com/fedhpob_{BENCHMARK}_{MODE}.zip
```
where `{BENCHMARK}` should be one of `cnn`, `bert`, `gcn`, `lr`, `mlp` or `cross_device`, and `{MODE}` should be one of `tabular` or `surrogate`.
Then unzip the data file, move data under tabular mode to `~/data/tabular_data/` and data under surrogate mode to `~/data/surrogate_model`.
Fortunately, we provide tools to automatically convert from tabular data to surrogate pickled model, so it is feasible to download only tabular data (it will take a few minutes to train the surrogate model).
### Step3. Start running
```python
from fedhpobench.config import fhb_cfg
from fedhpobench.benchmarks import TabularBenchmark
benchmark = TabularBenchmark('cnn', 'femnist', 'avg')
# get hyperparameters space
config_space = benchmark.get_configuration_space(CS=True)
# get fidelity space
fidelity_space = benchmark.get_fidelity_space(CS=True)
# get results
res = benchmark(config_space.sample_configuration(),
fidelity_space.sample_configuration(),
fhb_cfg=fhb_cfg,
seed=12345)
print(res)
```
## Reproduce the results in our paper
We take Figure 11 as an example.
* First get best seen value of each optimizer, the results are stored in the `~/exp_results` by default.
```
cd scripts/exp
bash run_mode.sh cora raw gcn 0 avg
bash run_mode.sh citeseer raw gcn 1 avg
bash run_mode.sh pubmed raw gcn 1 avg
```
* Then draw the figure with tools we provide, the figures will be saved in `~/figures`.
```python
from fedhpobench.utils.draw import rank_over_time
rank_over_time('exp_results', 'gcn', algo='avg', loss=False)
```
## Using FedHPOBench with HPOBench
HPOBench is a very popular and influential library in the HPO field, so we provide serval demos for exposing FedHPOBench through the interfaces of HPOBench in `demo\`:
How to use:
* Install HPOBench:
```bash
git clone https://github.com/automl/HPOBench.git
cd HPOBench
pip install .
```
* Start to use:
```python
from femnist_tabular_ benchmark import FEMNISTTabularFedHPOBench
b = FEMNISTTabularFedHPOBench('data', 1)
config = b.get_configuration_space(seed=1).sample_configuration()
result_dict = b.objective_function(configuration=config, rng=1)
print(result_dict)
```
## Publications
If you find FedHPO-Bench useful for your research or development, please cite the following [paper](https://arxiv.org/abs/2206.03966):
```tex
@article{Wang2022FedHPOBAB,
title={FedHPO-Bench: A Benchmark Suite for Federated Hyperparameter Optimization},
author={Zhen Wang and Weirui Kuang and Ce Zhang and Bolin Ding and Yaliang Li},
journal={ArXiv},
year={2022},
volume={abs/2206.03966}
}
```
## Look-up table
Available tabular triplets look-up table:
| Model | Dataset | FedAlgo |
| ------------ | ------------------------- | ------- |
| cnn | femnist | avg |
| bert | cola@huggingface_datasets | avg |
| bert | cola@huggingface_datasets | opt |
| bert | sst2@huggingface_datasets | avg |
| bert | sst2@huggingface_datasets | opt |
| gcn | cora | avg |
| gcn | cora | opt |
| gcn | cora | prox |
| gcn | cora | nbafl |
| gcn | citeseer | avg |
| gcn | citeseer | opt |
| gcn | pubmed | avg |
| gcn | pubmed | opt |
| lr | 31@openml | avg |
| lr | 31@openml | opt |
| lr | 53@openml | avg |
| lr | 53@openml | opt |
| lr | 3917@openml | avg |
| lr | 3917@openml | opt |
| lr | 10101@openml | avg |
| lr | 10101@openml | opt |
| lr | 146818@openml | avg |
| lr | 146818@openml | opt |
| lr | 146821@openml | avg |
| lr | 146821@openml | opt |
| lr | 146822@openml | avg |
| lr | 146822@openml | opt |
| mlp | 31@openml | avg |
| mlp | 31@openml | opt |
| mlp | 53@openml | avg |
| mlp | 53@openml | opt |
| mlp | 3917@openml | avg |
| mlp | 3917@openml | opt |
| mlp | 10101@openml | avg |
| mlp | 10101@openml | opt |
| mlp | 146818@openml | avg |
| mlp | 146818@openml | opt |
| mlp | 146821@openml | avg |
| mlp | 146821@openml | opt |
| mlp | 146822@openml | avg |
| mlp | 146822@openml | opt |
| cross_device | twitter | avg |