FS-TFP/benchmark/B-FHTL

Benchmark for Federated Hetero-Task Learning

B-FHTL is a benchmark for federated hetero-task learning. It contains three federated datasets: Graph-DC, Graph-DT and Text-DT. Here we provide the datasets and scripts for the baselines. More details about the benchmark please refer to https://arxiv.org/abs/2206.03436

Notice: Considering FederatedScope is an open-sourced library that updates frequently, to ensure the reproducibility of the experimental results, we create a new branch feature/B-FHTL. The users can reproduce the results by running the scripts under the directory scripts/B-FHTL_exp_scripts/.

Baseline

We have built in some baselines in FederatedScope, including federated optimization algorithms, federated personalization algorithms and federated meta learning algorithms. Specifically, the baselines includes

• Isolated
• FedAvg
• FedAvg+FT
• FedProx
• FedBn
• FedBn+FT
• Ditto
• FedMAML

Dataset

Graph-DC

The dataset Graph-DC is consisted of 13 graph classification tasks. Each task represents a client in federated learning, and hold different learning goals. You can choose this dataset by setting data.type as fs_contest_data.

Graph-DT

The dataset Graph-DT is consisted of 16 graph tasks, including 10 binary classification tasks, 6 regression tasks. You can choose this dataset by setting data.type as graph-dt.

Text-DT

The dataset Text-DT is consisted of 3 NLP tasks, including sentiment classification, reading compression and sentence pair similarity prediction. You can choose this dataset by setting data.type as text-dt.

Quick Start

You can find the configurations of the baselines under the directory scripts/B-FHTL_exp_scripts/${DATA_NAME}/, where DATA_NAME is the name of the dataset (Graph-DC, Graph-DT or Text-DT).
We also provide high performance operation (hpo) scripts to search for the best hyper-parameters, you can refer to scripts/B-FHTL_exp_scripts/${DATA_NAME}/hpo.

To run the script, you should

• First clone the repository FederatedScope,
• Then follow README.md to build the running environment for FederatedScope,
• Install the learn2learn and transformers packages that are required by benchmark,
• Switch to the branch feature/B-FHTL and run the scripts

# Step-1. clone the repository 
git clone https://github.com/alibaba/FederatedScope.git

# Step-2. follow https://github.com/alibaba/FederatedScope/blob/master/README.md to build the running environment

# Step-3. install packages required by the benchmark
pip install learn2learn transformers

# Step-3. switch to the branch `feature/B-FHTL` for the benchmark
git switch feature/B-FHTL

# Step-4. run the baseline (taking fedavg with Graph-DC as an example)
cd FederatedScope
python federatedscope/main.py --cfg scripts/B-FHTL_exp_scripts/Graph-DC/fedavg.yaml

# If you want to run the hpo scripts
cd scripts/B-FHTL_exp_scripts/Graph-DC/hpo
bash run_fedavg_ft.sh 0 ./data