out of memory for instrument "all"

[playgithub]

❓ Questions and Help

hareware

CPU memory 32GB
GPU memory 8GB

problem

In the following sample
for instrument "all", it'll be out of memory
for instrument "csi300", it's ok

sample

out of memroy by dataset = init_instance_by_config(task["dataset"]) below

...
stock_pool = 'all'

train_start_date = '2017-7-12'
train_end_date = '2020-12-31'
valid_start_date = '2021-01-01'
valid_end_date = '2023-12-31'
test_start_date = '2024-01-01'
test_end_date = '2025-12-31'
...
task = {
    "model": {
        "class": "LGBModel",
        "module_path": "qlib.contrib.model.gbdt",
        "kwargs": {
            "loss": "mse",
            "colsample_bytree": 0.8879,
            "learning_rate": 0.0421,
            "subsample": 0.8789,
            "lambda_l1": 205.6999,
            "lambda_l2": 580.9768,
            "max_depth": 8,
            "num_leaves": 210,
            "num_threads": 20,
            "early_stopping_rounds": 50,
            "num_boost_round": 1000
        }
    },
    "dataset": {
        "class": "DatasetH",
        "module_path": "qlib.data.dataset",
        "kwargs": {
            "handler": {
                "class": "Alpha158",
                "module_path": "qlib.contrib.data.handler",
                "kwargs": data_handler_config
            },
            "segments": {
                "train": (train_start_date, train_end_date),
                "valid": (valid_start_date, valid_end_date),
                "test": (test_start_date, test_end_date)
            }
        }
    }
}
...
dataset = init_instance_by_config(task["dataset"])
...

quesiton

any solution?

[SunsetWolf]

Hi, @playgithub , This is expected behavior. When instrument="all", init_instance_by_config(task["dataset"]) needs to load and process the full market universe, which significantly increases memory usage.

Based on memory profiling, the peak usage is ~28.5 GB during dataset initialization alone. After accounting for OS and other processes, a 32 GB machine is not sufficient.

[playgithub]

expected, any workaround? e.g.:

• a compressed dataframe in memory, which only uncompressed when get data
• or map data on disk to memory

memory is qutie expensive now ;)