U dh @sLddlZddlmZddlZddddgZddZddZd dZd dZ dS) N)Shufflerapply_shardingapply_shuffle_seedapply_shuffle_settingsget_all_graph_pipescCsBt}|D].\}}||t|}|D]}||q,q|S)N)setitemsaddr)graphresultsdatapipeZ sub_graphZ sub_itemsitemrC/tmp/pip-unpacked-wheel-ua33x9lu/torch/utils/data/graph_settings.pyr s cCsrtjjjj|dd}t|}d}|D]H}t|dr$|r$t|dr$|dk r\tdd|d|| |||}q$dS)NTZ only_datapipe is_shardablerzVThis implementation of sharding can be only applied once per instance of DataPipeline.zAlready applied tozwhile trying to apply to) torchutilsdatar traverserhasattrr RuntimeErrorr)r Znum_of_instancesZ instance_idr all_pipesZalready_applied_topiperrrrs    cCsn|dkr |Stjjjj|dd}t|}dd|D}|sV|rVtd|}|h}|D]}| |qZ|S)NTrcSsh|]}t|tr|qSr isinstancer.0rrrr ,s z)apply_shuffle_settings..z`shuffle=True` was set, but the datapipe does not contain a `Shuffler`. Adding one at the end. Be aware that the default buffer size might not be sufficient for your task.) rrrr rrwarningswarnshuffleZ set_shuffle)r r!r r shufflersshufflerrrrr&s cCsbtjjjj|dd}t|}dd|D}|D].}ttjdtjdj |d }| |q.|S)NTrcSsh|]}t|tr|qSrrrrrrr>s z%apply_shuffle_seed..r)Zdtype) generator) rrrr rrintemptyZint64Zrandom_r Zset_seed)r rngr rr"r#Z shuffle_seedrrrr;s  ) Ztorch.utils.data.graphrZtorch.utils.data.datapipes.iterrr__all__rrrrrrrrs