• 技术文章 >Python技术 >Python基础教程

    python分布式执行框架Ray的介绍

    小妮浅浅小妮浅浅2021-09-09 09:41:49转载3969

    说明

    1、Ray为构建分布式应用程序提供了一个简单、通用的API。

    2、Ray是一种分布式执行框架,便于大规模应用程序和利用先进的机器学习库。

    Ray通过以下方式完成这项任务:

    为构建和运行分布式应用程序提供简单的原语。

    使最终用户能够并行化单个机器代码,而代码更改很少到零。

    在核心Ray之上包含大量应用程序、库和工具,以支持复杂的应用程序。

    安装方法

    1

    2

    3

    4

    5

    6

    7

    8

    9

    10

    11

    12

    13

    14

    15

    16

    17

    18

    19

    20

    21

    22

    23

    24

    25

    26

    27

    28

    29

    30

    31

    32

    33

    34

    35

    36

    37

    38

    39

    40

    41

    42

    43

    44

    45

    46

    47

    48

    49

    50

    51

    52

    53

    54

    55

    56

    57

    58

    59

    60

    61

    62

    63

    64

    65

    66

    67

    68

    69

    70

    71

    72

    73

    74

    75

    76

    77

    78

    79

    80

    81

    82

    83

    84

    85

    86

    87

    88

    89

    90

    91

    92

    93

    94

    95

    96

    97

    98

    99

    100

    101

    102

    103

    104

    105

    106

    107

    108

    109

    110

    111

    112

    113

    114

    115

    116

    117

    118

    119

    120

    121

    122

    123

    124

    125

    126

    127

    128

    129

    130

    131

    132

    133

    134

    135

    136

    137

    138

    139

    140

    141

    142

    143

    144

    [root@node2 ~]# pip install 'ray[default]'

    Looking in indexes: https://mirrors.aliyun.com/pypi/simple/

    Collecting ray[default]

      Downloading https://mirrors.aliyun.com/pypi/packages/13/ec/f727ddd3fbcdc6102eace62c9d5dd9d9ad8112d40eeb7de8783676aca24d/ray-1.4.1-cp36-cp36m-manylinux2014_x86_64.whl (51.6 MB)

         |████████████████████████████████| 51.6 MB 269 kB/s

    Collecting aiohttp

      Downloading https://mirrors.aliyun.com/pypi/packages/31/28/7b49246d7825c61e1a14306ab050e8eec48fcf5b009b19c8f6ea4e312467/aiohttp-3.7.4.post0-cp36-cp36m-manylinux2014_x86_64.whl (1.3 MB)

         |████████████████████████████████| 1.3 MB 121.5 MB/s

    Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/site-packages (from ray[default]) (0.8)

    Collecting protobuf>=3.15.3

      Downloading https://mirrors.aliyun.com/pypi/packages/53/4e/e2db88d0bb0bda6a879eea62fddbaf813719ce3770d458bc5580512d9c95/protobuf-3.17.3-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)

         |████████████████████████████████| 1.0 MB 136.1 MB/s

    Collecting jsonschema

      Downloading https://mirrors.aliyun.com/pypi/packages/c5/8f/51e89ce52a085483359217bc72cdbf6e75ee595d5b1d4b5ade40c7e018b8/jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)

         |████████████████████████████████| 56 kB 7.1 MB/s

    Collecting aioredis

      Downloading https://mirrors.aliyun.com/pypi/packages/b0/64/1b1612d0a104f21f80eb4c6e1b6075f2e6aba8e228f46f229cfd3fdac859/aioredis-1.3.1-py3-none-any.whl (65 kB)

         |████████████████████████████████| 65 kB 6.3 MB/s

    Collecting pydantic>=1.8

      Downloading https://mirrors.aliyun.com/pypi/packages/2b/7c/7d0b3f2d7959b7193018896db236ded165f9bca1bb75f46f4c32fa6f4f9d/pydantic-1.8.2-cp36-cp36m-manylinux2014_x86_64.whl (10.2 MB)

         |████████████████████████████████| 10.2 MB 420 kB/s

    Collecting pyyaml

      Downloading https://mirrors.aliyun.com/pypi/packages/7a/5b/bc0b5ab38247bba158504a410112b6c03f153c652734ece1849749e5f518/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)

         |████████████████████████████████| 640 kB 19.1 MB/s

    Collecting numpy>=1.16

      Downloading https://mirrors.aliyun.com/pypi/packages/14/32/d3fa649ad7ec0b82737b92fefd3c4dd376b0bb23730715124569f38f3a08/numpy-1.19.5-cp36-cp36m-manylinux2010_x86_64.whl (14.8 MB)

         |████████████████████████████████| 14.8 MB 296 kB/s

    Collecting gpustat

      Downloading https://mirrors.aliyun.com/pypi/packages/b4/69/d8c849715171aeabd61af7da080fdc60948b5a396d2422f1f4672e43d008/gpustat-0.6.0.tar.gz (78 kB)

         |████████████████████████████████| 78 kB 7.3 MB/s

    Collecting redis>=3.5.0

      Downloading https://mirrors.aliyun.com/pypi/packages/a7/7c/24fb0511df653cf1a5d938d8f5d19802a88cef255706fdda242ff97e91b7/redis-3.5.3-py2.py3-none-any.whl (72 kB)

         |████████████████████████████████| 72 kB 968 kB/s

    Collecting aiohttp-cors

      Downloading https://mirrors.aliyun.com/pypi/packages/13/e7/e436a0c0eb5127d8b491a9b83ecd2391c6ff7dcd5548dfaec2080a2340fd/aiohttp_cors-0.7.0-py3-none-any.whl (27 kB)

    Collecting py-spy>=0.2.0

      Downloading https://mirrors.aliyun.com/pypi/packages/9d/4d/1a9cbe9a0b543e6733cb38afe26451522a9ef8e4897b59e74cc76838f245/py_spy-0.3.7-py2.py3-none-manylinux1_x86_64.whl (3.1 MB)

         |████████████████████████████████| 3.1 MB 131 kB/s

    Requirement already satisfied: requests in /usr/local/lib/python3.6/site-packages (from ray[default]) (2.24.0)

    Collecting colorama

      Downloading https://mirrors.aliyun.com/pypi/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl (16 kB)

    Collecting filelock

      Downloading https://mirrors.aliyun.com/pypi/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl (7.6 kB)

    Collecting prometheus-client>=0.7.1

      Downloading https://mirrors.aliyun.com/pypi/packages/09/da/4e8471ff825769581593b5b84769d32f58e5373b59fccaf355d3529ad530/prometheus_client-0.11.0-py2.py3-none-any.whl (56 kB)

         |████████████████████████████████| 56 kB 4.2 MB/s

    Collecting msgpack<2.0.0,>=1.0.0

      Downloading https://mirrors.aliyun.com/pypi/packages/0c/0d/b1d9d32d03ce38ba5e2a37fbae850afd4530a14cc441e8335f1865a03705/msgpack-1.0.2-cp36-cp36m-manylinux1_x86_64.whl (272 kB)

         |████████████████████████████████| 272 kB 127.6 MB/s

    Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.6/site-packages (from ray[default]) (7.1.2)

    Collecting opencensus

      Downloading https://mirrors.aliyun.com/pypi/packages/18/59/12044123133d000f705383ad98579aeb0dd82d66b33a254a21b54bf0d6bb/opencensus-0.7.13-py2.py3-none-any.whl (127 kB)

         |████████████████████████████████| 127 kB 128.9 MB/s

    Collecting grpcio>=1.28.1

      Downloading https://mirrors.aliyun.com/pypi/packages/eb/32/332a2bb0cee0d4040331952c5ef3d040d119a15b3c622b60bdcbe768bded/grpcio-1.39.0-cp36-cp36m-manylinux2014_x86_64.whl (4.3 MB)

         |████████████████████████████████| 4.3 MB 417 kB/s

    Collecting colorful

      Downloading https://mirrors.aliyun.com/pypi/packages/b0/8e/e386e248266952d24d73ed734c2f5513f34d9557032618c8910e605dfaf6/colorful-0.5.4-py2.py3-none-any.whl (201 kB)

         |████████████████████████████████| 201 kB 113.9 MB/s

    Requirement already satisfied: six>=1.5.2 in /usr/local/lib/python3.6/site-packages (from grpcio>=1.28.1->ray[default]) (1.15.0)

    Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.6/site-packages (from pydantic>=1.8->ray[default]) (3.10.0.0)

    Collecting multidict<7.0,>=4.5

      Downloading https://mirrors.aliyun.com/pypi/packages/a1/35/b22524d6b9cacfb4c5eff413a069bbc17c6ea628e54da5c6c989998ced5f/multidict-5.1.0-cp36-cp36m-manylinux2014_x86_64.whl (141 kB)

         |████████████████████████████████| 141 kB 120.6 MB/s

    Collecting idna-ssl>=1.0

      Downloading https://mirrors.aliyun.com/pypi/packages/46/03/07c4894aae38b0de52b52586b24bf189bb83e4ddabfe2e2c8f2419eec6f4/idna-ssl-1.1.0.tar.gz (3.4 kB)

    Collecting attrs>=17.3.0

      Downloading https://mirrors.aliyun.com/pypi/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl (53 kB)

         |████████████████████████████████| 53 kB 3.9 MB/s

    Requirement already satisfied: chardet<5.0,>=2.0 in /usr/local/lib/python3.6/site-packages (from aiohttp->ray[default]) (3.0.4)

    Collecting yarl<2.0,>=1.0

      Downloading https://mirrors.aliyun.com/pypi/packages/da/08/52b26b44bce7b818b410aee37c5e424c9ea420c557bca97dc2adac29b151/yarl-1.6.3-cp36-cp36m-manylinux2014_x86_64.whl (293 kB)

         |████████████████████████████████| 293 kB 130.8 MB/s

    Collecting async-timeout<4.0,>=3.0

      Downloading https://mirrors.aliyun.com/pypi/packages/e1/1e/5a4441be21b0726c4464f3f23c8b19628372f606755a9d2e46c187e65ec4/async_timeout-3.0.1-py3-none-any.whl (8.2 kB)

    Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.6/site-packages (from idna-ssl>=1.0->aiohttp->ray[default]) (2.10)

    Collecting hiredis

      Downloading https://mirrors.aliyun.com/pypi/packages/82/72/313fb6d30b7e413c8662447f9a0abe9105494ceb9266ecb08d442c14afbb/hiredis-2.0.0-cp36-cp36m-manylinux2010_x86_64.whl (84 kB)

         |████████████████████████████████| 84 kB 4.8 MB/s

    Collecting nvidia-ml-py3>=7.352.0

      Downloading https://mirrors.aliyun.com/pypi/packages/6d/64/cce82bddb80c0b0f5c703bbdafa94bfb69a1c5ad7a79cff00b482468f0d3/nvidia-ml-py3-7.352.0.tar.gz (19 kB)

    Collecting psutil

      Downloading https://mirrors.aliyun.com/pypi/packages/da/82/56cd16a4c5f53e3e5dd7b2c30d5c803e124f218ebb644ca9c30bc907eadd/psutil-5.8.0-cp36-cp36m-manylinux2010_x86_64.whl (291 kB)

         |████████████████████████████████| 291 kB 131.6 MB/s

    Collecting blessings>=1.6

      Downloading https://mirrors.aliyun.com/pypi/packages/03/74/489f85a78247609c6b4f13733cbf3ba0d864b11aa565617b645d6fdf2a4a/blessings-1.7-py3-none-any.whl (18 kB)

    Requirement already satisfied: setuptools in /usr/lib/python3.6/site-packages (from jsonschema->ray[default]) (39.2.0)

    Collecting pyrsistent>=0.14.0

      Downloading https://mirrors.aliyun.com/pypi/packages/6c/19/1af501f6f388a40ede6d0185ba481bdb18ffc99deab0dd0d092b173bc0f4/pyrsistent-0.18.0-cp36-cp36m-manylinux1_x86_64.whl (117 kB)

         |████████████████████████████████| 117 kB 138.2 MB/s

    Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.6/site-packages (from jsonschema->ray[default]) (4.3.0)

    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.6/site-packages (from importlib-metadata->jsonschema->ray[default]) (3.4.1)

    Collecting opencensus-context==0.1.2

      Downloading https://mirrors.aliyun.com/pypi/packages/f1/33/990f1bd9e7ee770fc8d3c154fc24743a96f16a0e49e14e1b7540cc2fdd93/opencensus_context-0.1.2-py2.py3-none-any.whl (4.4 kB)

    Collecting google-api-core<2.0.0,>=1.0.0

      Downloading https://mirrors.aliyun.com/pypi/packages/aa/51/629a31d4e7db2bbc0b0b789ac4034108c68647bea9cf5b54c8366df47b8e/google_api_core-1.31.0-py2.py3-none-any.whl (93 kB)

         |████████████████████████████████| 93 kB 1.6 MB/s

    Collecting contextvars

      Downloading https://mirrors.aliyun.com/pypi/packages/83/96/55b82d9f13763be9d672622e1b8106c85acb83edd7cc2fa5bc67cd9877e9/contextvars-2.4.tar.gz (9.6 kB)

    Collecting google-auth<2.0dev,>=1.25.0

      Downloading https://mirrors.aliyun.com/pypi/packages/d7/13/35b1e0a63e160ecad04985e60339382e48e6020d22f8328a2ab47226c910/google_auth-1.33.1-py2.py3-none-any.whl (152 kB)

         |████████████████████████████████| 152 kB 127.7 MB/s

    Collecting packaging>=14.3

      Downloading https://mirrors.aliyun.com/pypi/packages/3c/77/e2362b676dc5008d81be423070dd9577fa03be5da2ba1105811900fda546/packaging-21.0-py3-none-any.whl (40 kB)

         |████████████████████████████████| 40 kB 6.1 MB/s

    Collecting googleapis-common-protos<2.0dev,>=1.6.0

      Downloading https://mirrors.aliyun.com/pypi/packages/55/08/796a6bc0b550e2b7116041c953d3d5100016abea106131d71e5651826e7b/googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)

         |████████████████████████████████| 198 kB 136.3 MB/s

    Collecting setuptools

      Downloading https://mirrors.aliyun.com/pypi/packages/bd/25/5bdf7f1adeebd4e3fa76b2e2f045ae53ee208e40a4231ad0f0c3007e4353/setuptools-57.4.0-py3-none-any.whl (819 kB)

         |████████████████████████████████| 819 kB 127.0 MB/s

    Requirement already satisfied: pytz in /usr/local/lib/python3.6/site-packages (from google-api-core<2.0.0,>=1.0.0->opencensus->ray[default]) (2020.1)

    Collecting cachetools<5.0,>=2.0.0

      Downloading https://mirrors.aliyun.com/pypi/packages/bf/28/c4f5796c67ad06bb91d98d543a5e01805c1ff065e08871f78e52d2a331ad/cachetools-4.2.2-py3-none-any.whl (11 kB)

    Collecting pyasn1-modules>=0.2.1

      Downloading https://mirrors.aliyun.com/pypi/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)

         |████████████████████████████████| 155 kB 21.5 MB/s

    Collecting rsa<5,>=3.1.4

      Downloading https://mirrors.aliyun.com/pypi/packages/e9/93/0c0f002031f18b53af7a6166103c02b9c0667be528944137cc954ec921b3/rsa-4.7.2-py3-none-any.whl (34 kB)

    Collecting pyparsing>=2.0.2

      Downloading https://mirrors.aliyun.com/pypi/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)

         |████████████████████████████████| 67 kB 7.2 MB/s

    Collecting pyasn1<0.5.0,>=0.4.6

      Downloading https://mirrors.aliyun.com/pypi/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)

         |████████████████████████████████| 77 kB 7.0 MB/s

    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests->ray[default]) (2020.6.20)

    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests->ray[default]) (1.25.10)

    Collecting immutables>=0.9

      Downloading https://mirrors.aliyun.com/pypi/packages/4a/52/e64a14a99c509cbdfe0405e9f076aef0331cb9548a3efa1d5bacd524978a/immutables-0.15-cp36-cp36m-manylinux1_x86_64.whl (100 kB)

         |████████████████████████████████| 100 kB 18.2 MB/s

    Using legacy 'setup.py install' for idna-ssl, since package 'wheel' is not installed.

    Using legacy 'setup.py install' for gpustat, since package 'wheel' is not installed.

    Using legacy 'setup.py install' for nvidia-ml-py3, since package 'wheel' is not installed.

    Using legacy 'setup.py install' for contextvars, since package 'wheel' is not installed.

    Installing collected packages: pyasn1, setuptools, rsa, pyparsing, pyasn1-modules, protobuf, multidict, immutables, cachetools, yarl, packaging, idna-ssl, googleapis-common-protos, google-auth, contextvars, attrs, async-timeout, pyrsistent, psutil, opencensus-context, nvidia-ml-py3, hiredis, google-api-core, blessings, aiohttp, redis, pyyaml, pydantic, py-spy, prometheus-client, opencensus, numpy, msgpack, jsonschema, grpcio, gpustat, filelock, colorama, aioredis, aiohttp-cors, ray, colorful

      Attempting uninstall: setuptools

        Found existing installation: setuptools 39.2.0

        Uninstalling setuptools-39.2.0:

          Successfully uninstalled setuptools-39.2.0

        Running setup.py install for idna-ssl ... done

        Running setup.py install for contextvars ... done

        Running setup.py install for nvidia-ml-py3 ... done

        Running setup.py install for gpustat ... done

    Successfully installed aiohttp-3.7.4.post0 aiohttp-cors-0.7.0 aioredis-1.3.1 async-timeout-3.0.1 attrs-21.2.0 blessings-1.7 cachetools-4.2.2 colorama-0.4.4 colorful-0.5.4 contextvars-2.4 filelock-3.0.12 google-api-core-1.31.0 google-auth-1.33.1 googleapis-common-protos-1.53.0 gpustat-0.6.0 grpcio-1.39.0 hiredis-2.0.0 idna-ssl-1.1.0 immutables-0.15 jsonschema-3.2.0 msgpack-1.0.2 multidict-5.1.0 numpy-1.19.5 nvidia-ml-py3-7.352.0 opencensus-0.7.13 opencensus-context-0.1.2 packaging-21.0 prometheus-client-0.11.0 protobuf-3.17.3 psutil-5.8.0 py-spy-0.3.7 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydantic-1.8.2 pyparsing-2.4.7 pyrsistent-0.18.0 pyyaml-5.4.1 ray-1.4.1 redis-3.5.3 rsa-4.7.2 setuptools-57.4.0 yarl-1.6.3

    以上就是python分布式执行框架Ray的介绍,希望对大家有所帮助。更多Python学习指路:python基础教程

    本文教程操作环境:windows7系统、Python 3.9.1,DELL G3电脑。

    专题推荐:python ray
    上一篇:Python如何根据输入参数计算结果 下一篇:Python如何实现调用小程序接口

    相关文章推荐

    • 什么是网络协议• python中的去除重复项的操作• python中少见的函数map()和partial()• python的sort()排序方法• Python中的文件读写-理论知识

    全部评论我要评论

    © 2021 Python学习网 苏ICP备2021003149号-1

  • 取消发布评论
  • 

    Python学习网