Demo entry 6709321

python

   

Submitted by anonymous on Feb 02, 2018 at 03:12
Language: Python. Code size: 524 Bytes.

# test_spark.py
import os
import sys
from pyspark import SparkContext
from pyspark import SparkConf

conf = SparkConf()
conf.setAppName("get-hosts")

sc = SparkContext(conf=conf)

def noop(x):
    import socket
    import sys
    import tensorflow as tf
    return socket.gethostname() + ' '.join(sys.path) + ' :::='.join(os.environ.get('LD_LIBRARY_PATH')) # ' :::='.join(os.environ.get('LD_LIBRARY_PATH')) + 

rdd = sc.parallelize(range(1000), 100)
hosts = rdd.map(noop).distinct().collect()
print(hosts)

This snippet took 0.00 seconds to highlight.

Back to the Entry List or Home.

Delete this entry (admin only).