Set up cluster : Spark 1.3.1 en Ubuntu 14.04

Set up cluster manually: Spark 1.3.1 en Ubuntu 14.04

Introduction

When the cluster is running let configure Spark 1.3.1 and assuming Anaconda install to provide numpy.

Set up password-less SSH

Contact each node:
ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem ubuntu@10.8.3.127
ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem ubuntu@10.8.3.128

In the master

ubuntu@master:~$ ssh-keygen -t rsa -P ""
ubuntu@master:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
chmod 644 $HOME/.ssh/authorized_keys
ubuntu@master:~$ ssh localhost
with $HOME = /home/ubuntu

On workers:

copy ~/.ssh/id_dsa.pub from your master to the worker, then use:
cat /home/ubuntu/.ssh/id_rsa.pub >> /home/ubuntu/.ssh/authorized_keys
chmod 644 /home/ubuntu/.ssh/authorized_keys


References:
http://stackoverflow.com/questions/31899837/how-to-start-apache-spark-slave-instance-on-a-standalone-environment
With Maven

Configuration file: conf/slaves file on your master

We need to add hostname to /etc/host
  1. sudo vim /etc/hosts
  2. Add Full Qualified HostName (FQHN) and  Hostname (to search for it  command “ “hostname -f”)
  3. Add your node: 192.168.0.24 p1-spark-node-001.novalocal p1-spark-node-001

Start Master and Workers

run sbin/start-all.sh
the cluster manager’s web UI should appear at
http://masternode:8080 and show all your workers.

Start Master and Workers by Hand

Run some commands in the Master and in the Workers:

Master

cd /opt/spark/bin/
./spark-class org.apache.spark.deploy.master.Master
Now you can check those urls:
192.162.0.52:7077 where you should send the jobs and 192.168.0.52:8080 the MasterUI.
I am going to visit the last one:
Remember the port should be open. As you remember 192.168.0.52 is the internal IP of my master. Tunneling:
ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem -L 8200:localhost:8080 ubuntu@10.8.3.127
Now you can check in the browser: http://localhost:8200/


Workers

cd /opt/spark/bin/
./spark-class org.apache.spark.deploy.worker.Worker spark://192.168.0.52:7077
Now we can see in the browser the inclusion of the new worker:

To submit a job

You have to have a copy of all files in the some location in master and all workers. I have copied a folder with data and a python script in /opt/spark/bin:
cd /opt/spark/bin
./spark-submit --master spark://192.168.0.52:7077 paper_cluster_spark_to_run/recommendations.py
To run it you have to open another ssh:  ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem ubuntu@10.8.3.127

References


  2. Karau, H., Konwinski, A., Wendell, P., & Zaharia, M. (2015). Learning spark: lightning-fast big data analysis. " O'Reilly Media, Inc.".

Comentarios

Publicar un comentario

Entradas populares de este blog

Making MongoDB remotely available.

Imagen
Summary Here I show how to make your MongoDB accessible from a different server / computer/ instance, without using ssh. Motivation I have been using MongoDB to store and query some databases. This databases has been shared with other people and in most of the cases I have managed to use it as localhost or using ssh. However when using Docker it is easy to run into problems when using ssh. It is common to access other databases such as MySQL. For example in to use MySQL in Python you could have something like this: import pymysql.cursors # Connect to the database connection = pymysql.connect( host = ' localhost ' , user = ' user ' , password = ' passwd ' , db = ' db ' , charset = ' utf8mb4 ' , cursorclass = pymysql.cursors.DictCursor) In the same line you could query your MongoDB using ...
Leer más

How to create a function in python to be used from the terminal

Imagen
To write a python code able to read variables or inputs from the command line you only have to use sys.argv. Let's see a couple of examples. You can find the code here . File called example_v1.py: import sys def main(): print sys.argv if (__name__ == "__main__"): main(); Then you go to the folder where you have it and run this comman in the comman line: python example_v1.py Like here: As you can see the output is the name of the file. Let's try this: python example_v1.py hi hola ciao Basically the comand sys.argv is giving us a list of the commandline arguments received. Now I am going to create a program that is going to say hi and the name that you introduce after file name in the commandline. I called this script example_v2.py: import sys def main(): print 'Hi ' + sys.argv[1] + '!' if (__name__ == "__main__"): main(); So now we can write in the terminal: python ...
Leer más

Los mercados y las pensiones.

En estos tiempos muchos tratan a de demonizar a los mercados, en particular a los mercados de compra y venta de deuda pública, sin embargo, lo que no se dice es que estos mercados tienen en consideración y ponen a examen la economía y sostenibilidad de un país en su conjunto. Así que siendo realistas España tiene una serie de problemas  a corto, medio y largo plazo que también valoran los mercados. España tiene un sistema educativo en el que un alto porcentaje de alumnos no terminan la educación obligatoria, una universidad con poco prestigio  y proyección internacional. España dependen sobremanera del turismo (¿qué pasara cuando los países del norte de África vuelvan a ser relativamente seguros?). Nuestro inmenso paro, nuestras bajas tasas de innovación, nuestras altas tasas de pobreza relativa y creciente desarraigo. Pero no olvidemos algo mucho mayor, nuestro sistema de pensiones cuya sostenibilidad es bastante dudosa. Sobre el tema de las pensiones hay argumentos e...
Leer más