Set up cluster : Spark 1.3.1 en Ubuntu 14.04

Set up cluster manually: Spark 1.3.1 en Ubuntu 14.04

Introduction

When the cluster is running let configure Spark 1.3.1 and assuming Anaconda install to provide numpy.

Set up password-less SSH

Contact each node:
ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem ubuntu@10.8.3.127
ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem ubuntu@10.8.3.128

In the master

ubuntu@master:~$ ssh-keygen -t rsa -P ""
ubuntu@master:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
chmod 644 $HOME/.ssh/authorized_keys
ubuntu@master:~$ ssh localhost
with $HOME = /home/ubuntu

On workers:

copy ~/.ssh/id_dsa.pub from your master to the worker, then use:
cat /home/ubuntu/.ssh/id_rsa.pub >> /home/ubuntu/.ssh/authorized_keys
chmod 644 /home/ubuntu/.ssh/authorized_keys


References:
http://stackoverflow.com/questions/31899837/how-to-start-apache-spark-slave-instance-on-a-standalone-environment
With Maven

Configuration file: conf/slaves file on your master

We need to add hostname to /etc/host
  1. sudo vim /etc/hosts
  2. Add Full Qualified HostName (FQHN) and  Hostname (to search for it  command “ “hostname -f”)
  3. Add your node: 192.168.0.24 p1-spark-node-001.novalocal p1-spark-node-001

Start Master and Workers

run sbin/start-all.sh
the cluster manager’s web UI should appear at
http://masternode:8080 and show all your workers.

Start Master and Workers by Hand

Run some commands in the Master and in the Workers:

Master

cd /opt/spark/bin/
./spark-class org.apache.spark.deploy.master.Master
Now you can check those urls:
192.162.0.52:7077 where you should send the jobs and 192.168.0.52:8080 the MasterUI.
I am going to visit the last one:
Remember the port should be open. As you remember 192.168.0.52 is the internal IP of my master. Tunneling:
ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem -L 8200:localhost:8080 ubuntu@10.8.3.127
Now you can check in the browser: http://localhost:8200/


Workers

cd /opt/spark/bin/
./spark-class org.apache.spark.deploy.worker.Worker spark://192.168.0.52:7077
Now we can see in the browser the inclusion of the new worker:

To submit a job

You have to have a copy of all files in the some location in master and all workers. I have copied a folder with data and a python script in /opt/spark/bin:
cd /opt/spark/bin
./spark-submit --master spark://192.168.0.52:7077 paper_cluster_spark_to_run/recommendations.py
To run it you have to open another ssh:  ssh -i /home/raf/Documents/Cloud/rvf_keele_cloud.pem ubuntu@10.8.3.127

References


  2. Karau, H., Konwinski, A., Wendell, P., & Zaharia, M. (2015). Learning spark: lightning-fast big data analysis. " O'Reilly Media, Inc.".

Comentarios

Publicar un comentario

Entradas populares de este blog

Making MongoDB remotely available.

Imagen
Summary Here I show how to make your MongoDB accessible from a different server / computer/ instance, without using ssh. Motivation I have been using MongoDB to store and query some databases. This databases has been shared with other people and in most of the cases I have managed to use it as localhost or using ssh. However when using Docker it is easy to run into problems when using ssh. It is common to access other databases such as MySQL. For example in to use MySQL in Python you could have something like this: import pymysql.cursors # Connect to the database connection = pymysql.connect( host = ' localhost ' , user = ' user ' , password = ' passwd ' , db = ' db ' , charset = ' utf8mb4 ' , cursorclass = pymysql.cursors.DictCursor) In the same line you could query your MongoDB using
Leer más

Reflecting about SIR models and some examples

Imagen
Reflecting about SIR models  and some examples Feel free to send me you thoughts either in comments or here . This only reflect my personal opinion, none of my employees at any time. Summary SIR model trendy, but they also have some caveats Looks like in Europe and the States the peak could be reach between May and July, with different amount of population affected by country given different policies implemented (so the model need time to incorporate that effect). Potential limitations: Summer could reduce prevalence. Data limitations, brief discussion below. But we are learning about Covid19, we did not have enough tests and so on. Intro After massive disruption for Covid19 everyone is concerned about what is going on and how this could evolve in the future. Massive disruption has occurred and all of us are noticing the consequences (such as compulsory stay at home measures, terrible amounts of deaths and so on). All of us want to help a bit and trying to
Leer más

Adicction and Decision Making: a brainy view.

Imagen
  Why is this interesting? To answer questions such as how can we understand addictions within our brains? Why are people more affected than others? Is Decision Making relating to Drug addiction? In Bechara 2005 the following similarities between pathing with VMPC (see below, related to the decision making areas of the brain) damage and drug addicts are pointed out : often deny, or not aware, that they have a problem. When faced with a choice that brings immediate reward, even at the risk of incurring future negative outcomes, including loss of reputation, job, and family, they appear oblivious to the consequences of their actions. The above points out that there are similarities. How could define willpower in term of geography in the brain? Willpower as proposed on Bechara 2005: “... willpower emergers from the dynamic interaction of two separate, but interacting, neural systems: and impulsive system, in which the amygdala is a critical neural structure involved in triggering the affe
Leer más