Setting up EC2

dougieduk·2022년 4월 8일
0

SSH into the server

  • Open VS Code
  • Install Remove Development in the marketplace
  • Create a config file
Host {HostName}
	HostName {DNS Name}
	User {username on server} 
	IdentityFile {location of pem file}
  • press F1Configure SSH Hosts → Select the file
  • press F1Remote-SSH: Connect to Host

Install Conda

  • copy link from the website

  • wget https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh
    
  • update Conda

conda update --prefix /home/ubuntu/anaconda3/ anaconda
  • check the python version
which python3
  • select the conda python3 as the default version
source .bashrc

Configure Jupyter Notebook

  • create jupyter config
jupyter notebook --generate-config
mkdir certs 
cd certs 
sudo openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
  • type info

  • move to the jupyter directory
    cd ~/.jupyter/

  • vim jupyter_notebook_config.py

c = get_config()
c.NotebookApp.certfile = u'/home/ubuntu/certs/mycert.pem'
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8888
  • if you are getting the ssl_version error, add the below to jupyter_notebook_config.py
c.NotebookApp.ssl_options={
        "ssl_version": ssl.PROTOCOL_TLSv1_2
}
  • if you are getting 404 error on jupyter notebook, add the below to jupyter_notebook_config.py
c.NotebookApp.notebook_dir = u’{starting_directory}

Install Java and Scale

  • Install Java
sudo apt-get update 
sudo apt-get install default-jre
java -version
  • Install Scala
sudo apt-get install scala 
scala -version 

Install py4j

  • configure the PATH
export PATH=$PATH:$HOME/anaconda3/bin
  • install pip from conda
conda install pip
conda upgrade pip
  • install py4j
pip install py4j

Install pyspark

conda install pyspark

or

pip install pyspark 

0개의 댓글