SSH into the server

• Open VS Code
• Install Remove Development in the marketplace
  
• Create a config file

Host {HostName}
	HostName {DNS Name}
	User {username on server} 
	IdentityFile {location of pem file}

• press F1 → Configure SSH Hosts → Select the file
• press F1 → Remote-SSH: Connect to Host

Install Conda

• copy link from the website
  

• wget https://repo.anaconda.com/archive/Anaconda3-2021.11-Linux-x86_64.sh
  

• update Conda

conda update --prefix /home/ubuntu/anaconda3/ anaconda

• check the python version

which python3

• select the conda python3 as the default version

source .bashrc

Configure Jupyter Notebook

• create jupyter config

jupyter notebook --generate-config
mkdir certs 
cd certs 
sudo openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem

• type info
  

• move to the jupyter directory
  cd ~/.jupyter/

• vim jupyter_notebook_config.py

c = get_config()
c.NotebookApp.certfile = u'/home/ubuntu/certs/mycert.pem'
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8888

• if you are getting the ssl_version error, add the below to jupyter_notebook_config.py

c.NotebookApp.ssl_options={
        "ssl_version": ssl.PROTOCOL_TLSv1_2
}

• if you are getting 404 error on jupyter notebook, add the below to jupyter_notebook_config.py

c.NotebookApp.notebook_dir = u’{starting_directory}’

Install Java and Scale

• Install Java

sudo apt-get update 
sudo apt-get install default-jre
java -version

• Install Scala

sudo apt-get install scala 
scala -version 

Install py4j

• configure the PATH

export PATH=$PATH:$HOME/anaconda3/bin

• install pip from conda

conda install pip
conda upgrade pip

• install py4j

pip install py4j

Install pyspark

conda install pyspark

or

pip install pyspark