Home > 108 Data Analysis > 108.40.10.30 Data Analysis - Data science computing infrastructure

Data science computing infrastructure

AWS was the only cloud provider we used at Lucky8. Within AWS we used several components

VPN
- RDS instance with OpenVPN software
- Tunnelblick was software installed on local machines, used to connect to the VPN
- Extensive notes on VPN setup in the file “Lucky8 Bastion Exploration” and “Lucky8 OpenVPN” in dropbox/notes
File upload/storage
- 108.10.20.10.10 Data Analysis - Receiving raw data
- S3
- SFTP
Database
- Redshift - minimum clusters most of the time but the ability to scale up when necessary
Jupyter
- 108.40.20.40 Data Analysis - Jupyter notebooks
- For analysis
Vagrant
- The same exact development environment for everybody
Python
- with 121.001 Python - Python virtualenv

108.40.10.30 Data Analysis - Data science computing infrastructure to 108.40.10 Data Analysis - Working environment
108.40.10.30 Data Analysis - Data science computing infrastructure to 108.10.20.10.10 Data Analysis - Receiving raw data
108.40.10.30 Data Analysis - Data science computing infrastructure to 108.40.20.40 Data Analysis - Jupyter notebooks
108.40.10.30 Data Analysis - Data science computing infrastructure to 121.001 Python - Python virtualenv
108.40.10 Data Analysis - Working environment to 108.40.10.30 Data Analysis - Data science computing infrastructure