Receiving raw data
There should be multiple ways to receive raw data. When you’re working with big companies they have to worry about regulatory compliance and can’t just send csv files over email. Small companies don’t seem to have this issue, they’ll often just send a 10 megabyte file via email.
Multiple ways to receive raw data
- SFTP was usually the best route
- Direct upload to our Dropbox account worked for a short period of time as an easy way to allow somebody to upload their data securely.
- Secure web interface for smaller files
- They place files on their servers and we get it via SFTP or SSH
- Email (least preferable)
Graph:
- 108.10.20.10.10 Data Analysis - Receiving raw data to 108.10.20 Data Analysis - Step 2 ingest clean validate
- 108.10.20.10 Data Analysis - Ingest raw data into database to 108.10.20.10.10 Data Analysis - Receiving raw data
- 108.40.10.30 Data Analysis - Data science computing infrastructure to 108.10.20.10.10 Data Analysis - Receiving raw data