External Stage

 External Named Stage

    As we have already see unto now, we can load data from files into snowflake tables using internal Named Stage and copy command.

What is different in this External Stage and why do we need it.

    Imagine that we are working with multiple teams of business and data will be generated by them and as a Data engineer, it is your responsibility to load data into snowflake. Some time they may be big and may be it is one time process are may be once in a wile. 

Since the files are big and the data in the files may be sensitive or files movement is not suggestable.
So using external stage option is right one for us.
users can create files directly in S3 Bucket / specific folder in the S3 Bucket.
We can request access to the bucket and access files.
load data directly into our Snowflake tables. So the process is simplified.  

once the stage is established, it is  straightforward and it is matter of just running copy command that we have already created and we can either run it manually or using any programming or we can schedule it too.

create or replace STAGE emp_snowpipe_stage

STORAGE_INTEGRATION = customers_snowpipe_s3_integration <-- storage integration object

file_format = MY_csv_format<-- This name of file format

URL = 's3://customers-snowpipe-integration-tiruven/';   <-- This name of s3 bucket

What is External Stage in Snowflake :

It is multi step process.

Step 1 : Create S3 bucket 
Step 2 : IAM policy 
Step 3 : IAM Roles 
Step 4 : Create a database 
Step 5 : Create Schema 
Step 6 : Create Table 

Step 7 : Create File Format 
Step 8 : storage integration 
Step 9 : Update Trusted entities for IAM Role we created above
Step 10 : Create stage object

please refer the External Table for all steps explained