Python Cloud SDKs for Data Engineers: boto3 for AWS S3, azure-storage-blob for Azure Blob and ADLS Gen2, google-cloud-storage for GCS, Authentication, Upload, Download, List, Delete, pandas Integration, and Production Patterns

Leave a Comment / AWS, Azure, Data Engineering, Python

Table of Contents

Our File Formats post covered reading and writing local files. But in production, your data lives in the cloud — S3 buckets, Azure Blob containers, ADLS Gen2 storage accounts, or GCS buckets. This post covers the Python SDKs for all three major cloud providers, with a focus on the operations data engineers perform daily: uploading pipeline outputs, downloading source files, listing objects, reading data directly into pandas, and managing authentication securely.

Analogy — Three different airline check-in systems. S3, Azure Blob, and GCS are like Delta, United, and Air Canada — they all fly you from A to B (store and retrieve files), but each has a different check-in app (SDK), different boarding pass format (authentication), and different terminal layout (API). Once you learn the pattern (authenticate, connect, upload/download), switching between airlines is straightforward. This post teaches you all three.

Cloud Storage Comparison

FeatureAWS S3Azure Blob / ADLS Gen2Google Cloud Storage
**Python SDK**`boto3``azure-storage-blob``google-cloud-storage`
**Install**`pip install boto3``pip install azure-storage-blob azure-identity``pip install google-cloud-storage`
**Auth (dev)**AWS CLI (`aws configure`)Connection string or `az login``gcloud auth application-default login`
**Auth (prod)**IAM roles on EC2/LambdaManaged Identity + `DefaultAzureCredential`Service account key or Workload Identity
**Hierarchy**Bucket / Key (flat, prefix-simulated folders)Account / Container / Blob (flat, virtual folders)Bucket / Object (flat, prefix-simulated folders)
**Data lake layer**S3 (with Lake Formation)ADLS Gen2 (hierarchical namespace on Blob)GCS (with BigLake)
**Temp access**Presigned URLsSAS tokensSigned URLs

boto3 — AWS S3

boto3 is the AWS SDK for Python. It provides two interfaces: client (low-level, maps directly to AWS API calls) and resource (high-level, object-oriented). For data engineering, the client interface is more common because it gives you explicit control over pagination, error handling, and response parsing.

Authentication

boto3 looks for credentials in this order: (1) explicitly passed credentials, (2) environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), (3) ~/.aws/credentials file (created by aws configure), (4) IAM role attached to the compute resource (EC2 instance, Lambda function, ECS task). In production, always use IAM roles — never hardcode keys.

import boto3

# Option 1: Default credentials chain (recommended)
# Uses ~/.aws/credentials locally, IAM role in production
s3 = boto3.client("s3")

# Option 2: Explicit credentials (dev/testing only -- NEVER in production)
s3 = boto3.client(
    "s3",
    aws_access_key_id="AKIA...",
    aws_secret_access_key="...",
    region_name="us-east-1"
)

# Option 3: Named profile from ~/.aws/credentials
session = boto3.Session(profile_name="data-engineering")
s3 = session.client("s3")

Upload Files to S3

import boto3

s3 = boto3.client("s3")

# Upload a local file
s3.upload_file(
    Filename="output/orders.parquet",
    Bucket="my-data-lake",
    Key="bronze/orders/2026/07/19/orders.parquet"
)

# Upload in-memory data (bytes or string)
import json
data = json.dumps({"status": "success", "rows": 15000})
s3.put_object(
    Bucket="my-data-lake",
    Key="logs/pipeline_result.json",
    Body=data.encode("utf-8"),
    ContentType="application/json"
)

# Upload a pandas DataFrame directly as Parquet (without saving to disk)
import pandas as pd
import io

df = pd.DataFrame({"order_id": [1, 2, 3], "amount": [29.99, 49.99, 9.99]})
buffer = io.BytesIO()
df.to_parquet(buffer, index=False)
buffer.seek(0)
s3.put_object(Bucket="my-data-lake", Key="silver/orders.parquet", Body=buffer.getvalue())

Download Files from S3

import boto3
import pandas as pd
import io

s3 = boto3.client("s3")

# Download to a local file
s3.download_file(
    Bucket="my-data-lake",
    Key="bronze/orders/2026/07/19/orders.parquet",
    Filename="local_orders.parquet"
)

# Read directly into pandas (without saving to disk)
obj = s3.get_object(Bucket="my-data-lake", Key="bronze/orders.csv")
df = pd.read_csv(obj["Body"])
print(f"Loaded {len(df)} rows from S3")

# Read Parquet directly into pandas from S3
obj = s3.get_object(Bucket="my-data-lake", Key="silver/orders.parquet")
df = pd.read_parquet(io.BytesIO(obj["Body"].read()))

# Read using the s3:// URL (requires s3fs installed: pip install s3fs)
df = pd.read_parquet("s3://my-data-lake/silver/orders.parquet")
df = pd.read_csv("s3://my-data-lake/bronze/orders.csv")

List Objects with Pagination

S3 returns a maximum of 1,000 objects per API call. For buckets with more objects, you must paginate. The get_paginator method handles this automatically.

import boto3

s3 = boto3.client("s3")

# List all objects under a prefix (simulated folder)
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket="my-data-lake", Prefix="bronze/orders/2026/"):
    for obj in page.get("Contents", []):
        print(f"  {obj['Key']}  ({obj['Size']} bytes, modified {obj['LastModified']})")

# Collect all keys into a list
all_keys = []
for page in paginator.paginate(Bucket="my-data-lake", Prefix="bronze/orders/"):
    for obj in page.get("Contents", []):
        all_keys.append(obj["Key"])
print(f"Found {len(all_keys)} objects")

Delete Objects

import boto3

s3 = boto3.client("s3")

# Delete a single object
s3.delete_object(Bucket="my-data-lake", Key="temp/scratch.csv")

# Delete multiple objects (batch delete -- up to 1000 per call)
objects_to_delete = [
    {"Key": "temp/file1.csv"},
    {"Key": "temp/file2.csv"},
    {"Key": "temp/file3.csv"}
]
s3.delete_objects(
    Bucket="my-data-lake",
    Delete={"Objects": objects_to_delete}
)

Presigned URLs — Temporary Access

Presigned URLs let you share a private S3 object temporarily without exposing your credentials. The URL expires after a specified duration.

import boto3

s3 = boto3.client("s3")

# Generate a download URL valid for 1 hour
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-data-lake", "Key": "reports/monthly_summary.pdf"},
    ExpiresIn=3600  # seconds
)
print(f"Share this link (expires in 1 hour): {url}")

# Generate an upload URL (let external systems push files to your bucket)
upload_url = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": "my-data-lake", "Key": "vendor_drops/incoming.csv"},
    ExpiresIn=3600
)

azure-storage-blob — Azure Blob Storage and ADLS Gen2

The azure-storage-blob package provides three client classes that form a hierarchy: BlobServiceClient (account level), ContainerClient (container level), and BlobClient (individual blob level). For ADLS Gen2, the same SDK works — ADLS Gen2 is Blob Storage with hierarchical namespace enabled.

Authentication

Azure offers multiple authentication methods. For local development, use az login with DefaultAzureCredential. For production, use Managed Identity (no credentials to store). Connection strings and SAS tokens are also supported but less secure.

from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential

# Option 1: DefaultAzureCredential (recommended -- works locally and in production)
# Locally: uses az login credentials
# In Azure: uses Managed Identity automatically
credential = DefaultAzureCredential()
blob_service = BlobServiceClient(
    account_url="https://mystorageaccount.blob.core.windows.net",
    credential=credential
)

# Option 2: Connection string (dev/testing -- stores account key)
conn_str = "DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=...;EndpointSuffix=core.windows.net"
blob_service = BlobServiceClient.from_connection_string(conn_str)

# Option 3: SAS token (temporary access, similar to presigned URLs)
sas_url = "https://mystorageaccount.blob.core.windows.net/mycontainer?sv=2023-01-03&ss=b&srt=sco&sp=rwdlacupiytfx&se=2026-12-31&sig=..."
blob_service = BlobServiceClient(account_url=sas_url)

Upload Files to Azure Blob

from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
blob_service = BlobServiceClient(
    account_url="https://mystorageaccount.blob.core.windows.net",
    credential=credential
)
container = blob_service.get_container_client("data-lake")

# Upload a local file
with open("output/orders.parquet", "rb") as data:
    container.upload_blob(
        name="bronze/orders/2026/07/19/orders.parquet",
        data=data,
        overwrite=True  # Overwrite if blob already exists
    )

# Upload in-memory data
import json
log_data = json.dumps({"status": "success", "rows": 15000})
container.upload_blob(
    name="logs/pipeline_result.json",
    data=log_data.encode("utf-8"),
    overwrite=True
)

# Upload a pandas DataFrame directly as Parquet
import pandas as pd
import io

df = pd.DataFrame({"order_id": [1, 2, 3], "amount": [29.99, 49.99, 9.99]})
buffer = io.BytesIO()
df.to_parquet(buffer, index=False)
buffer.seek(0)
container.upload_blob(name="silver/orders.parquet", data=buffer, overwrite=True)

Download Files from Azure Blob

from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential
import pandas as pd
import io

credential = DefaultAzureCredential()
blob_service = BlobServiceClient(
    account_url="https://mystorageaccount.blob.core.windows.net",
    credential=credential
)
container = blob_service.get_container_client("data-lake")

# Download to a local file
blob_client = container.get_blob_client("bronze/orders/orders.parquet")
with open("local_orders.parquet", "wb") as f:
    stream = blob_client.download_blob()
    f.write(stream.readall())

# Read directly into pandas (without saving to disk)
blob_client = container.get_blob_client("bronze/orders.csv")
stream = blob_client.download_blob()
df = pd.read_csv(io.BytesIO(stream.readall()))
print(f"Loaded {len(df)} rows from Azure Blob")

# Read Parquet directly into pandas
blob_client = container.get_blob_client("silver/orders.parquet")
stream = blob_client.download_blob()
df = pd.read_parquet(io.BytesIO(stream.readall()))

# Read using the abfss:// URL (requires adlfs installed: pip install adlfs)
df = pd.read_parquet("abfss://data-lake@mystorageaccount.dfs.core.windows.net/silver/orders.parquet",
                      storage_options={"account_name": "mystorageaccount", "credential": credential})

List Blobs

from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
blob_service = BlobServiceClient(
    account_url="https://mystorageaccount.blob.core.windows.net",
    credential=credential
)
container = blob_service.get_container_client("data-lake")

# List all blobs under a prefix (pagination is handled automatically)
for blob in container.list_blobs(name_starts_with="bronze/orders/2026/"):
    print(f"  {blob.name}  ({blob.size} bytes, modified {blob.last_modified})")

# Collect all blob names into a list
all_blobs = [
    blob.name
    for blob in container.list_blobs(name_starts_with="bronze/orders/")
]
print(f"Found {len(all_blobs)} blobs")

Delete Blobs

from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
blob_service = BlobServiceClient(
    account_url="https://mystorageaccount.blob.core.windows.net",
    credential=credential
)
container = blob_service.get_container_client("data-lake")

# Delete a single blob
container.delete_blob("temp/scratch.csv")

# Delete all blobs under a prefix
for blob in container.list_blobs(name_starts_with="temp/"):
    container.delete_blob(blob.name)
    print(f"  Deleted {blob.name}")

google-cloud-storage — Google Cloud Storage

The google-cloud-storage library follows the same pattern as the other SDKs. Authentication uses Application Default Credentials (ADC), which automatically picks up credentials from gcloud auth locally or from the service account attached to the compute resource in GCP.

from google.cloud import storage
import pandas as pd
import io

# Authentication: uses Application Default Credentials
# Locally: run gcloud auth application-default login
# In GCP: uses the attached service account automatically
client = storage.Client()
bucket = client.bucket("my-data-lake")

# Upload a local file
blob = bucket.blob("bronze/orders/2026/07/19/orders.parquet")
blob.upload_from_filename("output/orders.parquet")

# Upload in-memory data
blob = bucket.blob("logs/pipeline_result.json")
blob.upload_from_string('{"status": "success"}', content_type="application/json")

# Download to a local file
blob = bucket.blob("bronze/orders/orders.csv")
blob.download_to_filename("local_orders.csv")

# Read directly into pandas
blob = bucket.blob("silver/orders.parquet")
content = blob.download_as_bytes()
df = pd.read_parquet(io.BytesIO(content))

# List objects
for blob in client.list_blobs("my-data-lake", prefix="bronze/orders/"):
    print(f"  {blob.name}  ({blob.size} bytes)")

# Delete
blob = bucket.blob("temp/scratch.csv")
blob.delete()

# Read using the gs:// URL (requires gcsfs installed: pip install gcsfs)
df = pd.read_parquet("gs://my-data-lake/silver/orders.parquet")

Cross-Cloud Operations Cheat Sheet

OperationAWS S3 (boto3)Azure BlobGCS
**Create client**`boto3.client(“s3”)``BlobServiceClient(url, credential)``storage.Client()`
**Upload file**`s3.upload_file(file, bucket, key)``container.upload_blob(name, data)``blob.upload_from_filename(file)`
**Upload bytes**`s3.put_object(Body=data)``container.upload_blob(name, data)``blob.upload_from_string(data)`
**Download file**`s3.download_file(bucket, key, file)``blob_client.download_blob().readall()``blob.download_to_filename(file)`
**Read into pandas**`pd.read_csv(s3.get_object(…)[“Body”])``pd.read_csv(BytesIO(stream.readall()))``pd.read_parquet(BytesIO(blob.download_as_bytes()))`
**List objects**`paginator.paginate(Prefix=…)``container.list_blobs(name_starts_with=…)``client.list_blobs(bucket, prefix=…)`
**Delete**`s3.delete_object(Bucket, Key)``container.delete_blob(name)``blob.delete()`
**pandas URL**`s3://bucket/key` (needs s3fs)`abfss://container@account.dfs.core.windows.net/path` (needs adlfs)`gs://bucket/key` (needs gcsfs)

Production Patterns

Pattern 1: Cloud-to-Local ETL

Download files from cloud storage, process locally, and upload results back.

import boto3
import pandas as pd
import io
from pathlib import Path

def s3_csv_to_parquet(bucket, prefix, output_prefix):
    # Download all CSVs under a prefix, transform, upload as Parquet.
    s3 = boto3.client("s3")

    # List all CSV files
    paginator = s3.get_paginator("list_objects_v2")
    csv_keys = []
    for page in paginator.paginate(Bucket=bucket, Prefix=prefix):
        for obj in page.get("Contents", []):
            if obj["Key"].endswith(".csv"):
                csv_keys.append(obj["Key"])

    print(f"Found {len(csv_keys)} CSV files")

    for key in csv_keys:
        # Download and read
        obj = s3.get_object(Bucket=bucket, Key=key)
        df = pd.read_csv(obj["Body"])

        # Transform
        df.columns = df.columns.str.strip().str.lower().str.replace(r"[^a-z0-9]+", "_", regex=True)
        df = df.dropna(how="all")

        # Upload as Parquet
        parquet_key = key.replace(prefix, output_prefix).replace(".csv", ".parquet")
        buffer = io.BytesIO()
        df.to_parquet(buffer, index=False, compression="snappy")
        buffer.seek(0)
        s3.put_object(Bucket=bucket, Key=parquet_key, Body=buffer.getvalue())
        print(f"  {key} -> {parquet_key} ({len(df)} rows)")

s3_csv_to_parquet("my-data-lake", "bronze/vendor/", "silver/vendor/")

Pattern 2: Partitioned Upload

Upload a DataFrame partitioned by date to cloud storage (Hive-style directory layout).

import boto3
import pandas as pd
import io

def upload_partitioned(df, bucket, base_key, partition_col):
    # Upload a DataFrame partitioned by a column to S3.
    s3 = boto3.client("s3")

    for partition_value, group_df in df.groupby(partition_col):
        key = f"{base_key}/{partition_col}={partition_value}/data.parquet"
        buffer = io.BytesIO()
        group_df.to_parquet(buffer, index=False)
        buffer.seek(0)
        s3.put_object(Bucket=bucket, Key=key, Body=buffer.getvalue())
        print(f"  Uploaded {len(group_df)} rows to {key}")

# Usage
df = pd.DataFrame({
    "order_id": range(1000),
    "region": ["Ontario"] * 400 + ["Quebec"] * 300 + ["Alberta"] * 300,
    "amount": [29.99] * 1000
})
upload_partitioned(df, "my-data-lake", "silver/orders", "region")
# Creates:
#   silver/orders/region=Ontario/data.parquet
#   silver/orders/region=Quebec/data.parquet
#   silver/orders/region=Alberta/data.parquet

Pattern 3: Cross-Cloud Copy

Copy data from AWS S3 to Azure Blob (or vice versa) using Python as the bridge.

import boto3
from azure.storage.blob import BlobServiceClient
from azure.identity import DefaultAzureCredential
import io

def copy_s3_to_azure(s3_bucket, s3_key, azure_account, azure_container, azure_blob_name):
    # Copy a file from S3 to Azure Blob Storage.
    # Download from S3
    s3 = boto3.client("s3")
    obj = s3.get_object(Bucket=s3_bucket, Key=s3_key)
    data = obj["Body"].read()

    # Upload to Azure
    credential = DefaultAzureCredential()
    blob_service = BlobServiceClient(
        account_url=f"https://{azure_account}.blob.core.windows.net",
        credential=credential
    )
    container = blob_service.get_container_client(azure_container)
    container.upload_blob(name=azure_blob_name, data=data, overwrite=True)

    print(f"Copied s3://{s3_bucket}/{s3_key} to Azure {azure_container}/{azure_blob_name}")

copy_s3_to_azure(
    "source-bucket", "data/orders.parquet",
    "mystorageaccount", "data-lake", "imported/orders.parquet"
)

Pattern 4: Reusable Cloud Storage Manager

A single class that abstracts cloud storage operations, so your pipeline code does not need to know which cloud provider it is using.

import io
import pandas as pd

class CloudStorage:
    # Unified interface for S3, Azure Blob, and GCS.

    def __init__(self, provider, **kwargs):
        self.provider = provider
        if provider == "s3":
            import boto3
            self.client = boto3.client("s3")
            self.bucket = kwargs["bucket"]
        elif provider == "azure":
            from azure.storage.blob import BlobServiceClient
            from azure.identity import DefaultAzureCredential
            blob_service = BlobServiceClient(
                account_url=kwargs["account_url"],
                credential=DefaultAzureCredential()
            )
            self.container = blob_service.get_container_client(kwargs["container"])
        elif provider == "gcs":
            from google.cloud import storage
            self.gcs_client = storage.Client()
            self.gcs_bucket = self.gcs_client.bucket(kwargs["bucket"])

    def upload_df(self, df, path, format="parquet"):
        buffer = io.BytesIO()
        if format == "parquet":
            df.to_parquet(buffer, index=False)
        elif format == "csv":
            df.to_csv(buffer, index=False)
        buffer.seek(0)

        if self.provider == "s3":
            self.client.put_object(Bucket=self.bucket, Key=path, Body=buffer.getvalue())
        elif self.provider == "azure":
            self.container.upload_blob(name=path, data=buffer, overwrite=True)
        elif self.provider == "gcs":
            blob = self.gcs_bucket.blob(path)
            blob.upload_from_file(buffer)

    def download_df(self, path, format="parquet"):
        if self.provider == "s3":
            obj = self.client.get_object(Bucket=self.bucket, Key=path)
            data = obj["Body"].read()
        elif self.provider == "azure":
            blob_client = self.container.get_blob_client(path)
            data = blob_client.download_blob().readall()
        elif self.provider == "gcs":
            blob = self.gcs_bucket.blob(path)
            data = blob.download_as_bytes()

        if format == "parquet":
            return pd.read_parquet(io.BytesIO(data))
        elif format == "csv":
            return pd.read_csv(io.BytesIO(data))


# Usage -- same code works for any cloud
storage = CloudStorage("s3", bucket="my-data-lake")
# storage = CloudStorage("azure", account_url="https://myaccount.blob.core.windows.net", container="data-lake")
# storage = CloudStorage("gcs", bucket="my-data-lake")

storage.upload_df(df, "silver/orders.parquet")
result = storage.download_df("silver/orders.parquet")

Common Mistakes

1. Hardcoding access keys in scripts. AWS access keys, Azure connection strings, and GCP service account keys in source code end up in Git history and logs. Use environment variables, credential files (~/.aws/credentials), or cloud-native authentication (IAM roles, Managed Identity, Workload Identity).

2. Not paginating when listing objects. S3 returns a maximum of 1,000 objects per list_objects_v2 call. Azure Blob handles pagination automatically in its iterator. If you use list_objects (v1) or list_objects_v2 without a paginator, you silently miss objects beyond the first 1,000.

3. Downloading large files to disk before reading into pandas. For files that fit in memory, read directly into a BytesIO buffer and pass to pd.read_csv() or pd.read_parquet(). This skips disk I/O entirely and is significantly faster for files under 1 GB.

4. Not setting overwrite=True in Azure Blob uploads. By default, upload_blob() raises a ResourceExistsError if the blob already exists. Pipeline reruns will fail unless you set overwrite=True. S3’s put_object and GCS’s upload_from_filename overwrite by default.

5. Using the S3 resource interface for high-throughput operations. The boto3.resource("s3") interface is convenient but slower than the boto3.client("s3") interface for bulk operations because it adds an extra abstraction layer. Use the client interface for data engineering pipelines.

6. Not handling NoSuchKey / BlobNotFound errors. Attempting to download a non-existent object raises an exception. Always wrap downloads in try/except and handle the “not found” case gracefully, especially in pipelines that process files that may or may not exist.

7. Uploading DataFrames by saving to disk first. Writing a DataFrame to a local file, then uploading the file, then deleting the local file is three operations. Upload directly from a BytesIO buffer — it is one operation, uses no disk, and is cleaner.

8. Not using cloud-native URLs with pandas. pandas can read directly from s3://, abfss://, and gs:// URLs with the right filesystem library installed (s3fs, adlfs, gcsfs). This eliminates manual download-and-read code entirely.

Interview Questions

Q: What is the difference between boto3 client and resource interfaces? A: The client interface is low-level — it maps directly to AWS API calls (one method per API action) and returns raw dictionaries. The resource interface is high-level — it provides object-oriented abstractions (Bucket objects, Object objects) with convenience methods. For data engineering, the client interface is preferred because it gives explicit control over pagination, error handling, and is faster for bulk operations. The resource interface is better for interactive exploration and simple scripts.

Q: How does authentication work differently across AWS, Azure, and GCP? A: All three follow a credential chain pattern. AWS (boto3) checks explicit credentials, then environment variables, then ~/.aws/credentials, then IAM role on the compute resource. Azure checks explicitly passed credentials, then environment variables, then az login, then Managed Identity. GCP checks GOOGLE_APPLICATION_CREDENTIALS environment variable, then gcloud auth, then the attached service account. In production, all three recommend using the compute-attached identity (IAM role, Managed Identity, Workload Identity) to avoid storing credentials.

Q: How do you read a Parquet file from S3 directly into a pandas DataFrame? A: Two approaches. First, using the boto3 client: call s3.get_object(), read the Body into a BytesIO buffer, and pass to pd.read_parquet(BytesIO(data)). Second, using the s3fs filesystem (install s3fs): call pd.read_parquet("s3://bucket/key") directly. The second approach is simpler but requires the s3fs library. Both avoid saving to disk.

Q: What is a presigned URL in S3 and what is its Azure equivalent? A: A presigned URL is a temporary, signed URL that grants time-limited access to a private S3 object without sharing credentials. You generate it with s3.generate_presigned_url() and set an expiration. Anyone with the URL can download (or upload, for presigned PUT) until it expires. Azure’s equivalent is a SAS (Shared Access Signature) token — a query string appended to the blob URL that grants temporary access with specified permissions and expiration.

Q: How would you copy data from AWS S3 to Azure Blob Storage? A: Download from S3 into memory (BytesIO buffer) using boto3, then upload to Azure Blob using the azure-storage-blob SDK. For large files, stream the download instead of loading the entire file into memory. For production pipelines with many files, use ThreadPoolExecutor to parallelize the copies. For very large migrations, consider using Azure Data Factory or AWS DataSync instead of a custom Python script.

Q: What is ADLS Gen2 and how does it relate to Azure Blob Storage? A: ADLS Gen2 (Azure Data Lake Storage Gen2) is Azure Blob Storage with a hierarchical namespace enabled. This means it supports true directory operations (rename a folder renames all files inside, permissions can be set at the directory level). The same azure-storage-blob Python SDK works for both — the API is identical. The difference is at the storage account level: when you create the account, you enable “hierarchical namespace” for ADLS Gen2 behavior. For data lake workloads, always use ADLS Gen2.

Q: How do you handle authentication securely in a production Python pipeline running in the cloud? A: Use the cloud provider’s compute-attached identity. On AWS EC2 or Lambda, attach an IAM role with the minimum required S3 permissions. On Azure Functions or Databricks, enable Managed Identity and grant it the “Storage Blob Data Contributor” role. On GCP, use the default service account or Workload Identity. The SDK’s default credential chain automatically picks up these identities with zero configuration in the code. Never store access keys, connection strings, or service account keys in environment variables or files on the compute resource.

Wrapping Up

Cloud storage SDKs are the connective tissue of data engineering. Every pipeline starts with reading data from cloud storage and ends with writing results back. Master the pattern for one cloud (authenticate, connect, upload/download/list) and the other two are a syntax change. Use cloud-native authentication in production, read directly into pandas when files fit in memory, and paginate when listing objects. The cross-cloud CloudStorage class pattern keeps your pipeline code portable across providers.

Related posts: — File Formats (CSV, JSON, Parquet, Excel) — AWS S3 for Data Engineers — ADLS Gen2 Complete Guide — Database Connections (SQLAlchemy, pyodbc)

Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top
Share via
Facebook
X (Twitter)
LinkedIn
Mix
Email
Print
Copy Link
Copy link
CopyCopied