Description

# AWS SDK for pandas (awswrangler) *Pandas on AWS* Easy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).   > An [AWS Professional Service](https://aws.amazon.com/professional-services/) open source initiative | aws-proserve-opensource@amazon.com [](https://pypi.org/project/awswrangler/) [](https://anaconda.org/conda-forge/awswrangler) [](https://pypi.org/project/awswrangler/) [](https://github.com/astral-sh/ruff) [](https://opensource.org/licenses/Apache-2.0) [](http://mypy-lang.org/)  [](https://aws-sdk-pandas.readthedocs.io/?badge=latest) | Source | Downloads | Installation Command | |--------|-----------|----------------------| | **[PyPi](https://pypi.org/project/awswrangler/)** | [](https://pypi.org/project/awswrangler/) | `pip install awswrangler` | | **[Conda](https://anaconda.org/conda-forge/awswrangler)** | [](https://anaconda.org/conda-forge/awswrangler) | `conda install -c conda-forge awswrangler` | > ⚠️ **Starting version 3.0, optional modules must be installed explicitly:**<br> ➡️`pip install 'awswrangler[redshift]'` ## Table of contents - [Quick Start](#quick-start) - [At Scale](#at-scale) - [Read The Docs](#read-the-docs) - [Getting Help](#getting-help) - [Logging](#logging) ## Quick Start Installation command: `pip install awswrangler` > ⚠️ **Starting version 3.0, optional modules must be installed explicitly:**<br> ➡️`pip install 'awswrangler[redshift]'` ```py3 import awswrangler as wr import pandas as pd from datetime import datetime df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]}) # Storing data on Data Lake wr.s3.to_parquet( df=df, path="s3://bucket/dataset/", dataset=True, database="my_db", table="my_table" ) # Retrieving the data directly from Amazon S3 df = wr.s3.read_parquet("s3://bucket/dataset/", dataset=True) # Retrieving the data from Amazon Athena df = wr.athena.read_sql_query("SELECT * FROM my_table", database="my_db") # Get a Redshift connection from Glue Catalog and retrieving data from Redshift Spectrum con = wr.redshift.connect("my-glue-connection") df = wr.redshift.read_sql_query("SELECT * FROM external_schema.my_table", con=con) con.close() # Amazon Timestream Write df = pd.DataFrame({ "time": [datetime.now(), datetime.now()], "my_dimension": ["foo", "boo"], "measure": [1.0, 1.1], }) rejected_records = wr.timestream.write(df, database="sampleDB", table="sampleTable", time_col="time", measure_col="measure", dimensions_cols=["my_dimension"], ) # Amazon Timestream Query wr.timestream.query(""" SELECT time, measure_value::double, my_dimension FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3 """) ``` ## At scale AWS SDK for pandas can also run your workflows at scale by leveraging [Modin](https://modin.readthedocs.io/en/stable/) and [Ray](https://www.ray.io/). Both projects aim to speed up data workloads by distributing processing over a cluster of workers. Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.16.0/scale.html) or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to learn more. ## [Read The Docs](https://aws-sdk-pandas.readthedocs.io/) - [**What is AWS SDK for pandas?**](https://aws-sdk-pandas.readthedocs.io/en/3.16.0/about.html) - [**Install**](https://aws-sdk-pandas.readthedocs.io/en/3.16.0/install.html) - [PyPi (pip)](https://aws-sdk-pandas.readthedocs.io/en/3.16.0/install.html#pypi-pip) - [Conda](https://aws-sdk-pandas.readthedocs.io/en/3.16.0/install.html#conda) - [AWS Lambda Layer](https://aws-sdk-pandas.readthedocs.io/en/3.16.0/install.html#aws-lambda-layer) - [AWS Glue Python Shell Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.16.0/install.html#aws-glue-python-shell-jobs) - [AWS Glue PySpark Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.16.0/install.html#aws-glue-pyspark-jobs) - [Amazon SageMa