freshcrate
Home > Frameworks > singer-sdk

singer-sdk

A framework for building Singer taps and targets

Description

<h1 align="center">Meltano Singer SDK</h1> <h3 align="center"> The Tap and Target SDKs are the fastest way to build custom data extractors and loaders! Taps and targets built on the SDK are automatically compliant with the <a href="https://hub.meltano.com/singer/spec">Singer Spec</a>, the de-facto open source standard for extract and load pipelines. </h3> ______________________________________________________________________ </br> <div align="center"> <img alt="Meltano Singer SDK Logo" src="https://user-images.githubusercontent.com/11428666/231584532-ffa694e6-60f9-4fd6-b2ee-5ff3e39d3ad6.svg" width="600"/> </div> </br> <div align="center"> <a href="https://meltano.com/slack"> <img alt="Meltano Community Slack" src="https://img.shields.io/badge/Slack-Join%20the%20Community-blue?logo=slack&labelColor=471E80&color=110B1E"/> </a> <a href="https://pypi.org/project/singer-sdk"> <img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/singer-sdk?label=Python%20Versions&labelColor=471E80&color=110B1E"/> </a> <a href="https://pypi.org/project/singer-sdk"> <img alt="Downloads" src="https://img.shields.io/pypi/dw/singer-sdk?label=Downloads&labelColor=471E80&color=110B1E"/> </a> </br> <a href="https://pypi.org/project/singer-sdk"> <img alt="PyPI Version" src="https://img.shields.io/pypi/v/singer-sdk?label=PyPI%20Version&labelColor=471E80&color=110B1E"/> </a> <a href="https://sdk.meltano.com/en/latest"> <img alt="Read the Docs (version)" src="https://img.shields.io/readthedocs/meltano-sdk/latest?label=Docs&labelColor=471E80&color=110B1E"> </a> <a href="https://codecov.io/gh/meltano/sdk"> <img alt="Codecov" src="https://img.shields.io/codecov/c/github/meltano/sdk?label=Coverage&labelColor=471E80&color=110B1E"> </a> <a href="https://github.com/astral-sh/uv"> <img alt="uv" src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json&labelColor=471E80&color=110B1E"/> </a> </div> ______________________________________________________________________ ## Future-proof extractors and loaders, with less code On average, developers tell us that they write about 70% less code by using the SDK, which makes learning the SDK a great investment. Furthermore, as new features and capabilities are added to the SDK, your taps and targets can always take advantage of the latest capabilities and bug fixes, simply by updating your SDK dependency to the latest version. ## Meltano _Not familiar with Meltano?_ [Meltano](https://docs.meltano.com/getting-started/meltano-at-a-glance) is your CLI for ELT+ that: - **Starts simple**: Meltano is pip-installable and comes in a prepackaged docker container, you can have your first ELT pipeline running within minutes. - **Has DataOps out-of-the-box**: Meltano provides tools that make DataOps best practices easy to use in every project. - **Integrates with everything**: 300+ natively supported data sources & targets, as well as additional plugins like great expectations or dbt are natively available. - **Is easily customizable**: Meltano isn't just extensible, it's built to be extended! The Singer SDK (for Connectors) & EDK (for Meltano Components) are easy to use. Meltano Hub helps you find all of the connectors and components created across the data community. - **Is a mature system**: Developed since 2018, runs in production at large companies like GitLab, and currently powers over a million pipeline runs monthly. - **Has first class ELT tooling built-in**: Extract data from any data source, load into any target, use inline maps to transform on data on the fly, and test the incoming data, all in one package. If you want to get started with Meltano, we suggest you: - head over to the [Installation](https://docs.meltano.com/getting-started/installation) - or if you have it installed, go through the [Meltano Tutorial](https://docs.meltano.com/getting-started/part1). ## Documentation - See our [online documentation](https://sdk.meltano.com) for instructions on how to get started with the SDK. ## Contributing back to the SDK - For more information on how to contribute, see our [Contributors Guide](https://sdk.meltano.com/en/latest/CONTRIBUTING.html). ## Making a new release of the SDK 1. Trigger a version bump [using the GitHub web UI](https://github.com/edgarrmondragon/sdk/actions/workflows/version_bump.yml) or the cli: ```console $ gh workflow run ``` The `increment: auto` option will figure out the most appropriate bump based on commit history. 1. Follow the checklist in the PR description. 1. Publish a new release [using the GitHub web UI](https://github.com/meltano/sdk/releases/new).

Release History

VersionChangesUrgencyDate
0.53.7Imported from PyPI (0.53.7)Low4/21/2026
v0.53.7## v0.53.7 (2026-04-15) ### πŸ› Fixes - [#3601](https://github.com/meltano/sdk/issues/3601) Support `simpleeval` 1.0.5+ by subclassing `simpleeval.EvalWithCompoundTypes` High4/15/2026
v0.53.6## v0.53.6 (2026-03-13) ### πŸ› Fixes - [#3562](https://github.com/meltano/sdk/issues/3562) Constrain `simpleeval` dependency to prevent issues with using `json` in stream maps expressions Low3/13/2026
v0.53.5## v0.53.5 (2026-01-06) ### πŸ› Fixes - [#3436](https://github.com/meltano/sdk/issues/3436) Do not try to parse the error response from the OAuth token request as JSON to avoid deserialization errors - [#3432](https://github.com/meltano/sdk/issues/3432) Add details to the `variant` prompt in templates - [#3426](https://github.com/meltano/sdk/issues/3426) Update template dependencies - [#3412](https://github.com/meltano/sdk/issues/3412) Add concurrency config to CI workflows in templates Low1/6/2026
v0.53.4## v0.53.4 (2025-11-28) ### πŸ› Fixes - [#3393](https://github.com/meltano/sdk/issues/3393) Make `StreamSchema.get_stream_schema` generic to avoid downstream warnings about violation of the LSP ### βš™οΈ Under the Hood - [#3391](https://github.com/meltano/sdk/issues/3391) Move to monthly instead of weekly updates to pre-commit hooks in templates Low11/29/2025
v0.53.3## v0.53.3 (2025-11-25) ### πŸ› Fixes - [#3387](https://github.com/meltano/sdk/issues/3387) Do not leave dangling `nullable` field in OpenAPI schema properties - [#3386](https://github.com/meltano/sdk/issues/3386) Do not add `"type": ["null"]` to OpenAPI spec properties without a type - [#3380](https://github.com/meltano/sdk/issues/3380) Use pytest 9+ in templates ### βš™οΈ Under the Hood - [#3388](https://github.com/meltano/sdk/issues/3388) Deprecate `Stream.reset_state_progress_markeLow11/25/2025
v0.53.2## v0.53.2 (2025-11-19) ### πŸ› Fixes - [#3378](https://github.com/meltano/sdk/issues/3378) Revert "feat(taps): Queue parent contexts and sync child streams only when the queue is full (#3058)" Low11/19/2025
v0.53.1## v0.53.1 (2025-11-19) ### πŸ› Fixes - [#3374](https://github.com/meltano/sdk/issues/3374) For log-based streams, write starting bookmark to the state to ensure the sync runs "incrementally" Low11/19/2025
v0.53.0## v0.53.0 (2025-11-18) ### ✨ New - [#3369](https://github.com/meltano/sdk/issues/3369) In OpenAPI schema normalization, treat all fields as `nullable` unless they are part of a primary key - [#3058](https://github.com/meltano/sdk/issues/3058) Queue parent contexts and sync child streams only when the queue is full - [#3364](https://github.com/meltano/sdk/issues/3364) Normalize `allOf` components in OpenAPI spec schemas - [#3363](https://github.com/meltano/sdk/issues/3363) Add `flatteniLow11/18/2025
v0.52.5## v0.52.5 (2025-10-30) ### πŸ› Fixes - [#3350](https://github.com/meltano/sdk/issues/3350) Configuration schema now reflects that casting to `bool(...)` in stream map `__filter__` expressions is not required - [#3349](https://github.com/meltano/sdk/issues/3349) Update templates Low10/30/2025
v0.52.4## v0.52.4 (2025-10-28) ### πŸ› Fixes - [#3346](https://github.com/meltano/sdk/issues/3346) Preserve properties with empty JSON schema when flattening them - [#3345](https://github.com/meltano/sdk/issues/3345) Use tuple primary keys in tap templates Low10/28/2025
v0.52.3## v0.52.3 (2025-10-28) ### πŸ› Fixes - [#3343](https://github.com/meltano/sdk/issues/3343) Fix regression in v0.52.0 where the `auth` attribute was removed from the `RESTStream.requests_session` instance - [#3333](https://github.com/meltano/sdk/issues/3333) Add more context and clear instructions to cookiecutter prompts ### πŸ“š Documentation Improvements - [#3338](https://github.com/meltano/sdk/issues/3338) Document how to cast values to different data types in inline stream maps - Low10/28/2025
v0.52.2## v0.52.2 (2025-10-22) ### πŸ› Fixes - [#3328](https://github.com/meltano/sdk/issues/3328) Support MIT as a license option in templates - [#3327](https://github.com/meltano/sdk/issues/3327) Emit a log when requesting a new access token in OAuth taps Low10/22/2025
v0.52.1## v0.52.1 (2025-10-22) ### πŸ“¦ Packaging changes - [#3325](https://github.com/meltano/sdk/issues/3325) Fix `python_version` marker in `backoff` dependency Low10/22/2025
v0.52.0## v0.52.0 (2025-10-21) ### ✨ New - [#3311](https://github.com/meltano/sdk/issues/3311) New `Stream.preprocess_context` lets developers make changes to the stream context before it is frozen ### πŸ› Fixes - [#3322](https://github.com/meltano/sdk/issues/3322) Improved error message when a request is being retried, including URL path and status code - [#3313](https://github.com/meltano/sdk/issues/3313) Ignore the `Programming Language :: Python :: 3 :: Only` classifier when generating Low10/21/2025
v0.51.0## v0.51.0 (2025-10-17) ### ✨ New - [#3307](https://github.com/meltano/sdk/issues/3307) Allow the user to override stream and sink meters (i.e. counters & timers) ### βš™οΈ Under the Hood - [#3302](https://github.com/meltano/sdk/issues/3302) In target logs, put the sink name in a trailing position Low10/17/2025
v0.50.2## v0.50.2 (2025-10-06) ### πŸ› Fixes - [#3297](https://github.com/meltano/sdk/issues/3297) Honor the `_LOG_REQUEST_METRICS` stream parameter Low10/6/2025
v0.50.1## v0.50.1 (2025-10-03) ### πŸ› Fixes - [#3295](https://github.com/meltano/sdk/issues/3295) Ensure plugins advertise `structured-logging` capability Low10/3/2025
v0.50.0## v0.50.0 (2025-10-01) ### ✨ New - [#3279](https://github.com/meltano/sdk/issues/3279) Emit structured logs and document this new capability for downstream consumers ### πŸ› Fixes - [#3287](https://github.com/meltano/sdk/issues/3287) Handle presence of `charset` in `Content-Type` header when retrieving schema from OpenAPI spec - [#3286](https://github.com/meltano/sdk/issues/3286) Only write starting replication value for incremental streams -- _**Thanks @ReubenFrankel!**_ ### βš™οΈ Low10/1/2025
v0.49.1## v0.49.1 (2025-09-14) ### πŸ› Fixes - [#3275](https://github.com/meltano/sdk/issues/3275) Add setter method to deprecated `APIAuthenticatorBase.tap_name` property Low9/14/2025
v0.49.0## v0.49.0 (2025-09-11) ### ✨ New - [#3262](https://github.com/meltano/sdk/issues/3262) By default `ObjectType` fields now use `additionalProperties: true` - [#3196](https://github.com/meltano/sdk/issues/3196) Use `ty` to check types in tap, target and mapper templates ### πŸ› Fixes - [#3257](https://github.com/meltano/sdk/issues/3257) Add `DecimalType` alias for `NumberType` ### βš™οΈ Under the Hood - [#3269](https://github.com/meltano/sdk/issues/3269) Move all SQLAlchemy-relatedLow9/11/2025
v0.48.1## v0.48.1 (2025-08-21) ### ✨ New - [#3238](https://github.com/meltano/sdk/issues/3238) Auto-detect and load YAML-formatted OpenAPI specs - [#3227](https://github.com/meltano/sdk/issues/3227) Get the supported Python versions in `--about` output from PyPI classifiers if they are available - [#3226](https://github.com/meltano/sdk/issues/3226) Declare Python 3.14 support in templates - [#3223](https://github.com/meltano/sdk/issues/3223) Add a metrics exclusion filter - [#3222](https://giLow8/21/2025
v0.48.0## v0.48.0 (2025-08-04) ### Highlights #### Schema sources The Singer SDK now provides an extensible API for loading schemas from various sources through the `SchemaSource` API. For example, ```python from typing import ClassVar from singer_sdk import OpenAPISchema, StreamSchema # Load from OpenAPI spec openapi_source = OpenAPISchema("https://api.example.com/openapi.json") class UsersStream(RESTStream): name = "users" schema: ClassVar[StreamSchema] = StreamSchemaLow8/4/2025
v0.47.4## v0.47.4 (2025-06-25) ### πŸ› Fixes - [#3138](https://github.com/meltano/sdk/issues/3138) Ensure Singer SDK warnings are logged (#3127) Low6/25/2025
v0.46.5## v0.46.5 (2025-06-24) ### πŸ› Fixes - [#3135](https://github.com/meltano/sdk/issues/3135) Ensure Singer SDK warnings are logged Low6/25/2025
v0.45.12## v0.45.12 (2025-06-24) ### πŸ› Fixes - [#3132](https://github.com/meltano/sdk/issues/3132) Ensure Singer SDK warnings are logged (#3127) Low6/24/2025
v0.44.4## v0.44.4 (2025-06-24) ### πŸ› Fixes - [#3128](https://github.com/meltano/sdk/issues/3128) Ensure Singer SDK warnings are logged ### πŸ“¦ Packaging changes - [#3129](https://github.com/meltano/sdk/issues/3129) Test with click 8.2.x Low6/24/2025
v0.47.3## v0.47.3 (2025-06-19) ### πŸ› Fixes - [#3113](https://github.com/meltano/sdk/issues/3113) Remove redundant `tool.ruff.target-version` from tap template ### πŸ“š Documentation Improvements - [#3116](https://github.com/meltano/sdk/issues/3116) Document `post_process` changes in 0.47 Low6/19/2025
v0.47.2## v0.47.2 (2025-06-17) ### πŸ› Fixes - [#3111](https://github.com/meltano/sdk/issues/3111) Ensure state progress markers in child streams are finalized - [#3108](https://github.com/meltano/sdk/issues/3108) Suppress unmapped warning for additional properties -- _**Thanks @ReubenFrankel!**_ Low6/17/2025
v0.47.1## v0.47.1 (2025-06-16) ### πŸ› Fixes - [#3106](https://github.com/meltano/sdk/issues/3106) Apply `.upper()` to `metrics_log_level` setting value - [#3102](https://github.com/meltano/sdk/issues/3102) Bump "build and inspect python package" action in templates and use the name output for the PyPI url Low6/16/2025
v0.47.0## v0.47.0 (2025-06-13) ### Highlights #### `ACTIVATE_VERSION` Taps can now emit [`ACTIVATE_VERSION`](https://hub.meltano.com/singer/docs/#activate-version) messages when the builtin `emit_activate_version_messages` config is set to `True`. #### Global log level Setting the `<APP_NAME>_LOGLEVEL` or `LOGLEVEL` environment variables will now apply that level to _all_ logs generated by taps , mappers and targets. #### `post_process` The stream `post_process` method is now execuLow6/13/2025
v0.46.4## v0.46.4 (2025-05-28) ### πŸ› Fixes - [#3066](https://github.com/meltano/sdk/issues/3066) Revert table name splitting by `-` in SQL targets when `default_target_schema` is set, introduced in #3020 Low5/28/2025
v0.46.3## v0.46.3 (2025-05-15) ### πŸ› Fixes - [#3042](https://github.com/meltano/sdk/issues/3042) Revert "fix(targets): Avoid emitting message `"No schema for record field."` when "additionalProperties" is set to true in stream schema ([#3031](https://github.com/meltano/sdk/issues/3031))" Low5/15/2025
v0.46.2## v0.46.2 (2025-05-15) ### πŸ› Fixes - [#3037](https://github.com/meltano/sdk/issues/3037) Avoid aborting parent stream before child stream is finished in standard tap tests - [#3031](https://github.com/meltano/sdk/issues/3031) Avoid emitting message `"No schema for record field."` when "additionalProperties" is set to true in stream schema -- _**Thanks @SidduHussain!**_ - [#3040](https://github.com/meltano/sdk/issues/3040) Avoid emitting empty state value from target -- _**Thanks @joaopLow5/15/2025
v0.46.1## v0.46.1 (2025-05-13) ### πŸ› Fixes - [#3034](https://github.com/meltano/sdk/issues/3034) Avoid emitting an empty state from the target if the tap has not sent any state messages - [#3030](https://github.com/meltano/sdk/issues/3030) Bump standard tap tests max records limit from 25 to 150 ### πŸ“š Documentation Improvements - [#3026](https://github.com/meltano/sdk/issues/3026) Auto-format markdown with `mdformat` ### πŸ“¦ Packaging changes - [#3033](https://github.com/meltano/sdkLow5/13/2025
v0.46.0## v0.46.0 (2025-05-06) ### ✨ New - [#2162](https://github.com/meltano/sdk/issues/2162) Expose metric dictionary to make logging metrics as JSON easier - [#2977](https://github.com/meltano/sdk/issues/2977) Limit extraction to 25 records by default in built-in tap tests - [#2989](https://github.com/meltano/sdk/issues/2989) Allow pagination to continue after an empty page ### πŸ› Fixes - [#3020](https://github.com/meltano/sdk/issues/3020) In SQL targets, use stream name as table name Low5/6/2025
v0.45.11## v0.45.11 (2025-04-28) ### πŸ› Fixes - [#3004](https://github.com/meltano/sdk/issues/3004) Ensure fields with an array of JSON schema types are properly handled by SQL targets - [#2995](https://github.com/meltano/sdk/issues/2995) Pin GitHub actions in templates - [#2987](https://github.com/meltano/sdk/issues/2987) Add a date to the warning message for removal of `Faker` from stream maps contexts - [#2982](https://github.com/meltano/sdk/issues/2982) Allow streams to override `effective_Low4/28/2025
v0.45.10## v0.45.10 (2025-04-16) ### πŸ› Fixes - [#2978](https://github.com/meltano/sdk/issues/2978) Restore private `_http_headers` attribute of REST streams Low4/16/2025
v0.45.9## v0.45.9 (2025-04-15) ### πŸ› Fixes - [#2975](https://github.com/meltano/sdk/issues/2975) Update cookiecutter templates - [#2974](https://github.com/meltano/sdk/issues/2974) Do not log a warning if `null` cannot be appended to schema types - [#2973](https://github.com/meltano/sdk/issues/2973) Recognize `oneOf` fields in Singer schema helpers Low4/15/2025
v0.45.8## v0.45.8 (2025-04-14) ### πŸ› Fixes - [#2965](https://github.com/meltano/sdk/issues/2965) `AttributeError: object has no attribute '_requests_session'` when accessing public property during stream initialization -- _**Thanks @ReubenFrankel!**_ - [#2970](https://github.com/meltano/sdk/issues/2970) Consider `oneOf` schemas when checking for `date` and `datetime` types Low4/14/2025
v0.45.7## v0.45.7 (2025-04-14) ### πŸ› Fixes - [#2968](https://github.com/meltano/sdk/issues/2968) `singer_sdk.singerlib.resolve_schema_references` now resolves `oneOf`Β schema references - [#2924](https://github.com/meltano/sdk/issues/2924) Generate standard stream metadata for nested fields - [#2940](https://github.com/meltano/sdk/issues/2940) Ensure streams with hardcoded `schema` attributes (e.g. REST and GraphQL streams) can have their schema overridden by the catalog ### πŸ“¦ Packaging chaLow4/14/2025
v0.45.6## v0.45.6 (2025-04-11) ### πŸ› Fixes - [#2961](https://github.com/meltano/sdk/issues/2961) Conform `NaN`, `Infinity` and `-Infinity` to null values ### πŸ“š Documentation Improvements - [#2960](https://github.com/meltano/sdk/issues/2960) Documented using `default=str` with `json.dumps` in stream maps - [#2955](https://github.com/meltano/sdk/issues/2955) Documented the `ignore_parent_replication_key` stream attribute Low4/11/2025
v0.45.5## v0.45.5 (2025-04-04) ### πŸ› Fixes - [#2952](https://github.com/meltano/sdk/issues/2952) Adapt field schema type to `string` when transformed with stream maps using `json.dumps` ### βš™οΈ Under the Hood - [#2948](https://github.com/meltano/sdk/issues/2948) `ignore_no_records` as an instance property in built-in tests -- _**Thanks @ReubenFrankel!**_ Low4/4/2025
v0.45.4## v0.45.4 (2025-04-01) ### πŸ› Fixes - [#2946](https://github.com/meltano/sdk/issues/2946) In the built-in tap tests, do not emit a warning for missing records on ignored streams Low4/1/2025
v0.45.3## v0.45.3 (2025-03-29) ### πŸ› Fixes - [#2937](https://github.com/meltano/sdk/issues/2937) Revert "fix: Ensure stream schema is overridden by the input catalog ([#2932](https://github.com/meltano/sdk/issues/2932))" ### πŸ“š Documentation Improvements - [#2938](https://github.com/meltano/sdk/issues/2938) Fixed the dynamic schema example Low3/29/2025
v0.45.2## v0.45.2 (2025-03-28) ### πŸ› Fixes - [#2935](https://github.com/meltano/sdk/issues/2935) Added `uv` venv backend to Meltano project in templates - [#2934](https://github.com/meltano/sdk/issues/2934) Updated cookiecutter pre-commit hooks and added `check-meltano` JSON schema check - [#2932](https://github.com/meltano/sdk/issues/2932) Ensure stream schema is overridden by the input catalog Low3/28/2025
v0.45.1## v0.45.1 (2025-03-27) ### πŸ› Fixes - [#2925](https://github.com/meltano/sdk/issues/2925) Let downstream packages import from deprecated `singer_sdk._singerlib.catalog` Low3/27/2025
v0.45.0## v0.45.0 (2025-03-25) ### Highlights #### Experimental msgspec IO Taps and targets now support custom message writing and reading interfaces, and the new (experimental) `msgspec`-based encoding is available. See the [docs](https://sdk.meltano.com/en/v0.45.0/guides/performance.html#use-a-different-message-writer-or-reader) for more information. Thanks to @BuzzCutNorman and @s7clarke10 for the thoughts and suggestions! #### `singerlib` is now a public module The low-level SingLow3/27/2025
v0.44.3## v0.44.3 (2025-02-05) ### πŸ› Fixes - [#2870](https://github.com/meltano/sdk/issues/2870) Do not emit a warning needlessly when `rest_method` is not set in a stream class Low2/5/2025
v0.44.2## v0.44.2 (2025-01-31) ### πŸ› Fixes - [#2862](https://github.com/meltano/sdk/issues/2862) Include package license metadata conditionally in templates -- _**Thanks @ReubenFrankel!**_ - [#2859](https://github.com/meltano/sdk/issues/2859) Use uv in tap, target and mapper templates ### πŸ“š Documentation Improvements - [#2861](https://github.com/meltano/sdk/issues/2861) Document plugin development with uv and how to migrate existing ones ### πŸ“¦ Packaging changes - [#2854](https://gLow1/31/2025
v0.44.1## v0.44.1 (2025-01-29) ### πŸ› Fixes - [#2847](https://github.com/meltano/sdk/issues/2847) Update Cookiecutter templates - [#2844](https://github.com/meltano/sdk/issues/2844) Avoid writing an empty state -- _**Thanks @joaopamaral!**_ - [#2843](https://github.com/meltano/sdk/issues/2843) Use SQLAlchemy to generate an insert statement Low1/30/2025

Dependencies & License Audit

Loading dependencies...

Similar Packages

azure-core-tracing-opentelemetryMicrosoft Azure Azure Core OpenTelemetry plugin Library for Pythonazure-template_0.1.0b6187637
boxsdkOfficial Box Python SDK10.6.0
alibabacloud-credentialsThe alibabacloud credentials module of alibabaCloud Python SDK.1.0.8
aws-xray-sdkThe AWS X-Ray SDK for Python (the SDK) enables Python developers to record and emit information from within their applications to the AWS X-Ray service.2.15.0
pre-commitA framework for managing and maintaining multi-language pre-commit hooks.v4.6.0