Description
<h1 align="center">Meltano Singer SDK</h1> <h3 align="center"> The Tap and Target SDKs are the fastest way to build custom data extractors and loaders! Taps and targets built on the SDK are automatically compliant with the <a href="https://hub.meltano.com/singer/spec">Singer Spec</a>, the de-facto open source standard for extract and load pipelines. </h3> ______________________________________________________________________ </br> <div align="center"> <img alt="Meltano Singer SDK Logo" src="https://user-images.githubusercontent.com/11428666/231584532-ffa694e6-60f9-4fd6-b2ee-5ff3e39d3ad6.svg" width="600"/> </div> </br> <div align="center"> <a href="https://meltano.com/slack"> <img alt="Meltano Community Slack" src="https://img.shields.io/badge/Slack-Join%20the%20Community-blue?logo=slack&labelColor=471E80&color=110B1E"/> </a> <a href="https://pypi.org/project/singer-sdk"> <img alt="Python Versions" src="https://img.shields.io/pypi/pyversions/singer-sdk?label=Python%20Versions&labelColor=471E80&color=110B1E"/> </a> <a href="https://pypi.org/project/singer-sdk"> <img alt="Downloads" src="https://img.shields.io/pypi/dw/singer-sdk?label=Downloads&labelColor=471E80&color=110B1E"/> </a> </br> <a href="https://pypi.org/project/singer-sdk"> <img alt="PyPI Version" src="https://img.shields.io/pypi/v/singer-sdk?label=PyPI%20Version&labelColor=471E80&color=110B1E"/> </a> <a href="https://sdk.meltano.com/en/latest"> <img alt="Read the Docs (version)" src="https://img.shields.io/readthedocs/meltano-sdk/latest?label=Docs&labelColor=471E80&color=110B1E"> </a> <a href="https://codecov.io/gh/meltano/sdk"> <img alt="Codecov" src="https://img.shields.io/codecov/c/github/meltano/sdk?label=Coverage&labelColor=471E80&color=110B1E"> </a> <a href="https://github.com/astral-sh/uv"> <img alt="uv" src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json&labelColor=471E80&color=110B1E"/> </a> </div> ______________________________________________________________________ ## Future-proof extractors and loaders, with less code On average, developers tell us that they write about 70% less code by using the SDK, which makes learning the SDK a great investment. Furthermore, as new features and capabilities are added to the SDK, your taps and targets can always take advantage of the latest capabilities and bug fixes, simply by updating your SDK dependency to the latest version. ## Meltano _Not familiar with Meltano?_ [Meltano](https://docs.meltano.com/getting-started/meltano-at-a-glance) is your CLI for ELT+ that: - **Starts simple**: Meltano is pip-installable and comes in a prepackaged docker container, you can have your first ELT pipeline running within minutes. - **Has DataOps out-of-the-box**: Meltano provides tools that make DataOps best practices easy to use in every project. - **Integrates with everything**: 300+ natively supported data sources & targets, as well as additional plugins like great expectations or dbt are natively available. - **Is easily customizable**: Meltano isn't just extensible, it's built to be extended! The Singer SDK (for Connectors) & EDK (for Meltano Components) are easy to use. Meltano Hub helps you find all of the connectors and components created across the data community. - **Is a mature system**: Developed since 2018, runs in production at large companies like GitLab, and currently powers over a million pipeline runs monthly. - **Has first class ELT tooling built-in**: Extract data from any data source, load into any target, use inline maps to transform on data on the fly, and test the incoming data, all in one package. If you want to get started with Meltano, we suggest you: - head over to the [Installation](https://docs.meltano.com/getting-started/installation) - or if you have it installed, go through the [Meltano Tutorial](https://docs.meltano.com/getting-started/part1). ## Documentation - See our [online documentation](https://sdk.meltano.com) for instructions on how to get started with the SDK. ## Contributing back to the SDK - For more information on how to contribute, see our [Contributors Guide](https://sdk.meltano.com/en/latest/CONTRIBUTING.html). ## Making a new release of the SDK 1. Trigger a version bump [using the GitHub web UI](https://github.com/edgarrmondragon/sdk/actions/workflows/version_bump.yml) or the cli: ```console $ gh workflow run ``` The `increment: auto` option will figure out the most appropriate bump based on commit history. 1. Follow the checklist in the PR description. 1. Publish a new release [using the GitHub web UI](https://github.com/meltano/sdk/releases/new).
Release History
| Version | Changes | Urgency | Date |
|---|---|---|---|
| 0.53.7 | Imported from PyPI (0.53.7) | Low | 4/21/2026 |
| v0.53.7 | ## v0.53.7 (2026-04-15) ### π Fixes - [#3601](https://github.com/meltano/sdk/issues/3601) Support `simpleeval` 1.0.5+ by subclassing `simpleeval.EvalWithCompoundTypes` | High | 4/15/2026 |
| v0.53.6 | ## v0.53.6 (2026-03-13) ### π Fixes - [#3562](https://github.com/meltano/sdk/issues/3562) Constrain `simpleeval` dependency to prevent issues with using `json` in stream maps expressions | Low | 3/13/2026 |
| v0.53.5 | ## v0.53.5 (2026-01-06) ### π Fixes - [#3436](https://github.com/meltano/sdk/issues/3436) Do not try to parse the error response from the OAuth token request as JSON to avoid deserialization errors - [#3432](https://github.com/meltano/sdk/issues/3432) Add details to the `variant` prompt in templates - [#3426](https://github.com/meltano/sdk/issues/3426) Update template dependencies - [#3412](https://github.com/meltano/sdk/issues/3412) Add concurrency config to CI workflows in templates | Low | 1/6/2026 |
| v0.53.4 | ## v0.53.4 (2025-11-28) ### π Fixes - [#3393](https://github.com/meltano/sdk/issues/3393) Make `StreamSchema.get_stream_schema` generic to avoid downstream warnings about violation of the LSP ### βοΈ Under the Hood - [#3391](https://github.com/meltano/sdk/issues/3391) Move to monthly instead of weekly updates to pre-commit hooks in templates | Low | 11/29/2025 |
| v0.53.3 | ## v0.53.3 (2025-11-25) ### π Fixes - [#3387](https://github.com/meltano/sdk/issues/3387) Do not leave dangling `nullable` field in OpenAPI schema properties - [#3386](https://github.com/meltano/sdk/issues/3386) Do not add `"type": ["null"]` to OpenAPI spec properties without a type - [#3380](https://github.com/meltano/sdk/issues/3380) Use pytest 9+ in templates ### βοΈ Under the Hood - [#3388](https://github.com/meltano/sdk/issues/3388) Deprecate `Stream.reset_state_progress_marke | Low | 11/25/2025 |
| v0.53.2 | ## v0.53.2 (2025-11-19) ### π Fixes - [#3378](https://github.com/meltano/sdk/issues/3378) Revert "feat(taps): Queue parent contexts and sync child streams only when the queue is full (#3058)" | Low | 11/19/2025 |
| v0.53.1 | ## v0.53.1 (2025-11-19) ### π Fixes - [#3374](https://github.com/meltano/sdk/issues/3374) For log-based streams, write starting bookmark to the state to ensure the sync runs "incrementally" | Low | 11/19/2025 |
| v0.53.0 | ## v0.53.0 (2025-11-18) ### β¨ New - [#3369](https://github.com/meltano/sdk/issues/3369) In OpenAPI schema normalization, treat all fields as `nullable` unless they are part of a primary key - [#3058](https://github.com/meltano/sdk/issues/3058) Queue parent contexts and sync child streams only when the queue is full - [#3364](https://github.com/meltano/sdk/issues/3364) Normalize `allOf` components in OpenAPI spec schemas - [#3363](https://github.com/meltano/sdk/issues/3363) Add `flatteni | Low | 11/18/2025 |
| v0.52.5 | ## v0.52.5 (2025-10-30) ### π Fixes - [#3350](https://github.com/meltano/sdk/issues/3350) Configuration schema now reflects that casting to `bool(...)` in stream map `__filter__` expressions is not required - [#3349](https://github.com/meltano/sdk/issues/3349) Update templates | Low | 10/30/2025 |
| v0.52.4 | ## v0.52.4 (2025-10-28) ### π Fixes - [#3346](https://github.com/meltano/sdk/issues/3346) Preserve properties with empty JSON schema when flattening them - [#3345](https://github.com/meltano/sdk/issues/3345) Use tuple primary keys in tap templates | Low | 10/28/2025 |
| v0.52.3 | ## v0.52.3 (2025-10-28) ### π Fixes - [#3343](https://github.com/meltano/sdk/issues/3343) Fix regression in v0.52.0 where the `auth` attribute was removed from the `RESTStream.requests_session` instance - [#3333](https://github.com/meltano/sdk/issues/3333) Add more context and clear instructions to cookiecutter prompts ### π Documentation Improvements - [#3338](https://github.com/meltano/sdk/issues/3338) Document how to cast values to different data types in inline stream maps - | Low | 10/28/2025 |
| v0.52.2 | ## v0.52.2 (2025-10-22) ### π Fixes - [#3328](https://github.com/meltano/sdk/issues/3328) Support MIT as a license option in templates - [#3327](https://github.com/meltano/sdk/issues/3327) Emit a log when requesting a new access token in OAuth taps | Low | 10/22/2025 |
| v0.52.1 | ## v0.52.1 (2025-10-22) ### π¦ Packaging changes - [#3325](https://github.com/meltano/sdk/issues/3325) Fix `python_version` marker in `backoff` dependency | Low | 10/22/2025 |
| v0.52.0 | ## v0.52.0 (2025-10-21) ### β¨ New - [#3311](https://github.com/meltano/sdk/issues/3311) New `Stream.preprocess_context` lets developers make changes to the stream context before it is frozen ### π Fixes - [#3322](https://github.com/meltano/sdk/issues/3322) Improved error message when a request is being retried, including URL path and status code - [#3313](https://github.com/meltano/sdk/issues/3313) Ignore the `Programming Language :: Python :: 3 :: Only` classifier when generating | Low | 10/21/2025 |
| v0.51.0 | ## v0.51.0 (2025-10-17) ### β¨ New - [#3307](https://github.com/meltano/sdk/issues/3307) Allow the user to override stream and sink meters (i.e. counters & timers) ### βοΈ Under the Hood - [#3302](https://github.com/meltano/sdk/issues/3302) In target logs, put the sink name in a trailing position | Low | 10/17/2025 |
| v0.50.2 | ## v0.50.2 (2025-10-06) ### π Fixes - [#3297](https://github.com/meltano/sdk/issues/3297) Honor the `_LOG_REQUEST_METRICS` stream parameter | Low | 10/6/2025 |
| v0.50.1 | ## v0.50.1 (2025-10-03) ### π Fixes - [#3295](https://github.com/meltano/sdk/issues/3295) Ensure plugins advertise `structured-logging` capability | Low | 10/3/2025 |
| v0.50.0 | ## v0.50.0 (2025-10-01) ### β¨ New - [#3279](https://github.com/meltano/sdk/issues/3279) Emit structured logs and document this new capability for downstream consumers ### π Fixes - [#3287](https://github.com/meltano/sdk/issues/3287) Handle presence of `charset` in `Content-Type` header when retrieving schema from OpenAPI spec - [#3286](https://github.com/meltano/sdk/issues/3286) Only write starting replication value for incremental streams -- _**Thanks @ReubenFrankel!**_ ### βοΈ | Low | 10/1/2025 |
| v0.49.1 | ## v0.49.1 (2025-09-14) ### π Fixes - [#3275](https://github.com/meltano/sdk/issues/3275) Add setter method to deprecated `APIAuthenticatorBase.tap_name` property | Low | 9/14/2025 |
| v0.49.0 | ## v0.49.0 (2025-09-11) ### β¨ New - [#3262](https://github.com/meltano/sdk/issues/3262) By default `ObjectType` fields now use `additionalProperties: true` - [#3196](https://github.com/meltano/sdk/issues/3196) Use `ty` to check types in tap, target and mapper templates ### π Fixes - [#3257](https://github.com/meltano/sdk/issues/3257) Add `DecimalType` alias for `NumberType` ### βοΈ Under the Hood - [#3269](https://github.com/meltano/sdk/issues/3269) Move all SQLAlchemy-related | Low | 9/11/2025 |
| v0.48.1 | ## v0.48.1 (2025-08-21) ### β¨ New - [#3238](https://github.com/meltano/sdk/issues/3238) Auto-detect and load YAML-formatted OpenAPI specs - [#3227](https://github.com/meltano/sdk/issues/3227) Get the supported Python versions in `--about` output from PyPI classifiers if they are available - [#3226](https://github.com/meltano/sdk/issues/3226) Declare Python 3.14 support in templates - [#3223](https://github.com/meltano/sdk/issues/3223) Add a metrics exclusion filter - [#3222](https://gi | Low | 8/21/2025 |
| v0.48.0 | ## v0.48.0 (2025-08-04) ### Highlights #### Schema sources The Singer SDK now provides an extensible API for loading schemas from various sources through the `SchemaSource` API. For example, ```python from typing import ClassVar from singer_sdk import OpenAPISchema, StreamSchema # Load from OpenAPI spec openapi_source = OpenAPISchema("https://api.example.com/openapi.json") class UsersStream(RESTStream): name = "users" schema: ClassVar[StreamSchema] = StreamSchema | Low | 8/4/2025 |
| v0.47.4 | ## v0.47.4 (2025-06-25) ### π Fixes - [#3138](https://github.com/meltano/sdk/issues/3138) Ensure Singer SDK warnings are logged (#3127) | Low | 6/25/2025 |
| v0.46.5 | ## v0.46.5 (2025-06-24) ### π Fixes - [#3135](https://github.com/meltano/sdk/issues/3135) Ensure Singer SDK warnings are logged | Low | 6/25/2025 |
| v0.45.12 | ## v0.45.12 (2025-06-24) ### π Fixes - [#3132](https://github.com/meltano/sdk/issues/3132) Ensure Singer SDK warnings are logged (#3127) | Low | 6/24/2025 |
| v0.44.4 | ## v0.44.4 (2025-06-24) ### π Fixes - [#3128](https://github.com/meltano/sdk/issues/3128) Ensure Singer SDK warnings are logged ### π¦ Packaging changes - [#3129](https://github.com/meltano/sdk/issues/3129) Test with click 8.2.x | Low | 6/24/2025 |
| v0.47.3 | ## v0.47.3 (2025-06-19) ### π Fixes - [#3113](https://github.com/meltano/sdk/issues/3113) Remove redundant `tool.ruff.target-version` from tap template ### π Documentation Improvements - [#3116](https://github.com/meltano/sdk/issues/3116) Document `post_process` changes in 0.47 | Low | 6/19/2025 |
| v0.47.2 | ## v0.47.2 (2025-06-17) ### π Fixes - [#3111](https://github.com/meltano/sdk/issues/3111) Ensure state progress markers in child streams are finalized - [#3108](https://github.com/meltano/sdk/issues/3108) Suppress unmapped warning for additional properties -- _**Thanks @ReubenFrankel!**_ | Low | 6/17/2025 |
| v0.47.1 | ## v0.47.1 (2025-06-16) ### π Fixes - [#3106](https://github.com/meltano/sdk/issues/3106) Apply `.upper()` to `metrics_log_level` setting value - [#3102](https://github.com/meltano/sdk/issues/3102) Bump "build and inspect python package" action in templates and use the name output for the PyPI url | Low | 6/16/2025 |
| v0.47.0 | ## v0.47.0 (2025-06-13) ### Highlights #### `ACTIVATE_VERSION` Taps can now emit [`ACTIVATE_VERSION`](https://hub.meltano.com/singer/docs/#activate-version) messages when the builtin `emit_activate_version_messages` config is set to `True`. #### Global log level Setting the `<APP_NAME>_LOGLEVEL` or `LOGLEVEL` environment variables will now apply that level to _all_ logs generated by taps , mappers and targets. #### `post_process` The stream `post_process` method is now execu | Low | 6/13/2025 |
| v0.46.4 | ## v0.46.4 (2025-05-28) ### π Fixes - [#3066](https://github.com/meltano/sdk/issues/3066) Revert table name splitting by `-` in SQL targets when `default_target_schema` is set, introduced in #3020 | Low | 5/28/2025 |
| v0.46.3 | ## v0.46.3 (2025-05-15) ### π Fixes - [#3042](https://github.com/meltano/sdk/issues/3042) Revert "fix(targets): Avoid emitting message `"No schema for record field."` when "additionalProperties" is set to true in stream schema ([#3031](https://github.com/meltano/sdk/issues/3031))" | Low | 5/15/2025 |
| v0.46.2 | ## v0.46.2 (2025-05-15) ### π Fixes - [#3037](https://github.com/meltano/sdk/issues/3037) Avoid aborting parent stream before child stream is finished in standard tap tests - [#3031](https://github.com/meltano/sdk/issues/3031) Avoid emitting message `"No schema for record field."` when "additionalProperties" is set to true in stream schema -- _**Thanks @SidduHussain!**_ - [#3040](https://github.com/meltano/sdk/issues/3040) Avoid emitting empty state value from target -- _**Thanks @joaop | Low | 5/15/2025 |
| v0.46.1 | ## v0.46.1 (2025-05-13) ### π Fixes - [#3034](https://github.com/meltano/sdk/issues/3034) Avoid emitting an empty state from the target if the tap has not sent any state messages - [#3030](https://github.com/meltano/sdk/issues/3030) Bump standard tap tests max records limit from 25 to 150 ### π Documentation Improvements - [#3026](https://github.com/meltano/sdk/issues/3026) Auto-format markdown with `mdformat` ### π¦ Packaging changes - [#3033](https://github.com/meltano/sdk | Low | 5/13/2025 |
| v0.46.0 | ## v0.46.0 (2025-05-06) ### β¨ New - [#2162](https://github.com/meltano/sdk/issues/2162) Expose metric dictionary to make logging metrics as JSON easier - [#2977](https://github.com/meltano/sdk/issues/2977) Limit extraction to 25 records by default in built-in tap tests - [#2989](https://github.com/meltano/sdk/issues/2989) Allow pagination to continue after an empty page ### π Fixes - [#3020](https://github.com/meltano/sdk/issues/3020) In SQL targets, use stream name as table name | Low | 5/6/2025 |
| v0.45.11 | ## v0.45.11 (2025-04-28) ### π Fixes - [#3004](https://github.com/meltano/sdk/issues/3004) Ensure fields with an array of JSON schema types are properly handled by SQL targets - [#2995](https://github.com/meltano/sdk/issues/2995) Pin GitHub actions in templates - [#2987](https://github.com/meltano/sdk/issues/2987) Add a date to the warning message for removal of `Faker` from stream maps contexts - [#2982](https://github.com/meltano/sdk/issues/2982) Allow streams to override `effective_ | Low | 4/28/2025 |
| v0.45.10 | ## v0.45.10 (2025-04-16) ### π Fixes - [#2978](https://github.com/meltano/sdk/issues/2978) Restore private `_http_headers` attribute of REST streams | Low | 4/16/2025 |
| v0.45.9 | ## v0.45.9 (2025-04-15) ### π Fixes - [#2975](https://github.com/meltano/sdk/issues/2975) Update cookiecutter templates - [#2974](https://github.com/meltano/sdk/issues/2974) Do not log a warning if `null` cannot be appended to schema types - [#2973](https://github.com/meltano/sdk/issues/2973) Recognize `oneOf` fields in Singer schema helpers | Low | 4/15/2025 |
| v0.45.8 | ## v0.45.8 (2025-04-14) ### π Fixes - [#2965](https://github.com/meltano/sdk/issues/2965) `AttributeError: object has no attribute '_requests_session'` when accessing public property during stream initialization -- _**Thanks @ReubenFrankel!**_ - [#2970](https://github.com/meltano/sdk/issues/2970) Consider `oneOf` schemas when checking for `date` and `datetime` types | Low | 4/14/2025 |
| v0.45.7 | ## v0.45.7 (2025-04-14) ### π Fixes - [#2968](https://github.com/meltano/sdk/issues/2968) `singer_sdk.singerlib.resolve_schema_references` now resolves `oneOf`Β schema references - [#2924](https://github.com/meltano/sdk/issues/2924) Generate standard stream metadata for nested fields - [#2940](https://github.com/meltano/sdk/issues/2940) Ensure streams with hardcoded `schema` attributes (e.g. REST and GraphQL streams) can have their schema overridden by the catalog ### π¦ Packaging cha | Low | 4/14/2025 |
| v0.45.6 | ## v0.45.6 (2025-04-11) ### π Fixes - [#2961](https://github.com/meltano/sdk/issues/2961) Conform `NaN`, `Infinity` and `-Infinity` to null values ### π Documentation Improvements - [#2960](https://github.com/meltano/sdk/issues/2960) Documented using `default=str` with `json.dumps` in stream maps - [#2955](https://github.com/meltano/sdk/issues/2955) Documented the `ignore_parent_replication_key` stream attribute | Low | 4/11/2025 |
| v0.45.5 | ## v0.45.5 (2025-04-04) ### π Fixes - [#2952](https://github.com/meltano/sdk/issues/2952) Adapt field schema type to `string` when transformed with stream maps using `json.dumps` ### βοΈ Under the Hood - [#2948](https://github.com/meltano/sdk/issues/2948) `ignore_no_records` as an instance property in built-in tests -- _**Thanks @ReubenFrankel!**_ | Low | 4/4/2025 |
| v0.45.4 | ## v0.45.4 (2025-04-01) ### π Fixes - [#2946](https://github.com/meltano/sdk/issues/2946) In the built-in tap tests, do not emit a warning for missing records on ignored streams | Low | 4/1/2025 |
| v0.45.3 | ## v0.45.3 (2025-03-29) ### π Fixes - [#2937](https://github.com/meltano/sdk/issues/2937) Revert "fix: Ensure stream schema is overridden by the input catalog ([#2932](https://github.com/meltano/sdk/issues/2932))" ### π Documentation Improvements - [#2938](https://github.com/meltano/sdk/issues/2938) Fixed the dynamic schema example | Low | 3/29/2025 |
| v0.45.2 | ## v0.45.2 (2025-03-28) ### π Fixes - [#2935](https://github.com/meltano/sdk/issues/2935) Added `uv` venv backend to Meltano project in templates - [#2934](https://github.com/meltano/sdk/issues/2934) Updated cookiecutter pre-commit hooks and added `check-meltano` JSON schema check - [#2932](https://github.com/meltano/sdk/issues/2932) Ensure stream schema is overridden by the input catalog | Low | 3/28/2025 |
| v0.45.1 | ## v0.45.1 (2025-03-27) ### π Fixes - [#2925](https://github.com/meltano/sdk/issues/2925) Let downstream packages import from deprecated `singer_sdk._singerlib.catalog` | Low | 3/27/2025 |
| v0.45.0 | ## v0.45.0 (2025-03-25) ### Highlights #### Experimental msgspec IO Taps and targets now support custom message writing and reading interfaces, and the new (experimental) `msgspec`-based encoding is available. See the [docs](https://sdk.meltano.com/en/v0.45.0/guides/performance.html#use-a-different-message-writer-or-reader) for more information. Thanks to @BuzzCutNorman and @s7clarke10 for the thoughts and suggestions! #### `singerlib` is now a public module The low-level Sing | Low | 3/27/2025 |
| v0.44.3 | ## v0.44.3 (2025-02-05) ### π Fixes - [#2870](https://github.com/meltano/sdk/issues/2870) Do not emit a warning needlessly when `rest_method` is not set in a stream class | Low | 2/5/2025 |
| v0.44.2 | ## v0.44.2 (2025-01-31) ### π Fixes - [#2862](https://github.com/meltano/sdk/issues/2862) Include package license metadata conditionally in templates -- _**Thanks @ReubenFrankel!**_ - [#2859](https://github.com/meltano/sdk/issues/2859) Use uv in tap, target and mapper templates ### π Documentation Improvements - [#2861](https://github.com/meltano/sdk/issues/2861) Document plugin development with uv and how to migrate existing ones ### π¦ Packaging changes - [#2854](https://g | Low | 1/31/2025 |
| v0.44.1 | ## v0.44.1 (2025-01-29) ### π Fixes - [#2847](https://github.com/meltano/sdk/issues/2847) Update Cookiecutter templates - [#2844](https://github.com/meltano/sdk/issues/2844) Avoid writing an empty state -- _**Thanks @joaopamaral!**_ - [#2843](https://github.com/meltano/sdk/issues/2843) Use SQLAlchemy to generate an insert statement | Low | 1/30/2025 |
