aws: Update compression support for AWS plugins

[ShelbyZ]

• Enable snappy compression support for AWS plugins (S3, Kinesis Firehose/Streams)
• Add compression support to Kinesis Streams

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

Summary view
https://gist.github.com/ShelbyZ/6a325762003785ba1804faeff958a1c7

Debug Logs
https://gist.github.com/ShelbyZ/9d0036c46600b44cb9a4cea99795230a

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [NA] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

fluent/fluent-bit-docs#2359

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Snappy compression added to AWS compression support.
  • Compression option added to Kinesis Streams and S3 outputs (gzip, zstd, snappy).

• Documentation

  • Firehose compression docs updated to list supported formats (gzip, zstd, snappy) and clarify availability of Arrow when enabled.

• Tests

  • Expanded test coverage for gzip, zstd, and snappy across Kinesis, Firehose, and S3, including aggregation scenarios.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds SNAPPY compression support across AWS outputs: new compression constant, snappy wiring in AWS compression dispatch, Kinesis config + runtime compression paths, S3 header handling, Firehose docs update, and expanded unit/runtime tests for snappy (and other compressions).

Changes

Cohort / File(s)	Summary
Compression Type Definition include/fluent-bit/aws/flb_aws_compress.h	Added FLB_AWS_COMPRESS_SNAPPY constant (value 5).
AWS Compression Dispatch src/aws/flb_aws_compress.c	Added #include <fluent-bit/flb_snappy.h>, flb_snappy_compress_wrapper, and dispatch entry for "snappy".
Kinesis Streams Config & Context plugins/out_kinesis_streams/kinesis.h, plugins/out_kinesis_streams/kinesis.c	Added int compression to flb_kinesis and init logic to parse/validate compression config (defaults to none).
Kinesis Record Processing plugins/out_kinesis_streams/kinesis_api.c, plugins/out_kinesis_streams/kinesis_api.h	Added compression-aware paths for per-record and aggregated-record flows (use flb_aws_compression_b64_truncate_compress when enabled); introduced MAX_B64_EVENT_SIZE macro for sizing.
S3 Content-Encoding / Config plugins/out_s3/s3.c	Added Content-Encoding: snappy handling and included snappy in header construction and config descriptions.
Firehose Docs plugins/out_kinesis_firehose/firehose.c	Updated compression config description to list gzip, zstd, and snappy (kept arrow note).
Tests — Internal Compression tests/internal/aws_compress.c	Added snappy uncompress wrapper, snappy test cases, and registered new tests in test harness.
Tests — Runtime Kinesis tests/runtime/out_kinesis.c	Added runtime tests for compression_gzip, compression_zstd, compression_snappy, and aggregation variant.
Tests — Runtime Firehose tests/runtime/out_firehose.c	Added runtime tests for zstd/snappy (including aggregation).
Tests — Runtime S3 tests/runtime/out_s3.c	Added runtime tests for gzip, zstd, snappy (standard and putobject variants).

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant KinesisOutput as Kinesis Output Plugin
    participant CompressDispatch as AWS Compression Dispatch
    participant RecordBuffer as Record/Event Buffer

    Client->>KinesisOutput: Submit record(s)
    KinesisOutput->>KinesisOutput: Read ctx->compression
    alt compression != NONE
        KinesisOutput->>CompressDispatch: flb_aws_compression_b64_truncate_compress(in_data)
        CompressDispatch-->>KinesisOutput: compressed_b64_data + size
        KinesisOutput->>RecordBuffer: Replace buffer with compressed data
    else compression == NONE
        KinesisOutput->>RecordBuffer: Base64-encode data (existing path)
        RecordBuffer-->>KinesisOutput: base64_data + size
    end
    KinesisOutput->>Client: send record(s) to AWS

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• zstd: aws: compress: fix incompatible pointer glitches #10833: Related edits to AWS compression dispatch and function-pointer handling in src/aws/flb_aws_compress.c.
• out_s3: Add parquet compression type with pure C #10691: Adds compression constants and registers compressor entries in the same compression subsystem.
• out_s3: Add ZSTD Compression Support #10794: Modifies AWS compression subsystem and S3 output handling; overlaps in approach for adding compression types.

Suggested reviewers

• edsiper
• cosmo0920
• fujimotos
• koleini
• PettitWesley

Poem

🐰 Snappy hops into the stream,
Compressing bytes like carrot cream.
Headers set and tests all run,
Data dances, light as fun.
Hooray — small hops, big throughput! 🥕✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 16.22% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'aws: Update compression support for AWS plugins' directly and accurately summarizes the main change: adding snappy compression support and compression support to AWS plugins (S3, Kinesis Firehose, Kinesis Streams).

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@plugins/out_kinesis_streams/kinesis_api.h`:
- Line 26: The macro MAX_B64_EVENT_SIZE currently defines 1398075 but the
comment formula ceil(1048556 / 3) * 4 evaluates to 1398076; update the macro or
the comment to be consistent: either change the macro MAX_B64_EVENT_SIZE to
1398076 to match the formula, or if the -1 is intentional (e.g., to
reserve/exclude a NUL), update the comment next to MAX_B64_EVENT_SIZE to clearly
state that the value intentionally subtracts one (explain reason). Ensure you
edit the definition of MAX_B64_EVENT_SIZE and its inline comment together so the
code and comment match.

In `@src/aws/flb_aws_compress.c`:
- Around line 35-41: The wrapper flb_snappy_compress_wrapper currently
propagates negative error codes from flb_snappy_compress; change it to call
flb_snappy_compress into an int variable (e.g., ret) and then return -1 if ret <
0, otherwise return ret, so all negative error codes are normalized to -1 to
match the AWS compression interface contract.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5e4ce37e5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".