Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(databend sink): support csv encoding & compression none #16829

Merged
merged 9 commits into from Mar 31, 2023
Merged

feat(databend sink): support csv encoding & compression none #16829

merged 9 commits into from Mar 31, 2023

Conversation

everpcpc
Copy link
Contributor

require: #16828

@everpcpc everpcpc requested review from spencergilbert and a team as code owners March 17, 2023 04:10
@netlify
Copy link

netlify bot commented Mar 17, 2023 •
edited

✅ Deploy Preview for vector-project ready!

Name Link
🔨 Latest commit 62a4ab6
🔍 Latest deploy log https://app.netlify.com/sites/vector-project/deploys/6426f42efbfe0d0008f30c9d
😎 Deploy Preview https://deploy-preview-16829--vector-project.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@netlify
Copy link

netlify bot commented Mar 17, 2023 •
edited

✅ Deploy Preview for vrl-playground canceled.

Name Link
🔨 Latest commit 3227fc0
🔍 Latest deploy log https://app.netlify.com/sites/vrl-playground/deploys/641e99afb9541300078e6292

@github-actions github-actions bot added the domain: sinks Anything related to the Vector's sinks label Mar 17, 2023
@everpcpc everpcpc requested a review from a team as a code owner March 17, 2023 04:53
@github-actions github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Mar 17, 2023
@everpcpc everpcpc changed the title feat(databend sink): support csv encoding feat(databend sink): support csv encoding & compression none Mar 17, 2023
@github-actions
Copy link

Regression Detector Results

Run ID: 12fbae7f-b915-405a-b271-a57bbed593e8
Baseline: 5cd741e
Comparison: 98318e4
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
http_text_to_http_json ingress throughput 467.96KiB/CPU-s 1.92 100.00% 23.81MiB/CPU-s 527.5KiB/CPU-s 6.49KiB/CPU-s 0.0 0.021631 24.27MiB/CPU-s 524.09KiB/CPU-s 6.45KiB/CPU-s 0.0 0.021086 False False
otlp_http_to_blackhole ingress throughput 25.7KiB/CPU-s 1.65 100.00% 1.52MiB/CPU-s 123.65KiB/CPU-s 1.52KiB/CPU-s 0.0 0.079597 1.54MiB/CPU-s 112.79KiB/CPU-s 1.39KiB/CPU-s 0.0 0.071423 False False
otlp_grpc_to_blackhole ingress throughput 11.6KiB/CPU-s 1.11 100.00% 1.02MiB/CPU-s 56.51KiB/CPU-s 711.89B/CPU-s 0.0 0.053936 1.03MiB/CPU-s 43.39KiB/CPU-s 546.84B/CPU-s 0.0 0.040962 False False
syslog_splunk_hec_logs ingress throughput 84.73KiB/CPU-s 0.96 100.00% 8.65MiB/CPU-s 238.19KiB/CPU-s 2.93KiB/CPU-s 0.0 0.026878 8.74MiB/CPU-s 198.91KiB/CPU-s 2.45KiB/CPU-s 0.0 0.022234 False False
splunk_hec_route_s3 ingress throughput 96.8KiB/CPU-s 0.82 100.00% 11.51MiB/CPU-s 554.88KiB/CPU-s 6.83KiB/CPU-s 0.0 0.047084 11.6MiB/CPU-s 526.72KiB/CPU-s 6.48KiB/CPU-s 0.0 0.044331 False False
http_to_http_acks ingress throughput 42.47KiB/CPU-s 0.80 61.51% 5.15MiB/CPU-s 2.75MiB/CPU-s 34.61KiB/CPU-s 0.0 0.532872 5.2MiB/CPU-s 2.74MiB/CPU-s 34.51KiB/CPU-s 0.0 0.527001 True False
socket_to_socket_blackhole ingress throughput 75.88KiB/CPU-s 0.56 100.00% 13.12MiB/CPU-s 216.41KiB/CPU-s 2.66KiB/CPU-s 0.0 0.016105 13.2MiB/CPU-s 234.02KiB/CPU-s 2.88KiB/CPU-s 0.0 0.017318 False False
datadog_agent_remap_datadog_logs_acks ingress throughput 174.43KiB/CPU-s 0.52 100.00% 32.85MiB/CPU-s 1.01MiB/CPU-s 12.71KiB/CPU-s 0.0 0.0307 33.02MiB/CPU-s 1.07MiB/CPU-s 13.52KiB/CPU-s 0.0 0.032487 False False
enterprise_http_to_http ingress throughput 3.84KiB/CPU-s 0.03 75.13% 13.62MiB/CPU-s 222.74KiB/CPU-s 2.74KiB/CPU-s 0.0 0.015969 13.62MiB/CPU-s 153.17KiB/CPU-s 1.89KiB/CPU-s 0.0 0.010978 False False
http_to_http_noack ingress throughput 3.6KiB/CPU-s 0.03 43.77% 13.61MiB/CPU-s 373.58KiB/CPU-s 4.6KiB/CPU-s 0.0 0.026808 13.61MiB/CPU-s 340.3KiB/CPU-s 4.19KiB/CPU-s 0.0 0.024414 False False
fluent_elasticsearch ingress throughput -1.01KiB/CPU-s -0.00 64.73% 45.41MiB/CPU-s 29.97KiB/CPU-s 373.56B/CPU-s 0.0 0.000644 45.41MiB/CPU-s 84.05KiB/CPU-s 1.02KiB/CPU-s 0.0 0.001807 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput 507.96B/CPU-s 0.00 6.29% 13.61MiB/CPU-s 355.24KiB/CPU-s 4.37KiB/CPU-s 0.0 0.025481 13.61MiB/CPU-s 367.84KiB/CPU-s 4.52KiB/CPU-s 0.0 0.026384 False False
splunk_hec_indexer_ack_blackhole ingress throughput -1.21KiB/CPU-s -0.01 20.88% 13.62MiB/CPU-s 261.22KiB/CPU-s 3.21KiB/CPU-s 0.0 0.018735 13.61MiB/CPU-s 266.42KiB/CPU-s 3.28KiB/CPU-s 0.0 0.019109 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -1.24KiB/CPU-s -0.01 20.94% 13.62MiB/CPU-s 264.23KiB/CPU-s 3.25KiB/CPU-s 0.0 0.01895 13.61MiB/CPU-s 272.62KiB/CPU-s 3.35KiB/CPU-s 0.0 0.019553 False False
datadog_agent_remap_blackhole ingress throughput -34.75KiB/CPU-s -0.11 93.69% 31.03MiB/CPU-s 1.04MiB/CPU-s 13.04KiB/CPU-s 0.0 0.033352 31.0MiB/CPU-s 1.06MiB/CPU-s 13.4KiB/CPU-s 0.0 0.034316 False False
syslog_loki ingress throughput -15.79KiB/CPU-s -0.18 100.00% 8.47MiB/CPU-s 206.65KiB/CPU-s 2.54KiB/CPU-s 0.0 0.023827 8.45MiB/CPU-s 192.17KiB/CPU-s 2.36KiB/CPU-s 0.0 0.022198 False False
http_to_http_json ingress throughput -62.61KiB/CPU-s -0.45 100.00% 13.61MiB/CPU-s 252.74KiB/CPU-s 3.11KiB/CPU-s 0.0 0.018134 13.55MiB/CPU-s 354.28KiB/CPU-s 4.36KiB/CPU-s 0.0 0.025535 False False
syslog_regex_logs2metric_ddmetrics ingress throughput -17.54KiB/CPU-s -0.47 99.63% 3.61MiB/CPU-s 384.41KiB/CPU-s 4.73KiB/CPU-s 0.0 0.10403 3.59MiB/CPU-s 305.84KiB/CPU-s 3.76KiB/CPU-s 0.0 0.083162 True True
syslog_humio_logs ingress throughput -47.79KiB/CPU-s -0.53 100.00% 8.73MiB/CPU-s 268.43KiB/CPU-s 3.3KiB/CPU-s 0.0 0.030017 8.69MiB/CPU-s 261.76KiB/CPU-s 3.22KiB/CPU-s 0.0 0.029429 False False
datadog_agent_remap_datadog_logs ingress throughput -229.44KiB/CPU-s -0.67 100.00% 33.21MiB/CPU-s 971.95KiB/CPU-s 11.96KiB/CPU-s 0.0 0.028576 32.99MiB/CPU-s 796.32KiB/CPU-s 9.8KiB/CPU-s 0.0 0.023571 False False
datadog_agent_remap_blackhole_acks ingress throughput -473.19KiB/CPU-s -1.47 100.00% 31.37MiB/CPU-s 816.81KiB/CPU-s 10.05KiB/CPU-s 0.0 0.025422 30.91MiB/CPU-s 896.48KiB/CPU-s 11.03KiB/CPU-s 0.0 0.028319 False False
syslog_log2metric_splunk_hec_metrics ingress throughput -160.36KiB/CPU-s -1.70 100.00% 9.2MiB/CPU-s 151.83KiB/CPU-s 1.87KiB/CPU-s 0.0 0.016118 9.04MiB/CPU-s 276.3KiB/CPU-s 3.4KiB/CPU-s 0.0 0.029839 False False
syslog_log2metric_humio_metrics ingress throughput -271.18KiB/CPU-s -4.31 100.00% 6.14MiB/CPU-s 213.21KiB/CPU-s 2.62KiB/CPU-s 0.0 0.033882 5.88MiB/CPU-s 419.2KiB/CPU-s 5.16KiB/CPU-s 0.0 0.069618 False False
file_to_blackhole egress throughput -833.07KiB/CPU-s -11.27 100.00% 7.22MiB/CPU-s 3.82MiB/CPU-s 128.07KiB/CPU-s 0.0 0.528791 6.4MiB/CPU-s 4.15MiB/CPU-s 125.19KiB/CPU-s 5.381944 0.647771 True True

@github-actions
Copy link

Regression Detector Results

Run ID: f5d796d5-6e34-4494-9caa-dfb63e7804a9
Baseline: 5cd741e
Comparison: ec51caa
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
otlp_http_to_blackhole ingress throughput 31.23KiB/CPU-s 2.03 100.00% 1.5MiB/CPU-s 127.56KiB/CPU-s 1.57KiB/CPU-s 0.0 0.083092 1.53MiB/CPU-s 111.87KiB/CPU-s 1.38KiB/CPU-s 0.0 0.07142 False False
splunk_hec_route_s3 ingress throughput 180.76KiB/CPU-s 1.54 100.00% 11.47MiB/CPU-s 558.19KiB/CPU-s 6.87KiB/CPU-s 0.0 0.047527 11.65MiB/CPU-s 533.73KiB/CPU-s 6.57KiB/CPU-s 0.0 0.044756 False False
datadog_agent_remap_datadog_logs_acks ingress throughput 515.12KiB/CPU-s 1.54 100.00% 32.6MiB/CPU-s 1.13MiB/CPU-s 14.19KiB/CPU-s 0.0 0.034545 33.1MiB/CPU-s 1.04MiB/CPU-s 13.13KiB/CPU-s 0.0 0.031486 False False
http_text_to_http_json ingress throughput 321.21KiB/CPU-s 1.32 100.00% 23.82MiB/CPU-s 646.0KiB/CPU-s 7.95KiB/CPU-s 0.0 0.026481 24.14MiB/CPU-s 646.88KiB/CPU-s 7.96KiB/CPU-s 0.0 0.026172 False False
http_to_http_acks ingress throughput 27.34KiB/CPU-s 0.51 42.85% 5.21MiB/CPU-s 2.71MiB/CPU-s 34.13KiB/CPU-s 0.0 0.520228 5.23MiB/CPU-s 2.72MiB/CPU-s 34.2KiB/CPU-s 0.0 0.518857 True False
syslog_humio_logs ingress throughput 14.66KiB/CPU-s 0.16 99.98% 8.75MiB/CPU-s 219.47KiB/CPU-s 2.7KiB/CPU-s 0.0 0.024483 8.77MiB/CPU-s 232.28KiB/CPU-s 2.86KiB/CPU-s 0.0 0.025869 False False
http_to_http_json ingress throughput 11.33KiB/CPU-s 0.08 99.35% 13.61MiB/CPU-s 251.63KiB/CPU-s 3.1KiB/CPU-s 0.0 0.018054 13.62MiB/CPU-s 225.67KiB/CPU-s 2.78KiB/CPU-s 0.0 0.016178 False False
enterprise_http_to_http ingress throughput 3.7KiB/CPU-s 0.03 66.89% 13.62MiB/CPU-s 245.05KiB/CPU-s 3.02KiB/CPU-s 0.0 0.017571 13.62MiB/CPU-s 188.19KiB/CPU-s 2.32KiB/CPU-s 0.0 0.01349 False False
http_to_http_noack ingress throughput 1.9KiB/CPU-s 0.01 21.79% 13.6MiB/CPU-s 402.06KiB/CPU-s 4.94KiB/CPU-s 0.0 0.028859 13.61MiB/CPU-s 386.06KiB/CPU-s 4.75KiB/CPU-s 0.0 0.027707 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput 495.6B/CPU-s 0.00 6.12% 13.61MiB/CPU-s 366.73KiB/CPU-s 4.51KiB/CPU-s 0.0 0.026306 13.61MiB/CPU-s 358.28KiB/CPU-s 4.41KiB/CPU-s 0.0 0.025699 False False
fluent_elasticsearch ingress throughput 85.43B/CPU-s 0.00 12.83% 45.41MiB/CPU-s 30.07KiB/CPU-s 374.76B/CPU-s 0.0 0.000647 45.41MiB/CPU-s 29.97KiB/CPU-s 373.55B/CPU-s 0.0 0.000644 False False
datadog_agent_remap_datadog_logs ingress throughput -3.64KiB/CPU-s -0.01 17.39% 33.16MiB/CPU-s 973.23KiB/CPU-s 11.97KiB/CPU-s 0.0 0.028659 33.16MiB/CPU-s 929.1KiB/CPU-s 11.43KiB/CPU-s 0.0 0.027362 False False
splunk_hec_indexer_ack_blackhole ingress throughput -1.1KiB/CPU-s -0.01 19.74% 13.62MiB/CPU-s 250.15KiB/CPU-s 3.08KiB/CPU-s 0.0 0.017938 13.62MiB/CPU-s 254.18KiB/CPU-s 3.13KiB/CPU-s 0.0 0.018229 False False
datadog_agent_remap_blackhole_acks ingress throughput -5.36KiB/CPU-s -0.02 23.58% 31.18MiB/CPU-s 1.1MiB/CPU-s 13.87KiB/CPU-s 0.0 0.035284 31.18MiB/CPU-s 917.39KiB/CPU-s 11.29KiB/CPU-s 0.0 0.028733 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -3.77KiB/CPU-s -0.03 60.15% 13.62MiB/CPU-s 240.99KiB/CPU-s 2.96KiB/CPU-s 0.0 0.01728 13.61MiB/CPU-s 271.32KiB/CPU-s 3.34KiB/CPU-s 0.0 0.01946 False False
datadog_agent_remap_blackhole ingress throughput -37.69KiB/CPU-s -0.12 94.53% 30.79MiB/CPU-s 1.26MiB/CPU-s 15.85KiB/CPU-s 0.0 0.040868 30.76MiB/CPU-s 939.1KiB/CPU-s 11.56KiB/CPU-s 0.0 0.029815 False False
socket_to_socket_blackhole ingress throughput -40.64KiB/CPU-s -0.31 100.00% 13.01MiB/CPU-s 298.6KiB/CPU-s 3.67KiB/CPU-s 0.0 0.02241 12.97MiB/CPU-s 319.36KiB/CPU-s 3.93KiB/CPU-s 0.0 0.024041 False False
syslog_loki ingress throughput -32.29KiB/CPU-s -0.38 100.00% 8.37MiB/CPU-s 308.16KiB/CPU-s 3.79KiB/CPU-s 0.0 0.035972 8.33MiB/CPU-s 260.31KiB/CPU-s 3.2KiB/CPU-s 0.0 0.030502 False False
otlp_grpc_to_blackhole ingress throughput -5.79KiB/CPU-s -0.55 100.00% 1.03MiB/CPU-s 48.67KiB/CPU-s 613.2B/CPU-s 0.0 0.045975 1.03MiB/CPU-s 50.53KiB/CPU-s 636.61B/CPU-s 0.0 0.047994 False False
syslog_splunk_hec_logs ingress throughput -140.31KiB/CPU-s -1.56 100.00% 8.77MiB/CPU-s 254.7KiB/CPU-s 3.13KiB/CPU-s 0.0 0.028371 8.63MiB/CPU-s 196.24KiB/CPU-s 2.41KiB/CPU-s 0.0 0.022206 False False
syslog_log2metric_splunk_hec_metrics ingress throughput -153.04KiB/CPU-s -1.66 100.00% 9.0MiB/CPU-s 295.76KiB/CPU-s 3.64KiB/CPU-s 0.0 0.032094 8.85MiB/CPU-s 280.02KiB/CPU-s 3.44KiB/CPU-s 0.0 0.0309 False False
syslog_log2metric_humio_metrics ingress throughput -139.26KiB/CPU-s -2.23 100.00% 6.11MiB/CPU-s 152.96KiB/CPU-s 1.88KiB/CPU-s 0.0 0.024441 5.97MiB/CPU-s 132.91KiB/CPU-s 1.64KiB/CPU-s 0.0 0.021721 False False
syslog_regex_logs2metric_ddmetrics ingress throughput -108.34KiB/CPU-s -2.90 100.00% 3.65MiB/CPU-s 377.99KiB/CPU-s 4.65KiB/CPU-s 0.0 0.10125 3.54MiB/CPU-s 309.12KiB/CPU-s 3.8KiB/CPU-s 0.0 0.085278 True True
file_to_blackhole egress throughput -743.86KiB/CPU-s -10.59 100.00% 6.86MiB/CPU-s 3.94MiB/CPU-s 124.92KiB/CPU-s 0.0 0.574384 6.14MiB/CPU-s 4.29MiB/CPU-s 120.64KiB/CPU-s 0.0 0.69912 True True

zamazan4ik
zamazan4ik reviewed Mar 17, 2023
View reviewed changes
website/cue/reference/components/sinks/base/databend.cue
default: "gzip"
enum: {
gzip: """
[Gzip][gzip] compression.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understand, Databend supports more compression algorithms: https://databend.rs/doc/load-data/#supported-file-formats

Is it true? If yes, would you like to add support for more compression algorithms, supported by databend?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's true.
Since there are only Gzip and Zlib in sink::util:buffer::compression::Compression, we could add more compression format support for vector later.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I do not recommend using sink::util:buffer::compression::Compression for sinks-specific compression settings since every sink has own set of supported compression algorithms. sink::util:buffer::compression::Compression is too generic.

I advise creating a Databend-specific Compression enum, filling it with supported by Databend algorithms, and then using it for compression purposes.

However, I see that here you are bounded by the trait so you cannot use this approach :( Hopefully, later we will extend sink::util:buffer::compression::Compression with more algorithms...

@github-actions
Copy link

Regression Detector Results

Run ID: 9573939e-8e2e-4971-b4f9-55cade13826b
Baseline: a5f7315
Comparison: 5afcc8a
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
splunk_hec_route_s3 ingress throughput 199.57KiB/CPU-s 1.75 100.00% 11.17MiB/CPU-s 654.26KiB/CPU-s 8.05KiB/CPU-s 0.0 0.057221 11.36MiB/CPU-s 603.71KiB/CPU-s 7.43KiB/CPU-s 0.0 0.051894 False False
syslog_log2metric_humio_metrics ingress throughput 45.08KiB/CPU-s 0.74 100.00% 5.96MiB/CPU-s 197.42KiB/CPU-s 2.43KiB/CPU-s 0.0 0.032329 6.01MiB/CPU-s 279.61KiB/CPU-s 3.44KiB/CPU-s 0.0 0.045454 False False
otlp_http_to_blackhole ingress throughput 6.15KiB/CPU-s 0.40 99.59% 1.51MiB/CPU-s 125.08KiB/CPU-s 1.54KiB/CPU-s 0.0 0.080684 1.52MiB/CPU-s 121.0KiB/CPU-s 1.49KiB/CPU-s 0.0 0.077747 False False
http_to_http_acks ingress throughput 20.62KiB/CPU-s 0.39 33.46% 5.14MiB/CPU-s 2.69MiB/CPU-s 33.95KiB/CPU-s 0.0 0.523853 5.16MiB/CPU-s 2.66MiB/CPU-s 33.48KiB/CPU-s 0.0 0.514636 True False
otlp_grpc_to_blackhole ingress throughput 3.68KiB/CPU-s 0.35 100.00% 1.02MiB/CPU-s 53.1KiB/CPU-s 669.0B/CPU-s 0.0 0.05066 1.03MiB/CPU-s 47.34KiB/CPU-s 596.56B/CPU-s 0.0 0.045006 False False
syslog_loki ingress throughput 26.85KiB/CPU-s 0.32 100.00% 8.29MiB/CPU-s 275.55KiB/CPU-s 3.39KiB/CPU-s 0.0 0.032475 8.31MiB/CPU-s 270.44KiB/CPU-s 3.33KiB/CPU-s 0.0 0.031772 False False
datadog_agent_remap_blackhole_acks ingress throughput 84.75KiB/CPU-s 0.27 99.98% 30.85MiB/CPU-s 1.43MiB/CPU-s 18.07KiB/CPU-s 0.0 0.046498 30.93MiB/CPU-s 1.13MiB/CPU-s 14.25KiB/CPU-s 0.0 0.036553 False False
http_text_to_http_json ingress throughput 66.1KiB/CPU-s 0.26 100.00% 24.4MiB/CPU-s 535.96KiB/CPU-s 6.59KiB/CPU-s 0.0 0.021451 24.46MiB/CPU-s 569.27KiB/CPU-s 7.01KiB/CPU-s 0.0 0.022724 False False
datadog_agent_remap_blackhole ingress throughput 66.02KiB/CPU-s 0.21 98.91% 30.54MiB/CPU-s 1.49MiB/CPU-s 18.73KiB/CPU-s 0.0 0.048692 30.61MiB/CPU-s 1.42MiB/CPU-s 17.95KiB/CPU-s 0.0 0.046534 False False
http_to_http_json ingress throughput 15.93KiB/CPU-s 0.11 99.98% 13.61MiB/CPU-s 269.06KiB/CPU-s 3.31KiB/CPU-s 0.0 0.01931 13.62MiB/CPU-s 224.49KiB/CPU-s 2.76KiB/CPU-s 0.0 0.016093 False False
enterprise_http_to_http ingress throughput 5.58KiB/CPU-s 0.04 82.79% 13.62MiB/CPU-s 272.01KiB/CPU-s 3.35KiB/CPU-s 0.0 0.019507 13.62MiB/CPU-s 190.88KiB/CPU-s 2.35KiB/CPU-s 0.0 0.013683 False False
fluent_elasticsearch ingress throughput -80.89B/CPU-s -0.00 12.04% 45.41MiB/CPU-s 30.16KiB/CPU-s 375.91B/CPU-s 0.0 0.000649 45.41MiB/CPU-s 30.44KiB/CPU-s 379.33B/CPU-s 0.0 0.000654 False False
http_to_http_noack ingress throughput -68.9B/CPU-s -0.00 0.92% 13.61MiB/CPU-s 335.85KiB/CPU-s 4.13KiB/CPU-s 0.0 0.024094 13.61MiB/CPU-s 338.38KiB/CPU-s 4.16KiB/CPU-s 0.0 0.024276 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput -1.23KiB/CPU-s -0.01 15.24% 13.61MiB/CPU-s 368.8KiB/CPU-s 4.54KiB/CPU-s 0.0 0.026451 13.61MiB/CPU-s 366.67KiB/CPU-s 4.51KiB/CPU-s 0.0 0.026301 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -2.31KiB/CPU-s -0.02 37.69% 13.62MiB/CPU-s 261.21KiB/CPU-s 3.21KiB/CPU-s 0.0 0.018733 13.61MiB/CPU-s 278.6KiB/CPU-s 3.43KiB/CPU-s 0.0 0.019984 False False
splunk_hec_indexer_ack_blackhole ingress throughput -3.03KiB/CPU-s -0.02 50.13% 13.62MiB/CPU-s 248.48KiB/CPU-s 3.06KiB/CPU-s 0.0 0.017818 13.61MiB/CPU-s 266.45KiB/CPU-s 3.28KiB/CPU-s 0.0 0.019111 False False
datadog_agent_remap_datadog_logs ingress throughput -35.88KiB/CPU-s -0.11 95.47% 32.94MiB/CPU-s 1.0MiB/CPU-s 12.63KiB/CPU-s 0.0 0.030444 32.9MiB/CPU-s 1.01MiB/CPU-s 12.71KiB/CPU-s 0.0 0.030654 False False
datadog_agent_remap_datadog_logs_acks ingress throughput -38.9KiB/CPU-s -0.12 92.36% 32.55MiB/CPU-s 1.33MiB/CPU-s 16.7KiB/CPU-s 0.0 0.040729 32.51MiB/CPU-s 1.13MiB/CPU-s 14.24KiB/CPU-s 0.0 0.034764 False False
syslog_humio_logs ingress throughput -20.01KiB/CPU-s -0.22 100.00% 8.74MiB/CPU-s 190.32KiB/CPU-s 2.34KiB/CPU-s 0.0 0.021262 8.72MiB/CPU-s 220.14KiB/CPU-s 2.71KiB/CPU-s 0.0 0.024648 False False
socket_to_socket_blackhole ingress throughput -127.46KiB/CPU-s -0.95 100.00% 13.11MiB/CPU-s 249.41KiB/CPU-s 3.07KiB/CPU-s 0.0 0.01858 12.98MiB/CPU-s 407.29KiB/CPU-s 5.01KiB/CPU-s 0.0 0.030632 False False
syslog_splunk_hec_logs ingress throughput -116.23KiB/CPU-s -1.30 100.00% 8.72MiB/CPU-s 234.11KiB/CPU-s 2.88KiB/CPU-s 0.0 0.026227 8.6MiB/CPU-s 244.52KiB/CPU-s 3.01KiB/CPU-s 0.0 0.027755 False False
syslog_log2metric_splunk_hec_metrics ingress throughput -125.98KiB/CPU-s -1.35 100.00% 9.08MiB/CPU-s 277.63KiB/CPU-s 3.42KiB/CPU-s 0.0 0.029858 8.96MiB/CPU-s 302.2KiB/CPU-s 3.72KiB/CPU-s 0.0 0.032947 False False
file_to_blackhole egress throughput -244.97KiB/CPU-s -3.53 82.48% 6.79MiB/CPU-s 4.25MiB/CPU-s 127.4KiB/CPU-s 7.112254 0.626098 6.55MiB/CPU-s 4.1MiB/CPU-s 128.09KiB/CPU-s 0.0 0.625896 True True
syslog_regex_logs2metric_ddmetrics ingress throughput -217.75KiB/CPU-s -6.00 100.00% 3.54MiB/CPU-s 309.96KiB/CPU-s 3.81KiB/CPU-s 0.0 0.085435 3.33MiB/CPU-s 374.55KiB/CPU-s 4.61KiB/CPU-s 0.0 0.109827 True True

spencergilbert
spencergilbert requested changes Mar 20, 2023
View reviewed changes
Copy link
Contributor

@spencergilbert spencergilbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We definitely want to take a look at both the encoding and compression and look to avoid exposing a number of unsupported options in the documentation.

At a high level, I'm not sure how this impacts a user though. Regardless of encoding I'd expect Databend to expose the data appropriately via a UI/SQL/etc, without leaking details of how the data is stored at rest?

Is the intention allow users to configure this to aid in manual debugging, by allowing them to read the files directly from blob storage without Databend in the middle?

src/sinks/databend/config.rs Outdated Show resolved Hide resolved
@everpcpc everpcpc marked this pull request as draft March 22, 2023 05:57
@everpcpc everpcpc marked this pull request as ready for review March 25, 2023 06:37
@everpcpc
Copy link
Contributor Author

everpcpc commented Mar 25, 2023 •
edited

Is the intention allow users to configure this to aid in manual debugging, by allowing them to read the files directly from blob storage without Databend in the middle?

This configuration is to helping people with performance optimization, such as reducing cpu usage by transferring without compressing & decompressing. There's no need to read them from blob storage directly, since they are temporary files.

@github-actions
Copy link

Regression Detector Results

Run ID: 2987e507-ccd2-4937-92ee-5628d1a55150
Baseline: 928c96e
Comparison: ee158b8
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

Changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:

experiment goal Δ mean Δ mean % confidence
http_text_to_http_json ingress throughput -1.67MiB/CPU-s -6.52 100.00%
Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
otlp_grpc_to_blackhole ingress throughput 15.3KiB/CPU-s 1.50 100.00% 1022.27KiB/CPU-s 45.52KiB/CPU-s 573.68B/CPU-s 0.0 0.044529 1.01MiB/CPU-s 42.31KiB/CPU-s 533.29B/CPU-s 0.0 0.040778 False False
datadog_agent_remap_blackhole ingress throughput 414.75KiB/CPU-s 1.31 100.00% 30.9MiB/CPU-s 1.19MiB/CPU-s 14.96KiB/CPU-s 0.0 0.038439 31.31MiB/CPU-s 1.21MiB/CPU-s 15.19KiB/CPU-s 0.0 0.038498 False False
syslog_loki ingress throughput 72.79KiB/CPU-s 0.85 100.00% 8.39MiB/CPU-s 247.17KiB/CPU-s 3.04KiB/CPU-s 0.0 0.028765 8.46MiB/CPU-s 179.5KiB/CPU-s 2.21KiB/CPU-s 0.0 0.020715 False False
datadog_agent_remap_blackhole_acks ingress throughput 267.4KiB/CPU-s 0.84 100.00% 31.08MiB/CPU-s 1.21MiB/CPU-s 15.24KiB/CPU-s 0.0 0.038925 31.34MiB/CPU-s 905.27KiB/CPU-s 11.14KiB/CPU-s 0.0 0.028203 False False
splunk_hec_route_s3 ingress throughput 66.46KiB/CPU-s 0.56 100.00% 11.68MiB/CPU-s 539.01KiB/CPU-s 6.63KiB/CPU-s 0.0 0.045063 11.74MiB/CPU-s 515.36KiB/CPU-s 6.34KiB/CPU-s 0.0 0.042848 False False
syslog_regex_logs2metric_ddmetrics ingress throughput 7.65KiB/CPU-s 0.21 77.73% 3.62MiB/CPU-s 363.83KiB/CPU-s 4.48KiB/CPU-s 0.0 0.098083 3.63MiB/CPU-s 357.37KiB/CPU-s 4.4KiB/CPU-s 0.0 0.096144 False True
http_to_http_noack ingress throughput 3.82KiB/CPU-s 0.03 50.92% 13.61MiB/CPU-s 340.78KiB/CPU-s 4.19KiB/CPU-s 0.0 0.024448 13.61MiB/CPU-s 295.59KiB/CPU-s 3.64KiB/CPU-s 0.0 0.0212 False False
enterprise_http_to_http ingress throughput 1.87KiB/CPU-s 0.01 46.63% 13.62MiB/CPU-s 191.5KiB/CPU-s 2.36KiB/CPU-s 0.0 0.013727 13.62MiB/CPU-s 151.42KiB/CPU-s 1.86KiB/CPU-s 0.0 0.010853 False False
fluent_elasticsearch ingress throughput 335.91B/CPU-s 0.00 47.09% 45.41MiB/CPU-s 30.43KiB/CPU-s 379.23B/CPU-s 0.0 0.000654 45.41MiB/CPU-s 30.12KiB/CPU-s 375.47B/CPU-s 0.0 0.000648 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput -711.42B/CPU-s -0.00 8.81% 13.61MiB/CPU-s 364.72KiB/CPU-s 4.49KiB/CPU-s 0.0 0.02616 13.61MiB/CPU-s 357.64KiB/CPU-s 4.4KiB/CPU-s 0.0 0.025653 False False
http_to_http_json ingress throughput -379.0B/CPU-s -0.00 7.44% 13.62MiB/CPU-s 226.78KiB/CPU-s 2.79KiB/CPU-s 0.0 0.016257 13.62MiB/CPU-s 228.56KiB/CPU-s 2.81KiB/CPU-s 0.0 0.016386 False False
splunk_hec_indexer_ack_blackhole ingress throughput -2.71KiB/CPU-s -0.02 45.20% 13.62MiB/CPU-s 253.97KiB/CPU-s 3.12KiB/CPU-s 0.0 0.018212 13.61MiB/CPU-s 265.03KiB/CPU-s 3.26KiB/CPU-s 0.0 0.019009 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -3.4KiB/CPU-s -0.02 54.09% 13.62MiB/CPU-s 253.09KiB/CPU-s 3.11KiB/CPU-s 0.0 0.018149 13.61MiB/CPU-s 274.56KiB/CPU-s 3.38KiB/CPU-s 0.0 0.019694 False False
syslog_humio_logs ingress throughput -18.19KiB/CPU-s -0.20 99.99% 8.97MiB/CPU-s 271.48KiB/CPU-s 3.34KiB/CPU-s 0.0 0.029565 8.95MiB/CPU-s 254.66KiB/CPU-s 3.13KiB/CPU-s 0.0 0.027789 False False
datadog_agent_remap_datadog_logs ingress throughput -242.37KiB/CPU-s -0.72 100.00% 32.96MiB/CPU-s 1.02MiB/CPU-s 12.82KiB/CPU-s 0.0 0.030872 32.72MiB/CPU-s 880.13KiB/CPU-s 10.83KiB/CPU-s 0.0 0.026265 False False
syslog_splunk_hec_logs ingress throughput -92.01KiB/CPU-s -1.01 100.00% 8.92MiB/CPU-s 204.41KiB/CPU-s 2.51KiB/CPU-s 0.0 0.022373 8.83MiB/CPU-s 236.61KiB/CPU-s 2.91KiB/CPU-s 0.0 0.02616 False False
otlp_http_to_blackhole ingress throughput -16.55KiB/CPU-s -1.05 100.00% 1.53MiB/CPU-s 114.0KiB/CPU-s 1.4KiB/CPU-s 0.0 0.072573 1.52MiB/CPU-s 116.16KiB/CPU-s 1.43KiB/CPU-s 0.0 0.074736 False False
syslog_log2metric_humio_metrics ingress throughput -97.09KiB/CPU-s -1.54 100.00% 6.15MiB/CPU-s 176.95KiB/CPU-s 2.18KiB/CPU-s 0.0 0.028101 6.05MiB/CPU-s 210.5KiB/CPU-s 2.59KiB/CPU-s 0.0 0.033952 False False
syslog_log2metric_splunk_hec_metrics ingress throughput -158.75KiB/CPU-s -1.66 100.00% 9.35MiB/CPU-s 162.01KiB/CPU-s 1.99KiB/CPU-s 0.0 0.016919 9.2MiB/CPU-s 360.59KiB/CPU-s 4.44KiB/CPU-s 0.0 0.038291 False False
http_to_http_acks ingress throughput -98.66KiB/CPU-s -1.82 95.49% 5.3MiB/CPU-s 2.76MiB/CPU-s 34.76KiB/CPU-s 0.0 0.520622 5.2MiB/CPU-s 2.77MiB/CPU-s 34.88KiB/CPU-s 0.0 0.532177 True False
datadog_agent_remap_datadog_logs_acks ingress throughput -607.54KiB/CPU-s -1.82 100.00% 32.69MiB/CPU-s 991.42KiB/CPU-s 12.2KiB/CPU-s 0.0 0.029616 32.1MiB/CPU-s 1.04MiB/CPU-s 13.11KiB/CPU-s 0.0 0.032427 False False
socket_to_socket_blackhole ingress throughput -322.33KiB/CPU-s -2.35 100.00% 13.41MiB/CPU-s 347.21KiB/CPU-s 4.27KiB/CPU-s 0.0 0.025276 13.1MiB/CPU-s 307.54KiB/CPU-s 3.78KiB/CPU-s 0.0 0.022926 False False
http_text_to_http_json ingress throughput -1.67MiB/CPU-s -6.52 100.00% 25.66MiB/CPU-s 513.47KiB/CPU-s 6.32KiB/CPU-s 0.0 0.019542 23.98MiB/CPU-s 608.77KiB/CPU-s 7.49KiB/CPU-s 0.0 0.024786 False False
file_to_blackhole egress throughput -1.35MiB/CPU-s -18.36 100.00% 7.34MiB/CPU-s 3.99MiB/CPU-s 127.64KiB/CPU-s 8.195122 0.543742 5.99MiB/CPU-s 4.19MiB/CPU-s 118.72KiB/CPU-s 1.298701 0.700141 True True

@github-actions
Copy link

Regression Detector Results

Run ID: 54c7c3ef-88e8-4443-aebf-8452a6f4864e
Baseline: 928c96e
Comparison: 3227fc0
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

Changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%:

experiment goal Δ mean Δ mean % confidence
http_text_to_http_json ingress throughput -1.88MiB/CPU-s -7.3 100.00%
Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
file_to_blackhole egress throughput 878.77KiB/CPU-s 14.97 100.00% 5.73MiB/CPU-s 4.49MiB/CPU-s 118.41KiB/CPU-s 0.0 0.782839 6.59MiB/CPU-s 4.02MiB/CPU-s 121.54KiB/CPU-s 4.612707 0.610066 True True
syslog_regex_logs2metric_ddmetrics ingress throughput 111.01KiB/CPU-s 3.04 100.00% 3.57MiB/CPU-s 425.11KiB/CPU-s 5.23KiB/CPU-s 0.0 0.116341 3.68MiB/CPU-s 453.19KiB/CPU-s 5.85KiB/CPU-s 0.0 0.120369 True True
datadog_agent_remap_blackhole_acks ingress throughput 631.53KiB/CPU-s 1.99 100.00% 30.97MiB/CPU-s 860.48KiB/CPU-s 10.59KiB/CPU-s 0.0 0.027129 31.59MiB/CPU-s 707.78KiB/CPU-s 8.71KiB/CPU-s 0.0 0.021879 False False
otlp_grpc_to_blackhole ingress throughput 18.99KiB/CPU-s 1.86 100.00% 1020.51KiB/CPU-s 52.55KiB/CPU-s 661.99B/CPU-s 0.0 0.051488 1.02MiB/CPU-s 42.94KiB/CPU-s 541.12B/CPU-s 0.0 0.041302 False False
syslog_log2metric_humio_metrics ingress throughput 105.45KiB/CPU-s 1.70 100.00% 6.06MiB/CPU-s 220.25KiB/CPU-s 2.71KiB/CPU-s 0.0 0.035509 6.16MiB/CPU-s 168.47KiB/CPU-s 2.07KiB/CPU-s 0.0 0.026707 False False
datadog_agent_remap_blackhole ingress throughput 444.98KiB/CPU-s 1.40 100.00% 31.12MiB/CPU-s 991.34KiB/CPU-s 12.2KiB/CPU-s 0.0 0.031105 31.56MiB/CPU-s 833.64KiB/CPU-s 10.26KiB/CPU-s 0.0 0.025797 False False
http_to_http_acks ingress throughput 60.16KiB/CPU-s 1.13 78.42% 5.21MiB/CPU-s 2.73MiB/CPU-s 34.34KiB/CPU-s 0.0 0.523338 5.27MiB/CPU-s 2.73MiB/CPU-s 34.38KiB/CPU-s 0.0 0.518058 True False
syslog_log2metric_splunk_hec_metrics ingress throughput 51.33KiB/CPU-s 0.54 100.00% 9.26MiB/CPU-s 228.32KiB/CPU-s 2.81KiB/CPU-s 0.0 0.024071 9.31MiB/CPU-s 213.46KiB/CPU-s 2.63KiB/CPU-s 0.0 0.022383 False False
splunk_hec_route_s3 ingress throughput 43.41KiB/CPU-s 0.37 100.00% 11.6MiB/CPU-s 564.42KiB/CPU-s 6.94KiB/CPU-s 0.0 0.047526 11.64MiB/CPU-s 544.58KiB/CPU-s 6.7KiB/CPU-s 0.0 0.045688 False False
enterprise_http_to_http ingress throughput 11.29KiB/CPU-s 0.08 99.08% 13.61MiB/CPU-s 317.75KiB/CPU-s 3.91KiB/CPU-s 0.0 0.022792 13.62MiB/CPU-s 151.44KiB/CPU-s 1.86KiB/CPU-s 0.0 0.010854 False False
fluent_elasticsearch ingress throughput -157.92B/CPU-s -0.00 23.36% 45.41MiB/CPU-s 30.12KiB/CPU-s 375.49B/CPU-s 0.0 0.000648 45.41MiB/CPU-s 30.2KiB/CPU-s 376.29B/CPU-s 0.0 0.000649 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput 135.06B/CPU-s 0.00 1.71% 13.61MiB/CPU-s 352.73KiB/CPU-s 4.34KiB/CPU-s 0.0 0.025301 13.61MiB/CPU-s 355.16KiB/CPU-s 4.37KiB/CPU-s 0.0 0.025475 False False
splunk_hec_indexer_ack_blackhole ingress throughput 31.05B/CPU-s 0.00 0.54% 13.62MiB/CPU-s 257.96KiB/CPU-s 3.17KiB/CPU-s 0.0 0.018501 13.62MiB/CPU-s 258.65KiB/CPU-s 3.18KiB/CPU-s 0.0 0.01855 False False
http_to_http_json ingress throughput 518.35B/CPU-s 0.00 10.28% 13.62MiB/CPU-s 226.45KiB/CPU-s 2.79KiB/CPU-s 0.0 0.016234 13.62MiB/CPU-s 224.11KiB/CPU-s 2.76KiB/CPU-s 0.0 0.016065 False False
http_to_http_noack ingress throughput -1.33KiB/CPU-s -0.01 17.01% 13.61MiB/CPU-s 339.43KiB/CPU-s 4.18KiB/CPU-s 0.0 0.024351 13.61MiB/CPU-s 352.04KiB/CPU-s 4.54KiB/CPU-s 0.0 0.025258 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput -2.28KiB/CPU-s -0.02 37.28% 13.62MiB/CPU-s 261.57KiB/CPU-s 3.22KiB/CPU-s 0.0 0.018759 13.61MiB/CPU-s 277.54KiB/CPU-s 3.41KiB/CPU-s 0.0 0.019907 False False
syslog_loki ingress throughput -3.68KiB/CPU-s -0.04 68.19% 8.38MiB/CPU-s 236.73KiB/CPU-s 2.91KiB/CPU-s 0.0 0.027574 8.38MiB/CPU-s 184.31KiB/CPU-s 2.27KiB/CPU-s 0.0 0.021476 False False
otlp_http_to_blackhole ingress throughput -10.11KiB/CPU-s -0.65 100.00% 1.53MiB/CPU-s 122.5KiB/CPU-s 1.51KiB/CPU-s 0.0 0.078221 1.52MiB/CPU-s 115.74KiB/CPU-s 1.42KiB/CPU-s 0.0 0.074383 False False
datadog_agent_remap_datadog_logs ingress throughput -248.32KiB/CPU-s -0.74 100.00% 32.94MiB/CPU-s 1.07MiB/CPU-s 13.54KiB/CPU-s 0.0 0.032625 32.7MiB/CPU-s 1010.92KiB/CPU-s 12.44KiB/CPU-s 0.0 0.030188 False False
syslog_humio_logs ingress throughput -89.39KiB/CPU-s -0.97 100.00% 8.96MiB/CPU-s 220.57KiB/CPU-s 2.71KiB/CPU-s 0.0 0.024042 8.87MiB/CPU-s 321.89KiB/CPU-s 3.96KiB/CPU-s 0.0 0.035431 False False
syslog_splunk_hec_logs ingress throughput -94.21KiB/CPU-s -1.03 100.00% 8.97MiB/CPU-s 208.94KiB/CPU-s 2.57KiB/CPU-s 0.0 0.022742 8.88MiB/CPU-s 204.93KiB/CPU-s 2.52KiB/CPU-s 0.0 0.022536 False False
datadog_agent_remap_datadog_logs_acks ingress throughput -474.47KiB/CPU-s -1.41 100.00% 32.77MiB/CPU-s 914.16KiB/CPU-s 11.25KiB/CPU-s 0.0 0.027239 32.31MiB/CPU-s 975.7KiB/CPU-s 12.0KiB/CPU-s 0.0 0.02949 False False
socket_to_socket_blackhole ingress throughput -444.84KiB/CPU-s -3.24 100.00% 13.41MiB/CPU-s 265.59KiB/CPU-s 3.27KiB/CPU-s 0.0 0.019346 12.97MiB/CPU-s 356.55KiB/CPU-s 4.39KiB/CPU-s 0.0 0.026842 False False
http_text_to_http_json ingress throughput -1.88MiB/CPU-s -7.30 100.00% 25.76MiB/CPU-s 522.83KiB/CPU-s 6.43KiB/CPU-s 0.0 0.019819 23.88MiB/CPU-s 662.33KiB/CPU-s 8.15KiB/CPU-s 0.0 0.027083 False False

@jszwedko jszwedko requested a review from fuchsnj March 29, 2023 13:07
@spencergilbert
Copy link
Contributor

Flakey failure in the Checks job - retrying.

spencergilbert
spencergilbert reviewed Mar 30, 2023
View reviewed changes
Copy link
Contributor

@spencergilbert spencergilbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still unclear as to when a user would prefer to choose CSV vs JSON encoding, but I'm not sure that's a blocker for me. One question I've raised to our team around consistency for defaults.

src/sinks/databend/compression.rs Outdated Show resolved Hide resolved
@everpcpc
Copy link
Contributor Author

I'm still unclear as to when a user would prefer to choose CSV vs JSON encoding

Actually the main concern is CSV is much smaller than JSON encoding in most cases.

@spencergilbert spencergilbert enabled auto-merge (squash) March 31, 2023 15:17
@spencergilbert
Copy link
Contributor

Thanks @everpcpc!

@github-actions
Copy link

Regression Detector Results

Run ID: f24c8a2c-ae21-482d-8d61-91d980654684
Baseline: b7abb08
Comparison: 62a4ab6
Total vector CPUs: 7

Explanation

A regression test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine quickly if vector performance is changed and to what degree by a pull request.

The table below, if present, lists those experiments that have experienced a statistically significant change in mean optimization goal performance between baseline and comparison SHAs with 90.00% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±5.00% change in their mean optimization goal are discarded. An experiment is erratic if its coefficient of variation is greater than 0.1. The abbreviated table will be omitted if no interesting change is observed.

No interesting changes in experiment optimization goals with confidence ≥ 90.00% and |Δ mean %| ≥ 5.00%.

Fine details of change detection per experiment.
experiment goal Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_log2metric_splunk_hec_metrics ingress throughput 329.18KiB/CPU-s 3.53 100.00% 9.12MiB/CPU-s 391.23KiB/CPU-s 4.81KiB/CPU-s 0.0 0.041911 9.44MiB/CPU-s 209.97KiB/CPU-s 2.58KiB/CPU-s 0.0 0.021727 False False
syslog_splunk_hec_logs ingress throughput 253.31KiB/CPU-s 2.83 100.00% 8.74MiB/CPU-s 304.5KiB/CPU-s 3.75KiB/CPU-s 0.0 0.034002 8.99MiB/CPU-s 268.87KiB/CPU-s 3.31KiB/CPU-s 0.0 0.029197 False False
syslog_humio_logs ingress throughput 164.3KiB/CPU-s 1.79 100.00% 8.98MiB/CPU-s 200.6KiB/CPU-s 2.47KiB/CPU-s 0.0 0.021806 9.14MiB/CPU-s 214.16KiB/CPU-s 2.63KiB/CPU-s 0.0 0.022872 False False
datadog_agent_remap_datadog_logs_acks ingress throughput 453.4KiB/CPU-s 1.38 100.00% 32.04MiB/CPU-s 1.26MiB/CPU-s 15.91KiB/CPU-s 0.0 0.039405 32.48MiB/CPU-s 1019.65KiB/CPU-s 12.55KiB/CPU-s 0.0 0.030651 False False
splunk_hec_route_s3 ingress throughput 54.52KiB/CPU-s 0.45 100.00% 11.79MiB/CPU-s 553.43KiB/CPU-s 6.81KiB/CPU-s 0.0 0.045847 11.84MiB/CPU-s 522.94KiB/CPU-s 6.43KiB/CPU-s 0.0 0.043126 False False
syslog_loki ingress throughput 29.74KiB/CPU-s 0.34 100.00% 8.49MiB/CPU-s 260.72KiB/CPU-s 3.21KiB/CPU-s 0.0 0.030001 8.52MiB/CPU-s 218.87KiB/CPU-s 2.69KiB/CPU-s 0.0 0.0251 False False
otlp_grpc_to_blackhole ingress throughput 2.82KiB/CPU-s 0.27 99.89% 1.03MiB/CPU-s 50.68KiB/CPU-s 638.6B/CPU-s 0.0 0.048245 1.03MiB/CPU-s 48.44KiB/CPU-s 610.34B/CPU-s 0.0 0.045987 False False
datadog_agent_remap_datadog_logs ingress throughput 91.34KiB/CPU-s 0.27 99.99% 32.65MiB/CPU-s 1.49MiB/CPU-s 18.73KiB/CPU-s 0.0 0.04552 32.74MiB/CPU-s 1.06MiB/CPU-s 13.34KiB/CPU-s 0.0 0.032336 False False
enterprise_http_to_http ingress throughput 7.76KiB/CPU-s 0.06 95.49% 13.62MiB/CPU-s 275.46KiB/CPU-s 3.39KiB/CPU-s 0.0 0.019754 13.62MiB/CPU-s 151.71KiB/CPU-s 1.87KiB/CPU-s 0.0 0.010874 False False
splunk_hec_to_splunk_hec_logs_acks ingress throughput 95.87B/CPU-s 0.00 1.23% 13.61MiB/CPU-s 340.05KiB/CPU-s 4.18KiB/CPU-s 0.0 0.024391 13.61MiB/CPU-s 360.68KiB/CPU-s 4.44KiB/CPU-s 0.0 0.025871 False False
splunk_hec_to_splunk_hec_logs_noack ingress throughput 237.74B/CPU-s 0.00 3.95% 13.61MiB/CPU-s 270.41KiB/CPU-s 3.33KiB/CPU-s 0.0 0.019395 13.61MiB/CPU-s 268.4KiB/CPU-s 3.3KiB/CPU-s 0.0 0.01925 False False
http_to_http_noack ingress throughput -224.33B/CPU-s -0.00 3.16% 13.61MiB/CPU-s 316.47KiB/CPU-s 3.89KiB/CPU-s 0.0 0.022701 13.61MiB/CPU-s 319.85KiB/CPU-s 3.94KiB/CPU-s 0.0 0.022944 False False
splunk_hec_indexer_ack_blackhole ingress throughput 176.92B/CPU-s 0.00 2.97% 13.61MiB/CPU-s 267.24KiB/CPU-s 3.29KiB/CPU-s 0.0 0.019168 13.61MiB/CPU-s 265.48KiB/CPU-s 3.27KiB/CPU-s 0.0 0.019042 False False
fluent_elasticsearch ingress throughput -7.97KiB/CPU-s -0.02 80.66% 45.41MiB/CPU-s 30.02KiB/CPU-s 374.28B/CPU-s 0.0 0.000646 45.41MiB/CPU-s 502.78KiB/CPU-s 6.12KiB/CPU-s 0.0 0.010812 False False
syslog_log2metric_humio_metrics ingress throughput -9.51KiB/CPU-s -0.15 97.31% 6.23MiB/CPU-s 239.38KiB/CPU-s 2.95KiB/CPU-s 0.0 0.037536 6.22MiB/CPU-s 253.9KiB/CPU-s 3.12KiB/CPU-s 0.0 0.039872 False False
http_to_http_json ingress throughput -20.86KiB/CPU-s -0.15 100.00% 13.62MiB/CPU-s 227.83KiB/CPU-s 2.8KiB/CPU-s 0.0 0.016334 13.6MiB/CPU-s 260.17KiB/CPU-s 3.2KiB/CPU-s 0.0 0.01868 False False
http_to_http_acks ingress throughput -9.33KiB/CPU-s -0.17 15.16% 5.25MiB/CPU-s 2.71MiB/CPU-s 34.16KiB/CPU-s 0.0 0.515884 5.25MiB/CPU-s 2.77MiB/CPU-s 34.86KiB/CPU-s 0.0 0.527422 True False
datadog_agent_remap_blackhole ingress throughput -66.44KiB/CPU-s -0.21 98.34% 30.4MiB/CPU-s 1.8MiB/CPU-s 22.69KiB/CPU-s 0.0 0.059255 30.34MiB/CPU-s 1.26MiB/CPU-s 15.92KiB/CPU-s 0.0 0.041654 False False
syslog_regex_logs2metric_ddmetrics ingress throughput -14.23KiB/CPU-s -0.38 96.14% 3.61MiB/CPU-s 398.29KiB/CPU-s 4.9KiB/CPU-s 0.0 0.107601 3.6MiB/CPU-s 392.64KiB/CPU-s 4.83KiB/CPU-s 0.0 0.106484 True True
otlp_http_to_blackhole ingress throughput -13.41KiB/CPU-s -0.83 100.00% 1.57MiB/CPU-s 99.1KiB/CPU-s 1.22KiB/CPU-s 0.0 0.061598 1.56MiB/CPU-s 106.72KiB/CPU-s 1.31KiB/CPU-s 0.0 0.066891 False False
socket_to_socket_blackhole ingress throughput -216.69KiB/CPU-s -1.55 100.00% 13.62MiB/CPU-s 394.87KiB/CPU-s 4.86KiB/CPU-s 0.0 0.028307 13.41MiB/CPU-s 550.83KiB/CPU-s 6.78KiB/CPU-s 0.0 0.040111 False False
http_text_to_http_json ingress throughput -419.13KiB/CPU-s -1.61 100.00% 25.49MiB/CPU-s 634.73KiB/CPU-s 7.81KiB/CPU-s 0.0 0.024313 25.08MiB/CPU-s 554.65KiB/CPU-s 6.83KiB/CPU-s 0.0 0.021592 False False
datadog_agent_remap_blackhole_acks ingress throughput -678.12KiB/CPU-s -2.16 100.00% 30.7MiB/CPU-s 1.58MiB/CPU-s 19.9KiB/CPU-s 0.0 0.051428 30.04MiB/CPU-s 1.55MiB/CPU-s 19.51KiB/CPU-s 0.0 0.051541 False False
file_to_blackhole egress throughput -967.96KiB/CPU-s -13.66 100.00% 6.92MiB/CPU-s 4.25MiB/CPU-s 126.19KiB/CPU-s 4.705882 0.613984 5.98MiB/CPU-s 4.6MiB/CPU-s 120.47KiB/CPU-s 0.0 0.76883 True True

@spencergilbert spencergilbert merged commit df7663a into vectordotdev:master Mar 31, 2023
37 checks passed
@everpcpc everpcpc deleted the feat-databend-csv branch March 31, 2023 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zamazan4ik zamazan4ik zamazan4ik left review comments

@fuchsnj fuchsnj fuchsnj approved these changes

@spencergilbert spencergilbert spencergilbert approved these changes

Assignees
No one assigned
Labels
domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@everpcpc @spencergilbert @fuchsnj @zamazan4ik