HTTP(s) Data Connector Deployment Guide

Production operating guide for the HTTP(s) data connector covering authentication, rate control, retry tuning, and observability.

Authentication & Secrets

The connector supports HTTP Basic, custom-header, and OAuth2 (refresh-token and client-credentials grants) authentication. Secrets must be sourced from a secret store in production.

Parameter	Description
http_username	Username for HTTP Basic authentication.
http_password	Password for HTTP Basic authentication. Use ${secrets:...} to resolve from a secret store.
http_headers	Custom headers (e.g. Authorization:Bearer ${secrets:api_token}). Treated as sensitive — not logged. Dynamic JSON API endpoints only; structured HTTP file datasets ignore these headers.
auth_token_url	OAuth2 token endpoint URL (must be HTTPS in production).
auth_grant_type	OAuth2 grant: refresh_token (default) or client_credentials.
http_auth_refresh_token	OAuth2 refresh token. Required for the (default) refresh-token grant; unused by client_credentials.
http_auth_client_id	OAuth2 client ID (required for confidential clients and for client_credentials).
http_auth_client_secret	OAuth2 client secret (required for confidential clients and for client_credentials). Use ${secrets:...}.
auth_header_name	Header carrying the access token. Default Authorization (Bearer <token>); any other name sends the bare token.

For OAuth2-protected APIs, prefer refresh-token flow over storing long-lived bearer tokens. The connector exchanges the refresh token for short-lived access tokens at startup and refreshes them before expiry.

TLS

Use HTTPS endpoints in production. auth_token_url must use HTTPS (loopback addresses are allowed for local testing only). Self-signed certificates require a trusted CA bundle in the container or host OS trust store.

For upstream servers that require mutual TLS (mTLS), the connector can present a client certificate during the TLS handshake. Supply the certificate and key as file paths or inline PEM — the two forms are mutually exclusive, and the certificate and key must be set together. mTLS client identity applies to dynamic JSON API endpoints only.

Parameter	Description
http_tls_client_certificate_file	Path to a PEM client certificate chain. Pair with http_tls_client_key_file.
http_tls_client_key_file	Path to the PEM private key matching the client certificate file.
http_tls_client_certificate	Inline PEM client certificate chain. Use ${secrets:...}. Pair with http_tls_client_key.
http_tls_client_key	Inline PEM private key matching the inline certificate. Use ${secrets:...}.

Resilience Controls

Rate Control

The HTTP connector participates in the shared HTTP rate control system. Concurrency and per-second/per-minute request limits can be configured per-dataset (in params) or globally (in runtime.params). Dataset-level settings override the global defaults. Multiple datasets targeting the same upstream origin share a single rate controller.

Parameter	Description
max_concurrent_requests	Maximum concurrent HTTP requests to the same origin. Disabled when unset.
requests_per_second_limit	Maximum HTTP requests per second to the same origin. Disabled when unset.
requests_per_minute_limit	Maximum HTTP requests per minute to the same origin. Disabled when unset.
rate_control_jitter_min	Minimum random delay before requests when rate control is active. Defaults to 5ms.
rate_control_jitter_max	Maximum random delay before requests when rate control is active. Defaults to 10ms.
rate_control_acquire_timeout	Maximum time a request waits for rate-control capacity before it fails. Defaults to client_timeout. 0 waits without a bound.
rate_control_failure_threshold	Upstream error rate above which adaptive rate control admits fewer requests. Defaults to 10%.
rate_control_window	Reaction and recovery window of adaptive rate control. Defaults to 10s.

The runtime equivalents (http_max_concurrent_requests, http_requests_per_second_limit, http_requests_per_minute_limit, http_rate_control_jitter_min, http_rate_control_jitter_max, http_rate_control_acquire_timeout, http_rate_control_failure_threshold, http_rate_control_window) set defaults that apply to every HTTP-based connector unless overridden per dataset. Datasets that share an origin must use the same values, or the later dataset fails to load.

runtime:
  params:
    http_max_concurrent_requests: 10
    http_requests_per_second_limit: 5

datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
      allowed_request_paths: '/data/**'
      max_concurrent_requests: 3 # Override for this dataset
      requests_per_minute_limit: 60

Use rate control when the upstream API enforces request quotas, when many datasets share a single origin, or when running large IN-list refreshes that would otherwise burst hundreds of concurrent requests.

Rate control is adaptive. While the origin's error rate, counting 408, 429, and 5xx responses, connection failures, and timeouts, stays at or below rate_control_failure_threshold, the configured limits apply unchanged. Above it, Spice admits fewer requests than the configured limits and logs a warning naming the origin; as the origin recovers, Spice returns to the full limits and logs that the origin recovered. The rate_control_adaptive_admission_ratio metric shows the share of the configured limits currently admitted. A request that cannot get capacity within rate_control_acquire_timeout fails with Timed out after ... waiting for rate-control capacity. See HTTP Rate Control for details.

Retry Behavior

HTTP-level retries follow the shared resilient_http policy: 408, 429, and 5xx responses plus transient network errors are retried. A request is retried at most max_retries times after the first attempt, so max_retries: 2 sends at most three requests. The connector respects Retry-After, retry-after-ms, and x-retry-after-ms headers.

Parameter	Default	Description
max_retries	3	Maximum retry attempts per request.
retry_backoff_method	fibonacci	Backoff strategy. Options: fibonacci, linear, exponential.
retry_max_duration	unset	Maximum total duration across all retries (e.g. 30s, 5m). When set, retries stop after this elapsed time.
retry_jitter	0.3	Randomization factor (0.0–1.0) applied to retry delays. Set to 0 to disable jitter.

Retries are independent of rate control. If a retry would exceed the configured per-second or per-minute rate, it waits for the rate window to open before issuing the request, for at most rate_control_acquire_timeout.

Timeouts and Connection Pool

Parameter	Default	Description
client_timeout	30	Maximum time (seconds) to wait for the entire request-response cycle.
connect_timeout	10	Maximum time (seconds) to establish a TCP/TLS connection.
pool_max_idle_per_host	10	Maximum idle connections held per upstream host.
pool_idle_timeout	90	Idle connection lifetime (seconds) before the pool closes them.

Increase client_timeout for endpoints with large response bodies or expensive server-side computation. Reduce pool_max_idle_per_host when running many small datasets against the same host to keep the runtime's open file descriptors bounded.

Caching Mode

When using refresh_mode: caching, a 5xx or 429 that is still failing after retries fails the fetch and nothing is written to the cache. The error propagates to clients unless caching_stale_if_error serves the expired cached entry instead. See Error Responses. Set caching_stale_if_error to serve expired cached data on upstream failure — prefer a duration (caching_stale_if_error: 600s), which bounds both how stale a served entry may be and how long it is retained; enabled is the unbounded form. Always set caching_ttl explicitly — the default of 30s is rarely the desired window.

Set caching_max_size (a byte budget, e.g. 512MiB) or caching_max_items (a row budget) to bound the acceleration. A TTL alone does not: a workload that keeps fetching new request paths grows it indefinitely, and with caching_stale_if_error: enabled expired entries are deliberately kept as fallback material and are never expired away (a duration value instead derives an eviction deadline, so it does not have this effect). The runtime warns at startup, naming the dataset, when a caching accelerator has nothing bounding it. The eviction sweep runs at caching_ttl, clamped to 30s–5m, so the acceleration may overshoot its budget by whatever the workload writes between sweeps. See Cache Size and Item Limits.

Capacity & Sizing

• Throughput: Bounded by the upstream rate limit, then by max_concurrent_requests and connect_timeout. Plan limits to stay within the API quota.
• Memory: Response bodies are streamed; memory footprint is bounded by max_request_body_bytes (filter inputs) and DataFusion's record-batch size for response rows.
• Response cache: Each dynamic JSON API dataset holds its own response cache, bounded by response_cache_max_size_bytes — 67108864 (64 MiB) by default, per dataset. The runtime's total exposure therefore scales with the number of HTTP datasets, not with the budget alone: budget it as datasets × response_cache_max_size_bytes and raise the value only for the datasets that earn it. Set 0 on a dataset whose responses are never repeated. This cache is not one of the caches under runtime.caching, so its memory is not counted against those limits.
• Connection setup: TLS handshake adds latency. The connection pool keeps pool_max_idle_per_host warm connections to absorb burst traffic.
• Partitioned refreshes: When using IN-list filters or cross-product partitioning, the runtime issues one HTTP request per partition. Use max_request_partitions to cap the request count for unbounded filter combinations, and max_concurrent_requests to throttle their fan-out.

Metrics

The connector reports two metric families: response-cache occupancy, and per-origin rate control. Both are registered automatically — no metrics configuration is required.

Response cache

Occupancy of the dataset's response cache. These are reported for every HTTP dataset, because the memory the cache holds is otherwise attributable to nothing; structured file-format datasets do not use the cache and report 0.

Metric Name	Type	Description
response_cache_size_bytes	Gauge	Bytes retained by the response cache, counting response bodies, their headers, and the request keys they are held under. Excludes the cache's own per-entry bookkeeping. Compare against response_cache_max_size_bytes to see how close a dataset is to its budget.
response_cache_items_count	Gauge	Number of responses held by the response cache. Read beside the byte figure, this separates a cache holding a few large responses from one holding very many small ones.

Both are refreshed when a request consults the cache, so an idle dataset reports its last observed occupancy — which is the same figure, since nothing enters or leaves the cache except on a request.

Rate control

Per-origin rate-control metrics, exposed for dynamic JSON API datasets. The limit gauges report 0 when the corresponding limit is not configured. Structured file-format datasets (parquet, csv, and the other listing-table formats) do not expose them:

Metric Name	Type	Description
inflight_operations	Gauge	Current number of HTTP requests holding a rate-control permit.
rate_control_max_concurrent_requests	Gauge	Configured maximum concurrent HTTP requests for this upstream origin; 0 means disabled.
rate_control_requests_per_second_limit	Gauge	Configured HTTP request-per-second limit for this upstream origin; 0 means disabled.
rate_control_requests_per_minute_limit	Gauge	Configured HTTP request-per-minute limit for this upstream origin; 0 means disabled.
rate_control_jitter_min_ms	Gauge	Configured minimum rate-control jitter (ms) before HTTP requests.
rate_control_jitter_max_ms	Gauge	Configured maximum rate-control jitter (ms) before HTTP requests.
rate_control_available_permits	Gauge	Current available permits in the HTTP request concurrency semaphore; 0 when concurrency is disabled.
rate_control_acquisitions_total	Counter	Total HTTP request rate-control permits acquired.
rate_control_acquire_errors_total	Counter	Total HTTP request rate-control permit acquisition errors.
rate_control_wait_duration_ms	Counter	Cumulative time (ms) spent waiting for HTTP rate-control permits, quotas, and jitter.
rate_limit_retry_after_updates_total	Counter	Total upstream cooldown hints accepted from Retry-After or RateLimit reset headers.
rate_limit_retry_after_waits_total	Counter	Total waits caused by Retry-After or RateLimit reset headers.
rate_limit_retry_after_wait_duration_ms	Counter	Cumulative time (ms) spent waiting because of Retry-After or RateLimit reset headers.
rate_limit_retry_after_remaining_ms	Gauge	Current remaining Retry-After / RateLimit cooldown (ms) for this upstream origin.
rate_control_adaptive_admission_ratio	Gauge	Share of the configured limits that adaptive rate control currently admits for this upstream origin; 1 admits the full limits. Absent when the origin has no limit configured.
rate_control_adaptive_throttled_total	Counter	Total requests that adaptive rate control throttled because the origin was failing. Absent when the origin has no limit configured.

These metrics are auto-registered — no configuration is required to export them. To turn one off for a dataset, set enabled: false in the dataset's metrics section:

datasets:
  - from: https://api.example.com/v1
    name: api_data
    params:
      file_format: json
    metrics:
      - name: rate_control_wait_duration_ms
        enabled: false

Instruments from both families are exposed with the prefix dataset_http_ — the HTTP connector's component name is http, not https — so the exported names are dataset_http_response_cache_size_bytes, dataset_http_rate_control_wait_duration_ms, and so on. The two families are attributed differently: the rate-control instruments carry an origin attribute (scheme://host:port) identifying the upstream origin instead of a dataset name, because datasets sharing an origin share one rate controller, while the response-cache gauges carry the dataset name, because each dataset has its own cache. See Component Metrics for general configuration.

For broader observability, also monitor:

• Spice query execution metrics (query_duration_ms, query_returned_rows, query_failures) from runtime.metrics.

Task History

HTTP requests participate in task history through the HTTP client's span. Each partitioned request and each pagination page is a child of the enclosing sql_query or acceleration_refresh task.

Known Limitations

• Read-only: The connector is read-only. Only GET and POST (via request_body filters) are supported.
• Filter pushdown is opt-in: request_path, request_query, request_body, and request_headers filters require explicit allowlists or _filters: enabled parameters.
• OAuth2 OOS scope: The refresh-token and client-credentials grants are supported. The authorization-code and device-code flows are not exposed.
• OR across virtual filter columns: WHERE request_path = '/a' OR request_query = 'b=1' is rejected. Use separate datasets or UNION ALL for cross-column alternatives. Single-column OR (and IN-lists) is supported.

Troubleshooting

Symptom	Likely cause	Resolution
401 Unauthorized	Wrong/expired token or password.	Rotate the credential in the secret store.
429 Too Many Requests (frequent)	Upstream rate limit hit; concurrency too high.	Set requests_per_second_limit / requests_per_minute_limit; reduce max_concurrent_requests.
Refresh blocked / queue building up	max_concurrent_requests set too low for the workload.	Raise the dataset-level limit or move heavy datasets to their own origin.
Timed out after ... waiting for rate-control capacity	Requests queue for capacity longer than rate_control_acquire_timeout (default client_timeout).	Raise the limit, lower request concurrency, or raise rate_control_acquire_timeout; 0 waits without a bound.
Fewer requests than the configured limits reach the origin	The origin is failing above rate_control_failure_threshold, so adaptive rate control throttles it.	Check dataset_http_rate_control_adaptive_admission_ratio and the origin's health; raise rate_control_failure_threshold to tolerate more errors.
Multiple HTTP-based components target ... with different rate-control settings	Datasets sharing an origin resolve to different rate-control values.	Use the same values on every dataset and catalog that targets the origin.
OAuth2 token refresh fails	auth_token_url not HTTPS, or wrong client credentials.	Verify the token endpoint URL; check http_auth_client_id/secret and required scopes.
Request rejected: "OR across HTTP filter columns"	WHERE request_path = '...' OR request_query = '...'.	Split into separate refreshes or UNION ALL.
Many partitions created from cross-product	Multiple IN-list filters multiplied into many requests.	Set max_request_partitions to cap; tighten filters.
Slow first refresh	Cold connection pool + TLS handshake per request.	Raise pool_max_idle_per_host; ensure pool_idle_timeout is long enough to keep connections warm.
Runtime memory grows with HTTP traffic	Response caches are held per dataset, 64 MiB each by default.	Check dataset_http_response_cache_size_bytes per dataset; lower response_cache_max_size_bytes, or set it to 0 where responses are never repeated.
Repeat queries still hit the origin	The origin refuses retention (no-store, no-cache, private, Vary: *), or sends no Cache-Control at all.	Confirm with dataset_http_response_cache_items_count staying at 0. For a header-less origin, set response_cache_fallback_ttl; an origin that refuses explicitly is always honored.