Merge pull request #126 from jk-cluster/feat/gotenberg-decoupled-limits

gotenberg: small requests, explicit higher limits

gotenberg/Chart.yaml

@@ -5,7 +5,7 @@ icon: https://gotenberg.dev/img/logo.png
 
 type: application
 
-version: 6.0.0+up8.36.0
+version: 7.0.0+up8.36.0
 appVersion: "8.36.0"
 
 maintainers:

gotenberg/templates/NOTES.txt

@@ -38,28 +38,45 @@ NO AUTHENTICATION, AND THIS CHART CANNOT ADD ANY
   and the URL routes make the server fetch whatever URL they are handed. Keep
   it cluster-internal; do not put an Ingress in front of it without auth.
 
-MEMORY IS THE LIMIT YOU HIT FIRST
+BIG CONVERSIONS RUN OUT OF TIME BEFORE THEY RUN OUT OF MEMORY
-  Chromium and LibreOffice hold the whole document in memory. With this
+  Chromium and LibreOffice hold the whole document in memory, so a conversion
-  release's limit of {{ index $limits "memory" | default "(none set)" }}, a 20-page HTML converts in well under a second,
+  is a short, memory-hungry burst. This release limits the container to
-  while a 25 MB HTML is enough to get the container OOM-killed.
-  How you notice: the caller gets no HTTP error — the connection is dropped
-  mid-request — and the pod's RESTARTS counter goes up. Confirm with
 
-    kubectl -n {{ .Release.Namespace }} get pod -l app={{ .Release.Name }}-{{ .Chart.Name }}-deployment -o jsonpath='{.items[*].status.containerStatuses[*].lastState.terminated.reason}'
+    memory:            {{ index $limits "memory" | default "(none set)" }}
+    ephemeral-storage: {{ index $limits "ephemeral-storage" | default "(none set)" }}
 
-  which then prints OOMKilled. Raise gotenberg.resources.requests.memory (the
+  against deliberately smaller requests: the pod reserves little while it
-  limit follows at limitFactor x request) or set an explicit limit.
+  idles and grows into the limits only while converting.
 
-TEMP SPACE IS TIGHT, AND RUNNING OUT LOOKS DIFFERENT
+  Those limits are set so the 30-second request timeout below is what a
-  /tmp and /home/gotenberg are emptyDirs, so they count against the container's
+  too-large document hits first. An 18 MB HTML converts in about 22 seconds
-  ephemeral-storage limit of {{ index $limits "ephemeral-storage" | default "(none set)" }}. Chromium's profile alone takes about 4 MB,
+  and stays well inside them; a document heavy enough to exhaust the memory
-  and the uploaded file plus the rendered PDF come on top.
+  limit would have run past the 30 seconds anyway, so raising memory further
-  How you notice: the pod does not restart, it disappears — a fresh one takes
+  does not turn a failing conversion into a successful one — splitting the
-  its place and the old one is left in status Evicted, with the message naming
+  document does.
-  ephemeral-storage. Raise gotenberg.resources.requests.ephemeralStorage.
+
+  If you do need to change them, set gotenberg.resources.limits.memory and
+  .limits.ephemeralStorage. Both are set explicitly now, so raising the
+  matching request no longer drags the limit along at limitFactor x request.
+
+  Should a conversion still hit a limit, the two failures look different and
+  neither of them reports an HTTP error to the caller:
+
+  - Out of memory: the connection is dropped mid-request and the pod's
+    RESTARTS counter goes up. Confirm with
+
+      kubectl -n {{ .Release.Namespace }} get pod -l app={{ .Release.Name }}-{{ .Chart.Name }}-deployment -o jsonpath='{.items[*].status.containerStatuses[*].lastState.terminated.reason}'
+
+    which then prints OOMKilled.
+
+  - Out of ephemeral storage: /tmp and /home/gotenberg are emptyDirs and count
+    against the ephemeral-storage limit — Chromium's profile alone takes about
+    4 MB, and the uploaded file plus the rendered PDF come on top. The pod does
+    not restart, it disappears: a fresh one takes its place and the old one is
+    left in status Evicted, with the message naming ephemeral-storage.
 
 REQUESTS TIME OUT AFTER 30 SECONDS
   That is Gotenberg's own default, and since the chart passes no flags it
-  cannot be changed here. Large conversions get close: the 25 MB document
+  cannot be changed here. This is the ceiling large conversions actually meet:
-  above still needed 19 seconds once it had memory enough to finish. Callers
+  an 18 MB HTML needs about 22 seconds, and a 25 MB one does not finish inside
-  that need longer must split the document instead.
+  the 30. Callers that need longer must split the document instead.

gotenberg/templates/_helpers.tpl

@@ -263,7 +263,8 @@ instead of a container without any resource accounting.
 {{- define "gotenberg.effectiveResources" -}}
 {{- $defaults := dict
       "limitFactor" 3
-      "requests" (dict "cpu" 0.1 "memory" "512Mi" "ephemeralStorage" "10Mi") -}}
+      "requests" (dict "cpu" 0.1 "memory" "512Mi" "ephemeralStorage" "10Mi")
+      "limits" (dict "memory" "3Gi" "ephemeralStorage" "256Mi") -}}
 {{- $merged := fromYaml (include "gotenberg.defaultedDict" (dict "defaults" $defaults "given" . "path" "gotenberg.resources")) -}}
 {{- include "gotenberg.resources" $merged -}}
 {{- end -}}

gotenberg/values.yaml

@@ -54,7 +54,27 @@ gotenberg:
     # limitFactor x the request, an explicitly set limit always wins, and a
     # CPU limit is only set when configured explicitly under "limits".
     limitFactor: 3
+    # The requests stay small on purpose: gotenberg idles at a few dozen MB,
+    # so reserving conversion-sized capacity on the node would waste it
+    # around the clock. Conversions are short bursts, and the limits below
+    # cover the burst instead (issue #106).
     requests:
       cpu: 0.1
       memory: 512Mi
       ephemeralStorage: 10Mi
+    # Set explicitly rather than left to limitFactor x request, because the
+    # computed limits (1536Mi / 30Mi) were too small for real documents.
+    # Measured with the official image, HTML converted via Chromium:
+    #   - 18 MB HTML: OOMKilled at 2Gi and 2560Mi, converts at 3Gi in ~22s
+    #     (21 MB PDF). 12 MB HTML already OOM-kills at the old 1536Mi.
+    #   - Raising memory alone only moves the failure: with 3Gi memory and
+    #     the old 30Mi ephemeral limit, /tmp peaked at 40 MB on a single
+    #     conversion and at ~125 MB over a short series, and the pod was
+    #     evicted ("ephemeral local storage usage exceeds the total limit of
+    #     containers 30Mi") instead of OOM-killed.
+    # Memory beyond 3Gi buys nothing: gotenberg's own request timeout is 30s
+    # and the chart passes no flags to change it, so a document that needs
+    # more memory than this has already run out of time.
+    limits:
+      memory: 3Gi
+      ephemeralStorage: 256Mi