Trigger.dev worked fine in development env, but once I deployed to production, none of my tasks are running! They get to dequeued status then they hang there until they expire.
Similar issues found on Discord
- Task not running on Staging - the issue was their redis client
- Issue while running the task on production by Pythonista645 - no resolution
- Prod not running by Mortalife - outdated. they’re using provider and coordinator which is not present in v4
- Staging runs not executing by felotus - Nick from trigger team says to check supervisor logs. Says the containers are either not being created or can’t connect to the supervisor.
Similar issues found on github
I realize that my tasks in staging do not run either. So it’s an issue on staging on prod, but it’s fine and working on development. Let’s see if we can solve staging before we move onto production.
Ok so from what I’ve gathered, the supervisor is what runs on the worker instances. I’m using the coolify docker-compose which is all-in-one, consisting of both webapp and worker. My supervisor logs show that tasks are being dequeued.
{
"timestamp": "2026-06-02T12:04:52.903Z",
"message": "create succeeded",
"$name": "docker-workload-manager",
"$level": "debug",
"opts": {
"dequeuedAt": "2026-06-02T12:04:52.265Z",
"dequeueResponseMs": 9683,
"pollingIntervalMs": 1000,
"warmStartCheckMs": 0,
"envId": "cmp55xnjc0008pr27gt0c13dp",
"envType": "PRODUCTION",
"image": "gitea.sbtp.xyz/trigger/proj_zlvybyxzqtafjwonlxid:20260522.5.production.i88gmspz@sha256:5bf2c41254c0f067d9e06c724314052edee0bf8a9582d286b294075fe4c26c42",
"machine": {
"name": "small-1x",
"cpu": 0.5,
"memory": 0.5,
"centsPerMs": 0
},
"orgId": "cmp55xi160004pr27rt2o3nxn",
"projectId": "cmp55xnj20007pr27ycgb5ws5",
"deploymentFriendlyId": "deployment_gio4ig7x56f4qo8na9yv7",
"deploymentVersion": "20260522.5",
"runId": "cmphe6zlk0cd029mqdkrtpnmk",
"runFriendlyId": "run_cmphe6zlk0cd029mqdkrtpnmk",
"version": "1",
"nextAttemptNumber": 1,
"snapshotId": "cmpwlbg1k00zv28out3fwbcsk",
"snapshotFriendlyId": "snapshot_cmpwlbg1k00zv28out3fwbcsk",
"placementTags": [{ "key": "paid", "values": ["true"] }],
"traceContext": {
"traceparent": "00-9ab53f0d93628f6efb1721e5f44b2338-39761e6c1816cbaa-01"
},
"annotations": {
"triggerSource": "schedule",
"triggerAction": "trigger",
"rootTriggerSource": "schedule",
"rootScheduleId": "cmp8zcfzr10eqph299hxl40xk"
}
},
"containerCreateOpts": {
"name": "runner-cmphe6zlk0cd029mqdkrtpnmk",
"Hostname": "runner-cmphe6zlk0cd029mqdkrtpnmk",
"HostConfig": {
"AutoRemove": true,
"NetworkMode": "",
"NanoCpus": 500000000,
"Memory": 536870912
},
"Image": "gitea.sbtp.xyz/trigger/proj_zlvybyxzqtafjwonlxid:20260522.5.production.i88gmspz",
"AttachStdout": false,
"AttachStderr": false,
"AttachStdin": false
},
"startResult": { "type": "Buffer", "data": [] },
"containerId": "15e81243543ffa786c74e1bf402153ad1183d65685cc1ad07643d875ebeead1d"
}that looks good. I think this means the supervisor is receiving the task and de-queuing it. so that leaves the next part, which is creating the docker container in which the task will run. It must be failing at that part.
Maybe it’s a failure of docker-proxy compnent? I didn’t know what that was, so I asked trigger.dev’s AI assistant.
what is the docker-proxy component?
The Docker Socket Proxy is a security component included by default in the worker
docker-compose.yml. Instead of giving the supervisor direct access to the Docker socket (which would be a significant security risk), the proxy sits in between and only exposes the specific Docker API calls the supervisor actually needs.
As the docs put it: “No direct Docker socket access. The compose file now comes with Docker Socket Proxy by default. Yes, you want this.”
So it’s essentially a least-privilege guard for Docker socket access. You don’t need to configure it — it’s included automatically.
So are there any logs in my docker-proxy container?
2026-Jun-02 12:25:04 ::ffff:172.20.0.8:48034 [02/Jun/2026:12:25:04.397] dockerfrontend dockerbackend/dockersocket 0/0/0/279/279 204 152 - - ---- 1/1/0/0/0 0/0 "POST /containers/a9e3c85369311219a8cee3cb1ad6424d8b46394c530901d81bf3c27522f36b52/start HTTP/1.1"
2026-Jun-02 12:25:05 ::ffff:172.20.0.8:48034 [02/Jun/2026:12:25:05.619] dockerfrontend dockerbackend/dockersocket 0/0/0/55/55 200 2307 - - ---- 1/1/0/0/0 0/0 "GET /images/gitea.sbtp.xyz/trigger/proj_zlvybyxzqtafjwonlxid:20260524.2.production.k6z9ybcj/json HTTP/1.1"
2026-Jun-02 12:25:05 ::ffff:172.20.0.8:48034 [02/Jun/2026:12:25:05.683] dockerfrontend dockerbackend/dockersocket 0/0/0/165/165 201 281 - - ---- 1/1/0/0/0 0/0 "POST /containers/create?name=runner-cmpoa642i0ydc28s6fzqz8dei&Hostname=runner-cmpoa642i0ydc28s6fzqz8dei&HostConfig=%7B%22AutoRemove%22%3Atrue%2C%22NetworkMode%22%3A%22%22%2C%22NanoCpus%22%3A500000000%2C%22Memory%22%3A536870912%7D&Image=gitea.sbtp.xyz%2Ftrigger%2Fproj_zlvybyxzqtafjwonlxid%3A20260524.2.production.k6z9ybcj&AttachStdout=false&AttachStderr=false&AttachStdin=false&Env=OTEL_EXPORTER_OTLP_ENDPOINT%3Dhttps%3A%2F%2Ftrigger.sbtp.xyz%2Fotel&Env=TRIGGER_DEQUEUED_AT_MS%3D1780403105613&Env=TRIGGER_POD_SCHEDULED_AT_MS%3D1780403105619&Env=TRIGGER_ENV_ID%3Dcmp55xnjc0008pr27gt0c13dp&Env=TRIGGER_DEPLOYMENT_ID%3Ddeployment_xwj1q2epp8utcfnevk1ng&Env=TRIGGER_DEPLOYMENT_VERSION%3D20260524.2&Env=TRIGGER_RUN_ID%3Drun_cmpoa642i0ydc28s6fzqz8dei&Env=TRIGGER_SNAPSHOT_ID%3Dsnapshot_cmpwm1ge101eg28ouvf7pk6n5&Env=TRIGGER_SUPERVISOR_API_PROTOCOL%3Dhttp&Env=TRIGGER_SUPERVISOR_API_PORT%3D80
2026-Jun-02 12:25:06 ::ffff:172.20.0.8:48034 [02/Jun/2026:12:25:05.849] dockerfrontend dockerbackend/dockersocket 0/0/0/207/207 204 152 - - ---- 1/1/0/0/0 0/0 "POST /containers/8759a528f677ad77fb920a4516cbd3630d9ba795b7a317e7f6d877c00f18ed84/start HTTP/1.1"
2026-Jun-02 12:25:57 ::ffff:172.20.0.8:59334 [02/Jun/2026:12:25:57.112] dockerfrontend dockerbackend/dockersocket 0/0/0/40/40 200 2625 - - ---- 1/1/0/0/0 0/0 "GET /images/gitea.sbtp.xyz/trigger/proj_fkzqaojdaxeapkcmffmb:20260602.3.production.9dfczeuh/json HTTP/1.1"
2026-Jun-02 12:25:58 ::ffff:172.20.0.8:59334 [02/Jun/2026:12:25:57.154] dockerfrontend dockerbackend/dockersocket 0/0/0/1731/1731 201 281 - - ---- 1/1/0/0/0 0/0 "POST /containers/create?name=runner-cmpwj7of15p6f28s6pjtpohsq&Hostname=runner-cmpwj7of15p6f28s6pjtpohsq&HostConfig=%7B%22AutoRemove%22%3Atrue%2C%22NetworkMode%22%3A%22%22%2C%22NanoCpus%22%3A500000000%2C%22Memory%22%3A536870912%7D&Image=gitea.sbtp.xyz%2Ftrigger%2Fproj_fkzqaojdaxeapkcmffmb%3A20260602.3.production.9dfczeuh&AttachStdout=false&AttachStderr=false&AttachStdin=false&Env=OTEL_EXPORTER_OTLP_ENDPOINT%3Dhttps%3A%2F%2Ftrigger.sbtp.xyz%2Fotel&Env=TRIGGER_DEQUEUED_AT_MS%3D1780403157104&Env=TRIGGER_POD_SCHEDULED_AT_MS%3D1780403157110&Env=TRIGGER_ENV_ID%3Dcmpjbra4e01bts628jexljcy3&Env=TRIGGER_DEPLOYMENT_ID%3Ddeployment_7j5zj3tsyumxb3xd9jjfs&Env=TRIGGER_DEPLOYMENT_VERSION%3D20260602.3&Env=TRIGGER_RUN_ID%3Drun_cmpwj7of15p6f28s6pjtpohsq&Env=TRIGGER_SNAPSHOT_ID%3Dsnapshot_cmpwm2k4901f428ouej0ygdjv&Env=TRIGGER_SUPERVISOR_API_PROTOCOL%3Dhttp&Env=TRIGGER_SUPERVISOR_API_PORT%3D
2026-Jun-02 12:25:59 ::ffff:172.20.0.8:59334 [02/Jun/2026:12:25:58.886] dockerfrontend dockerbackend/dockersocket 0/0/0/179/179 204 152 - - ---- 1/1/0/0/0 0/0 "POST /containers/285e69ea3bce7d28bad8312e0c0273cda99189b35f4641921a489ac42ba7ea6e/start HTTP/1.1"
Those requests look fine. They’re all HTTP 2xx which would mean they’re succeeding.
Let’s look at running containers on the server.
root@coolwork2:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b2d9c546fa1d ghcr.io/triggerdotdev/supervisor:v4.4.6 "docker-entrypoint.s…" About an hour ago Up About an hour (healthy) 8000/tcp supervisor-xhhau5nzt9ddscesc19ltk3v
ab120511e7cb ghcr.io/triggerdotdev/trigger.dev:v4.4.6 "docker-entrypoint.s…" About an hour ago Up About an hour (healthy) 3000/tcp trigger-xhhau5nzt9ddscesc19ltk3v
c6702d4a3809 electricsql/electric:1.6.7 "/app/bin/entrypoint…" About an hour ago Up About an hour (healthy) electric-xhhau5nzt9ddscesc19ltk3v
fb63da346e97 postgres:16-alpine "docker-entrypoint.s…" About an hour ago Up About an hour (healthy) 5432/tcp postgresql-xhhau5nzt9ddscesc19ltk3v
a9be9960f2e9 tecnativa/docker-socket-proxy:v0.4.2 "docker-entrypoint.s…" About an hour ago Up About an hour (healthy) 2375/tcp docker-proxy-xhhau5nzt9ddscesc19ltk3v
c0ee3ef6f04e redis:7-alpine "docker-entrypoint.s…" About an hour ago Up About an hour (healthy) 6379/tcp redis-xhhau5nzt9ddscesc19ltk3v
fc800e92fd1b clickhouse/clickhouse-server:26.5 "/entrypoint.sh" About an hour ago Up About an hour (healthy) 8123/tcp, 9000/tcp, 9009/tcp clickhouse-xhhau5nzt9ddscesc19ltk3v
b63d73406178 ghcr.io/coollabsio/sentinel:0.0.21 "/app/sentinel" 10 days ago Up 10 days (healthy) coolify-sentinel
e1d73937e4b3 traefik:v3.6 "/entrypoint.sh --pi…" 10 days ago Up 10 days (healthy) 0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp, 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp, 0.0.0.0:443->443/udp, [::]:443->443/udp coolify-proxy
This looks like trigger.dev infrastrucutre is running, but where are the worker containers?
I went back and forth with ChatGPT. I started by posting my docker-compose file. It wanted to check that docker-proxy was able to communicate with supervisor. It had me run some wget from my supervisor container.
wget -qO- http://docker-proxy:2375/images/json || echo FAIL
This succeeded, and I got a bunch of JSON back. So that means that supervisor can communicate with docker-proxy, the service which actually creates the worker containers.
I had ChatGPT parse my supervisor logs and it found something I missed. Supervisor was succeeding in creating containers, they just weren’t sticking around, because of my config.
"HostConfig":{
"AutoRemove":true,
"NetworkMode":""
}
To troubleshoot further, I disabled autoremoval of containers by setting my env var DOCKER_AUTOREMOVE_EXITED_CONTAINERS=0
I restarted my trigger resource in Coolify.
next I checked that the changes were applied, by checking supervisor logs.
{"timestamp":"2026-06-02T15:41:34.781Z","message":"create succeeded","$name":"docker-workload-manager","$level":"debug","opts":{"dequeuedAt":"2026-06-02T15:41:33.701Z","dequeueResponseMs":9147,"pollingIntervalMs":1000,"warmStartCheckMs":0,"envId":"cmpjbra4e01bts628jexljcy3","envType":"PRODUCTION","image":"gitea.sbtp.xyz/trigger/proj_fkzqaojdaxeapkcmffmb:20260602.3.production.9dfczeuh@sha256:cd7ee819cc37b81e4cf827680b7845f3048ebe1236c960089cc1521d3d774674","machine":{"name":"small-1x","cpu":0.5,"memory":0.5,"centsPerMs":0},"orgId":"cmp55xi160004pr27rt2o3nxn","projectId":"cmpjbra3q01bss6284piok2et","deploymentFriendlyId":"deployment_7j5zj3tsyumxb3xd9jjfs","deploymentVersion":"20260602.3","runId":"cmpwk2jn6008z28ouv44jvshe","runFriendlyId":"run_cmpwk2jn6008z28ouv44jvshe","version":"1","nextAttemptNumber":1,"snapshotId":"cmpwt244w000s2kmuhfvf3ub7","snapshotFriendlyId":"snapshot_cmpwt244w000s2kmuhfvf3ub7","placementTags":[{"key":"paid","values":["true"]}],"traceContext":{"traceparent":"00-50abd3c63ab4adc059c03267e0a13507-39082500a5f42af8-01"},"annotations":{"triggerSource":"schedule","triggerAction":"trigger","rootTriggerSource":"schedule","rootScheduleId":"cmpmnfkku1454s6289eez275f"}},"containerCreateOpts":{"name":"runner-cmpwk2jn6008z28ouv44jvshe","Hostname":"runner-cmpwk2jn6008z28ouv44jvshe","HostConfig":{"AutoRemove":false,"NetworkMode":"","NanoCpus":500000000,"Memory":536870912},"Image":"gitea.sbtp.xyz/trigger/proj_fkzqaojdaxeapkcmffmb:20260602.3.production.9dfczeuh","AttachStdout":false,"AttachStderr":false,"AttachStdin":false},"startResult":{"type":"Buffer","data":[]},"containerId":"a3633cb8bbf2b12b0391b4bc16726409b6ae3da106dca7650e111e13795bcfd2"} {"timestamp":"2026-06-02T15:41:34.797Z","message":"create succeeded","$name":"docker-workload-manager","$level":"debug","opts":{"dequeuedAt":"2026-06-02T15:41:33.905Z","dequeueResponseMs":8914,"pollingIntervalMs":1000,"warmStartCheckMs":0,"envId":"cmp55xnjc0008pr27gt0c13dp","envType":"PRODUCTION","image":"gitea.sbtp.xyz/trigger/proj_zlvybyxzqtafjwonlxid:20260524.2.production.k6z9ybcj@sha256:31fce2048e11be391c505b49140b9cd072821fc525675b0abb7a6f3c7d7d9ee8","machine":{"name":"small-1x","cpu":0.5,"memory":0.5,"centsPerMs":0},"orgId":"cmp55xi160004pr27rt2o3nxn","projectId":"cmp55xnj20007pr27ycgb5ws5","deploymentFriendlyId":"deployment_xwj1q2epp8utcfnevk1ng","deploymentVersion":"20260524.2","runId":"cmpodcud90zj528s6qd5j9b5s","runFriendlyId":"run_cmpodcud90zj528s6qd5j9b5s","version":"1","nextAttemptNumber":1,"snapshotId":"cmpwt24aj000t2kmudwn6cmmp","snapshotFriendlyId":"snapshot_cmpwt24aj000t2kmudwn6cmmp","placementTags":[{"key":"paid","values":["true"]}],"traceContext":{"traceparent":"00-2805745b0c70278bd90591fa3943d5d8-d9edd473a12f9083-01"},"annotations":{"triggerSource":"schedule","triggerAction":"trigger","rootTriggerSource":"schedule","rootScheduleId":"cmp8zcfzr10eqph299hxl40xk"}},"containerCreateOpts":{"name":"runner-cmpodcud90zj528s6qd5j9b5s","Hostname":"runner-cmpodcud90zj528s6qd5j9b5s","HostConfig":{"AutoRemove":false,"NetworkMode":"","NanoCpus":500000000,"Memory":536870912},"Image":"gitea.sbtp.xyz/trigger/proj_zlvybyxzqtafjwonlxid:20260524.2.production.k6z9ybcj","AttachStdout":false,"AttachStderr":false,"AttachStdin":false},"startResult":{"type":"Buffer","data":[]},"containerId":"3992cba8c51f8ae31c809ff7c9dbbacec0fc19ed27b2e26594c498e70cec42b2"} {"timestamp":"2026-06-02T15:41:40.747Z","message":"GET /health","$name":"http-server","$level":"debug","url":"/health"}
"AutoRemove": false. Looks good.
Next, are we seeing any runner containers? we run this on the server’s terminal.
docker ps -a | grep runner-
success! there are several. here’s a sample
57f113d4abef gitea.sbtp.xyz/trigger/proj_fkzqaojdaxeapkcmffmb:20260602.1.staging.w4zh71qv "dumb-init node /app…" 4 minutes ago Exited (1) 4 minutes ago runner-cmpwsmrf000972jpdqops63a1
95e6425b036a gitea.sbtp.xyz/trigger/proj_fkzqaojdaxeapkcmffmb:20260602.3.production.9dfczeuh "dumb-init node /app…" 4 minutes ago Exited (1) 4 minutes ago runner-cmpwnucbh02f028ouwtdyxvdo
d94bf1c257c5 gitea.sbtp.xyz/trigger/proj_zlvybyxzqtafjwonlxid:20260524.2.production.k6z9ybcj "dumb-init node /app…" 4 minutes ago Exited (1) 4 minutes ago runner-cmpoda3xm0zhz28s6bgtkrzrs
and we log one of them
root@coolwork2:~# docker logs runner-cmpoda3xm0zhz28s6bgtkrzrs
{"timestamp":"2026-06-02T15:38:35.837Z","message":"[controller] Creating run controller","$name":"managed-run-logger","$level":"log","TRIGGER_CONTENT_HASH":"4d0d3675f57f47285db28ea40fd57993","TRIGGER_PROJECT_ID":"cmp55xnj20007pr27ycgb5ws5","TRIGGER_PROJECT_REF":"proj_zlvybyxzqtafjwonlxid","NODE_ENV":"production","NODE_EXTRA_CA_CERTS":"","UV_USE_IO_URING":"0","TRIGGER_DEPLOYMENT_ID":"deployment_xwj1q2epp8utcfnevk1ng","TRIGGER_DEPLOYMENT_VERSION":"20260524.2","TRIGGER_WORKLOAD_CONTROLLER_ID":"controller_deda2b9c-abb2-41ce-86d8-cc205c1e4be0","TRIGGER_ENV_ID":"cmp55xnjc0008pr27gt0c13dp","OTEL_EXPORTER_OTLP_ENDPOINT":"https://trigger.sbtp.xyz/otel","TRIGGER_WARM_START_CONNECTION_TIMEOUT_MS":30000,"TRIGGER_WARM_START_KEEPALIVE_MS":300000,"TRIGGER_MACHINE_CPU":"0.5","TRIGGER_MACHINE_MEMORY":"0.5","TRIGGER_RUNNER_ID":"runner-cmpoda3xm0zhz28s6bgtkrzrs","TRIGGER_POD_SCHEDULED_AT_MS":"2026-06-02T15:38:32.751Z","TRIGGER_DEQUEUED_AT_MS":"2026-06-02T15:38:32.640Z","TRIGGER_RUN_ID":"run_cmpoda3xm0zhz28s6bgtkrzrs","TRIGGER_SNAPSHOT_ID":"snapshot_cmpwsy8f100052kmu8qff422h","TRIGGER_SUPERVISOR_API_PROTOCOL":"http","TRIGGER_SUPERVISOR_API_DOMAIN":"","TRIGGER_SUPERVISOR_API_PORT":8020,"TRIGGER_WORKER_INSTANCE_NAME":"8e11de13-d0fe-4603-8f20-da923aafb86c","TRIGGER_HEARTBEAT_INTERVAL_SECONDS":20,"TRIGGER_SNAPSHOT_POLL_INTERVAL_SECONDS":5,"TRIGGER_SUCCESS_EXIT_CODE":0,"TRIGGER_FAILURE_EXIT_CODE":1,"runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpoda3xm0zhz28s6bgtkrzrs","runnerId":"runner-cmpoda3xm0zhz28s6bgtkrzrs","workerName":"8e11de13-d0fe-4603-8f20-da923aafb86c"}
node:internal/url:804
this.#updateContext(bindingUrl.parse(input, base));
^
TypeError: Invalid URL
at new URL (node:internal/url:804:36)
at ManagedRunController.createSupervisorSocket (file:///app/tmp/bunx-1000-trigger.dev@latest/node_modules/trigger.dev/dist/esm/entryPoints/managed-run-controller.mjs:6535:19)
at new ManagedRunController (file:///app/tmp/bunx-1000-trigger.dev@latest/node_modules/trigger.dev/dist/esm/entryPoints/managed-run-controller.mjs:6263:24)
at file:///app/tmp/bunx-1000-trigger.dev@latest/node_modules/trigger.dev/dist/esm/entryPoints/managed-run-controller.mjs:6682:1 {
code: 'ERR_INVALID_URL',
input: '/workload',
base: 'http://:8020'
}
Node.js v21.7.3
The smoking gun!
Looking at the supervisor env vars and our docker compose file, it looks like our TRIGGER_WORKLOAD_API_DOMAIN is unset. Because we’re running trigger.dev all-in-one on the same server, I set this env var to simply, supervisor.
I restarted the trigger.dev resource via Coolify. Then I checked docker logs again, looking for recently exited runner- containers.
root@coolwork2:~# docker logs runner-cmpwtujwh000g2ko0y2daim42
{"timestamp":"2026-06-02T16:03:45.272Z","message":"[controller] Creating run controller","$name":"managed-run-logger","$level":"log","TRIGGER_CONTENT_HASH":"1dee2582eb5ae981e393cf939bfa615d","TRIGGER_PROJECT_ID":"cmpjbra3q01bss6284piok2et","TRIGGER_PROJECT_REF":"proj_fkzqaojdaxeapkcmffmb","NODE_ENV":"production","NODE_EXTRA_CA_CERTS":"","UV_USE_IO_URING":"0","TRIGGER_DEPLOYMENT_ID":"deployment_xhkfsfrrbdzbfso8nay29","TRIGGER_DEPLOYMENT_VERSION":"20260602.1","TRIGGER_WORKLOAD_CONTROLLER_ID":"controller_b1cdeb01-a6c3-40c4-b2ac-f0dfaf04d120","TRIGGER_ENV_ID":"cmpjbra5u01bws628qu6m5e4s","OTEL_EXPORTER_OTLP_ENDPOINT":"https://trigger.sbtp.xyz/otel","TRIGGER_WARM_START_CONNECTION_TIMEOUT_MS":30000,"TRIGGER_WARM_START_KEEPALIVE_MS":300000,"TRIGGER_MACHINE_CPU":"0.5","TRIGGER_MACHINE_MEMORY":"0.5","TRIGGER_RUNNER_ID":"runner-cmpwtujwh000g2ko0y2daim42","TRIGGER_POD_SCHEDULED_AT_MS":"2026-06-02T16:03:41.552Z","TRIGGER_DEQUEUED_AT_MS":"2026-06-02T16:03:41.497Z","TRIGGER_RUN_ID":"run_cmpwtujwh000g2ko0y2daim42","TRIGGER_SNAPSHOT_ID":"snapshot_cmpwtukiz000i2ko0y4ythl7y","TRIGGER_SUPERVISOR_API_PROTOCOL":"http","TRIGGER_SUPERVISOR_API_DOMAIN":"supervisor","TRIGGER_SUPERVISOR_API_PORT":8020,"TRIGGER_WORKER_INSTANCE_NAME":"706813fc-aebf-4327-86b4-6174e0dd9dfd","TRIGGER_HEARTBEAT_INTERVAL_SECONDS":20,"TRIGGER_SNAPSHOT_POLL_INTERVAL_SECONDS":5,"TRIGGER_SUCCESS_EXIT_CODE":0,"TRIGGER_FAILURE_EXIT_CODE":1,"runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:45.301Z","message":"[controller] Starting up","$name":"managed-run-logger","$level":"log","runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:45.303Z","message":"[controller] startAndExecuteRunAttempt()","$name":"managed-run-logger","$level":"log","runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:45.307Z","message":"[poller] start","$name":"managed-run-logger","$level":"log","lastPollAt":null,"pollCount":0,"snapshotId":"snapshot_cmpwtukiz000i2ko0y4ythl7y","pollIntervalMs":5000,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:45.309Z","message":"[notifier] start","$name":"managed-run-logger","$level":"log","lastNotificationAt":null,"notificationCount":0,"lastInvalidNotificationAt":null,"invalidNotificationCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:45.310Z","message":"[execution] starting attempt","$name":"managed-run-logger","$level":"log","isWarmStart":"undefined","runId":"run_cmpwtujwh000g2ko0y2daim42","snapshotId":"snapshot_cmpwtukiz000i2ko0y4ythl7y","executionId":"4c720c60","executionRestoreCount":0,"runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:45.367Z","message":"[controller] Socket connection error","$name":"managed-run-logger","$level":"log","error":"websocket error","runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:46.249Z","message":"[controller] Socket connection error","$name":"managed-run-logger","$level":"log","error":"websocket error","runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:48.554Z","message":"[controller] Socket connection error","$name":"managed-run-logger","$level":"log","error":"websocket error","runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:50.310Z","message":"[poller] polling for latest snapshot","$name":"managed-run-logger","$level":"log","lastPollAt":null,"pollCount":0,"snapshotId":"snapshot_cmpwtukiz000i2ko0y4ythl7y","pollIntervalMs":5000,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:52.917Z","message":"[controller] Socket connection error","$name":"managed-run-logger","$level":"log","error":"websocket error","runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
Failed to send debug log { success: false, error: 'Connection error.' }
Failed to send debug log { success: false, error: 'Connection error.' }
Failed to send debug log { success: false, error: 'Connection error.' }
Failed to send debug log { success: false, error: 'Connection error.' }
Failed to send debug log { success: false, error: 'Connection error.' }
Failed to send debug log { success: false, error: 'Connection error.' }
Failed to send debug log { success: false, error: 'Connection error.' }
{"timestamp":"2026-06-02T16:03:53.082Z","message":"[execution] failed to start attempt","$name":"managed-run-logger","$level":"log","error":"Start API call failed: Connection error.","runId":"run_cmpwtujwh000g2ko0y2daim42","snapshotId":"snapshot_cmpwtukiz000i2ko0y4ythl7y","executionId":"4c720c60","executionRestoreCount":0,"runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:53.083Z","message":"[execution] [shutdown] failed to start attempt","$name":"managed-run-logger","$level":"log","runId":"run_cmpwtujwh000g2ko0y2daim42","snapshotId":"snapshot_cmpwtukiz000i2ko0y4ythl7y","executionId":"4c720c60","executionRestoreCount":0,"runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:53.084Z","message":"[poller] stop","$name":"managed-run-logger","$level":"log","lastPollAt":"2026-06-02T16:03:50.311Z","pollCount":1,"snapshotId":"snapshot_cmpwtukiz000i2ko0y4ythl7y","pollIntervalMs":5000,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:53.084Z","message":"[poller] stopped while executing","$name":"managed-run-logger","$level":"log","lastPollAt":"2026-06-02T16:03:50.311Z","pollCount":1,"snapshotId":"snapshot_cmpwtukiz000i2ko0y4ythl7y","pollIntervalMs":5000,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:53.085Z","message":"[snapshot] stop","$name":"managed-run-logger","$level":"log","snapshotId":"snapshot_cmpwtukiz000i2ko0y4ythl7y","status":"PENDING_EXECUTING","suspendable":false,"queueLength":0,"isProcessingQueue":false,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:53.086Z","message":"[notifier] stop","$name":"managed-run-logger","$level":"log","lastNotificationAt":null,"notificationCount":0,"lastInvalidNotificationAt":null,"invalidNotificationCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:53.087Z","message":"[controller] waitForNextRun()","$name":"managed-run-logger","$level":"log","runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:53.087Z","message":"[controller] waitForNextRun: warm starts disabled, shutting down","$name":"managed-run-logger","$level":"log","runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:53.088Z","message":"[controller] Exiting process","$name":"managed-run-logger","$level":"log","code":0,"runnerWarmStartCount":0,"runnerRestoreCount":0,"runId":"run_cmpwtujwh000g2ko0y2daim42","runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
{"timestamp":"2026-06-02T16:03:53.089Z","message":"[execution] [shutdown] kill (already shutting down)","$name":"managed-run-logger","$level":"log","firstShutdownReason":"failed to start attempt","runId":"run_cmpwtujwh000g2ko0y2daim42","snapshotId":"snapshot_cmpwtukiz000i2ko0y4ythl7y","executionId":"4c720c60","executionRestoreCount":0,"runnerId":"runner-cmpwtujwh000g2ko0y2daim42","workerName":"706813fc-aebf-4327-86b4-6174e0dd9dfd"}
now we’ve got new errors.
`Failed to send debug log { success: false, error: 'Connection error.' }`
Could this be because of "NetworkMode": ""? Let’s experiment, by specifying the network in our docker-compose.
networks: trigger: name: trigger external: true
Also we need to make sure we set DOCKER_RUNNER_NETWORKS=trigger in env.
long story short, configuring networks in docker-compose is circumventing the Coolify managed internal network. Doing that made it so my compose stack was no longer in communication with my reverse proxy, which meant that I could no longer log into the trigger web ui.
I backtracked, removing all the network specs from the docker compose file.
Then, I found the docker network name that Coolify is creating.
docker inspect trigger-xhhau5nzt9ddscesc19ltk3v --format '{{json .NetworkSettings.Networks}}'
I set DOCKER_RUNNER_NETWORKS=trigger-xhhau5nzt9ddscesc19ltk3v env var and restarted the Coolify resource.
Success! test tasks are running.
In conclusion, our issue was two-fold. We had two environment variables which were not configured, which led to runner containers being unable to communicate with the rest of the system.
DOCKER_RUNNER_NETWORKS was undefined. TRIGGER_WORKLOAD_API_DOMAIN was undefined.
Update 2026-06-07
It’s happening again. development env is fine, production is not. Production runs get stuck in dequeued, with no logs. After 10 minutes, they retry, and fail again. No logs.
First thing I do is I verify that DOCKER_RUNNER_NETWORKS and TRIGGER_WORKLOAD_API_DOMAIN are defined in coolify. They are, and the values look correct.
Actually, maybe DOCKER_RUNNER_NETWORKS might be wrong. it is set to xhhau5nzt9ddscesc19ltk3v but my notes say it should be trigger-xhhau5nzt9ddscesc19ltk3v. I changed it back to xhhau5nzt9ddscesc19ltk3v and restarted the compose stack.
Just as I restarted, I saw that tasks were running again, because I was running a test-email task that simply sends me a test e-mail. But then the stack restarted, and wasn’t coming back up. All services red. Was that env var actually wrong? Logs show errors
Error response from daemon: No such container: supervisor-xhhau5nzt9ddscesc19ltk3v
let me change it back.
Ok so
DOCKER_RUNNER_NETWORKS=xhhau5nzt9ddscesc19ltk3v, set in Coolify environment variables under my trigger app. Deploy, and wait for services to come back up. If this works, we might have had some other, weirder issue at play. Maybe trigger compose stack just needed a restart?
All but supervisor and trigger services came back up. Supervisor and trigger are stuck in yellow. No logs. wtf is going on?
I restarted.
All services green.
Nothing is running. Wait a second, there are tasks that are waiting on wait tokens. Was the queue full or something? Another delayed test-email task just went through and I received an e-mail. Ok so maybe things are just delayed because there is a bunch of task processing going on.
I guess I’ll wait, be patient, and make sure that tasks don’t get stuck waiting for humans (wait tokens.) But yeah, the DOCKER_RUNNER_NETWORKS should definitely be xhhau5nzt9ddscesc19ltk3v.
Update 2026-06-23
Still tasks are not running. There are 13K tasks that have been stuck at Queued for days. None are dequeueing.
Curiously, this issue seems to affect scheduled tasks only. I am able to run a test task just fine. The test task deques and executes. The scheduled task on the other hand, does not dequeue.
I upgraded system packages, restarted trigger, cleared the queue, and re-deployed my latest trigger tasks and that was enough to get the scheduled tasks dequeueing and executing again.
Update 2026-07-19
It’s happening again. Task are not dequeueing. I
- Cancelled all tasks in the queue.
- Updated system packages.
shutdown -r nowthe system.- Restarted trigger in Coolify
- Re-deployed trigger
bunx trigger.dev@latest deploy --api-url https://trigger.sbtp.xyzDuring the trigger deployment, there were 3 outdated npm packages that trigger asked to update. I said yes.
Still, no tasks are being dequeued. 😠
I checked server CPU usage in Vultr. It’s fluctuating between 30-60%. Running uptime on the terminal painted a different picture.
load average: 6.11, 6.33, 6.47
Yeah, that’s a stressed system. Do we need to beef up the server?
We have 4 vCPU, 8192 MB RAM, 160 GB SSD, 4.00 TB Transfer. That seems fine for the task we run on it. I think we pay 80/mo. for upgrade. Let’s continue to play with the settings to see if software is the issue.
I ran docker stats and looked at what was running
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
6d9ad799e5db coolify-sentinel 0.00% 13.5MiB / 7.749GiB 0.17% 62.8kB / 1.01MB 5.35MB / 32.8kB 9
7b93cbb9dcf8 supervisor-xhhau5nzt9ddscesc19ltk3v 1.82% 139.3MiB / 7.749GiB 1.75% 21.8MB / 24.1MB 1.48MB / 0B 23
dc013c1a7857 trigger-xhhau5nzt9ddscesc19ltk3v 54.29% 798.6MiB / 7.749GiB 10.06% 427MB / 292MB 194MB / 49.2kB 17
3608a074f81f electric-xhhau5nzt9ddscesc19ltk3v 1.01% 233.3MiB / 7.749GiB 2.94% 3.89MB / 6.32MB 40.2MB / 32.8kB 32
1db5c5f90768 postgresql-xhhau5nzt9ddscesc19ltk3v 5.04% 168.2MiB / 7.749GiB 2.12% 57.9MB / 272MB 1.21GB / 118MB 49
cc8a08a28243 docker-proxy-xhhau5nzt9ddscesc19ltk3v 0.00% 11.25MiB / 7.749GiB 0.14% 1.73MB / 1.77MB 3.15MB / 4.1kB 5
2eb52a31d963 redis-xhhau5nzt9ddscesc19ltk3v 2.25% 48.02MiB / 7.749GiB 0.61% 192MB / 78.9MB 2.4MB / 307MB 6
bdc4271c66a9 clickhouse-xhhau5nzt9ddscesc19ltk3v 71.61% 1.607GiB / 7.749GiB 20.74% 27.6MB / 13.1MB 397MB / 8.83GB 872
37169df0aaca coolify-proxy 2.60% 39.12MiB / 7.749GiB 0.49% 63.2MB / 81MB 11.6MB / 0B 11
fa71057afdfe runner-cmrsbmiuj00t72knvwbdgtxh3 51.75% 68.3MiB / 512MiB 13.34% 51.4kB / 95.7kB 0B / 0B 23
0898277c05f6 runner-cmrsbmmpv00t82knv114w3m00 48.50% 70.96MiB / 512MiB 13.86% 85.3kB / 87.9kB 0B / 0B 23
21155ff98d22 runner-cmrplfme82asv28rwjzczwpbm 19.29% 101.3MiB / 512MiB 19.78% 36.2kB / 35.1kB 0B / 0B 23
watching the live output, I’m seeing that runner-xxx containers are coming and going. So it seems like trigger tasks are running, we just aren’t seeing their story displayed in the web UI. wtf?
I checked some of my other Trigger projects. Confettihat had a running task! Ok so it’s just that futurenet doesn’t have any progress?
ok so back to futurenet, i paused the queue, and then resumed it, just in case there’s weirdness there.
Curiously, on https://trigger.sbtp.xyz/orgs/sbtpxyz-8bb8/projects/futureporn-APGR/env/prod/queues I see that the Running count goes up and down, as if things are running, but they’re just not showing in the details page at https://trigger.sbtp.xyz/orgs/sbtpxyz-8bb8/projects/futureporn-APGR/env/prod/runs?statuses=DEQUEUED&statuses=EXECUTING&period=30d&rootOnly=false. wtf!?
That page is completely empty.
No runs match your filters. Try refreshing, modifying your filters or run a test.
Ok so I realized the queue was NOT cleared— there were 1K+ queued tasks. I cleared the tasks, and now everything is working as expected.
I had a buggy cron which repeatedly re-added the same task with same args. I pushed new code to make new task creation idempotent, which should help.