[Solved] Acme.sh DNS challenge

Something seems to have changed in ISPC 3.3.0(px) regarding API.
I've been using acme.sh with ispc dns challenges on several different systems (firewalls, proxmox, different linux flavors) for a long time without any problems (except the occasional user error).
Now that I've updated to ISP 3.3.0(p2) I can't renew any of the certificates anymore.
Everywhere an error is returned saying the TXT record couldn't be added. Therefor the challenge can't be executed and the renewal fails.

Since this error is returned everywhere it's in no way caused by the acme.sh script.
Several of the systems have acme.sh build-in so don't have the latest version. A new/updated version of acme.sh causing it is therefor out of the question.

An example:
(redacted error from my firewall)

Code:

Firewall
Renewing certificate
account: LetsEncrypt
server: letsencrypt-production-2

/usr/local/pkg/acme/acme.sh --issue --domain '<sub1.domain.tld>' --dns 'dns_ispconfig' --domain '<sub3.domain.tld>' --dns 'dns_ispconfig' --domain '<sub3.domain.tld>' --dns 'dns_ispconfig' --home '/tmp/acme/Firewall/' --accountconf '/tmp/acme/Firewall/accountconf.conf' --force --always-force-new-domain-key --reloadCmd '/tmp/acme/Firewall/reloadcmd.sh' --dnssleep '900' --log-level 3 --log '/tmp/acme/Firewall/acme_issuecert.log'
Array
(
[path] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
[PATH] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
[SSL_CERT_DIR] => /etc/ssl/certs/
[ISPC_User] => <api user>
[ISPC_Password] => <api password>
[ISPC_Api] => https://<ispc server>/remote/json.php
[ISPC_Api_Insecure] => 1
)
[Fri Jul 18 13:17:57 CEST 2025] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Fri Jul 18 13:17:57 CEST 2025] Using pre-generated key: /tmp/acme/Firewall/<sub1.domain.tld>/<sub1.domain.tld>.key.next
[Fri Jul 18 13:17:57 CEST 2025] Generating next pre-generate key.
[Fri Jul 18 13:17:58 CEST 2025] Multi domain='DNS:<sub1.domain.tld>,DNS:<sub2.domain.tld>,DNS:<sub3.domain.tld>'
[Fri Jul 18 13:18:02 CEST 2025] Getting webroot for domain='<sub1.domain.tld>'
[Fri Jul 18 13:18:02 CEST 2025] Getting webroot for domain='<sub2.domain.tld>'
[Fri Jul 18 13:18:02 CEST 2025] Getting webroot for domain='<sub3.domain.tld>'
[Fri Jul 18 13:18:02 CEST 2025] Adding TXT value: <challenge> for domain: _acme-challenge.<sub1.domain.tld>
[Fri Jul 18 13:18:02 CEST 2025] Getting Session ID
[Fri Jul 18 13:18:02 CEST 2025] Retrieved Session ID.
[Fri Jul 18 13:18:02 CEST 2025] Getting Zoneinfo
[Fri Jul 18 13:18:02 CEST 2025] Retrieved zone data.
[Fri Jul 18 13:18:02 CEST 2025] Retrieved Server ID
[Fri Jul 18 13:18:02 CEST 2025] Retrieved Zone ID
[Fri Jul 18 13:18:02 CEST 2025] Retrieved SYS User ID.
[Fri Jul 18 13:18:02 CEST 2025] Retrieved Client ID.
[Fri Jul 18 13:18:02 CEST 2025] Couldn't add ACME Challenge TXT record to zone.
[Fri Jul 18 13:18:02 CEST 2025] Error adding TXT record to domain: _acme-challenge.<sub1.domain.tld>

 

Can you please enable DEBUG and rerun the command again?

It seems to fail in this part of the provider:

Code:

_ISPC_addTxt() {
  curSerial="$(date +%s)"
  curStamp="$(date +'%F %T')"
  params="\"server_id\":\"${server_id}\",\"zone\":\"${zone}\",\"name\":\"${fulldomain}.\",\"type\":\"txt\",\"data\":\"${txtvalue}\",\"aux\":\"0\",\"ttl\":\"3600\",\"active\":\"y\",\"stamp\":\"${curStamp}\",\"serial\":\"${curSerial}\""
  curData="{\"session_id\":\"${sessionID}\",\"client_id\":\"${client_id}\",\"params\":{${params}},\"update_serial\":true}"
  curResult="$(_post "${curData}" "${ISPC_Api}?dns_txt_add")"
  _debug "Calling _ISPC_addTxt: '${curData}' '${ISPC_Api}?dns_txt_add'"
  _debug "Result of _ISPC_addTxt: '$curResult'"
  record_id=$(echo "${curResult}" | _egrep_o "\"response.*" | cut -d ':' -f 2 | cut -d '"' -f 2)
  _debug "Record ID: '${record_id}'"
  case "${record_id}" in
  '' | *[!0-9]*)
    _err "Couldn't add ACME Challenge TXT record to zone."
    return 1
    ;;
  *) _info "Added ACME Challenge TXT record to zone." ;;
  esac
}

 

I think I found the bug!

Result of _ISPC_addTxt: '{"code":"remote_fault","message":"ALTER command denied to user '<ispc user>'@'<ispc host>' for table `<ispc db>`.`mail_user` INSERT INTO `dns_rr` (`server_id`, `zone`, `name`, `type`, `data`, `ttl`, `active`, `stamp`, `serial`, `sys_userid`, `sys_groupid`, `sys_perm_user`, `sys_perm_group`, `sys_perm_other`) VALUES ('<server id>', '<zone id>', '_acme-challenge.<sub1.domain.tld>.', 'txt', '<challenge>', '<ttl>', 'y', '<timestamp>', '<serial>', '<sys user id>', '<sys group id>', 'riud', 'riud', '')","response":false}'

<ispc db>.mail_user cannot be right!

Now to find the file that needs to be corrected ...

 

Yea I've check already before my initial reply. Maybe @remkoh changed some things around in his setup for testing.

 

I've changed nothing (of any significance).
Only updated from the latest 3.2.12 to 3.3.0 and shortly after 3.3.0p2.
Everything was working fine when running 3.2.12

Which file/script constructs the sql query that's at fault?

 

That's rather strange. Can you diff the local interface/lib/classes/remote.d/dns.inc.php against the current version in git?

I'll have a look later if that does not bring up anything.

 

They are identical

 

I do run all databases on 2 dedicated db servers (galera synced) including ISPC's master db.
But that hasn't been any problem this far.
And doesn't explain the wrong db in the query in any way.

The ISPC panel is still working just fine.
I only use the API for acme.sh dns challenges so I can't tell if any other API features might experience similar issues.

 

I dont think this has anything to do with it. I will check on a Dev VM later

 

Mh this seem to work flawless.
System: ispcdev.net.local (Debian Bookworm) ISPConfig 3.3.0p1

I tested the API dns_txt_add function which worked fine. I then setup acme.sh and the dns_ispconfig provider against the dev machine which worked just fine too.

Code:

[Fri Jul 18 01:37:15 PM EDT 2025] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Fri Jul 18 01:37:15 PM EDT 2025] Creating domain key
[Fri Jul 18 01:37:15 PM EDT 2025] The domain key is here: /root/.acme.sh/pyte.dev_ecc/pyte.dev.key
[Fri Jul 18 01:37:15 PM EDT 2025] Multi domain='DNS:pyte.dev,DNS:www.pyte.dev'
[Fri Jul 18 01:37:20 PM EDT 2025] Getting webroot for domain='pyte.dev'
[Fri Jul 18 01:37:20 PM EDT 2025] Getting webroot for domain='www.pyte.dev'
[Fri Jul 18 01:37:20 PM EDT 2025] Adding TXT value: m0iePo4xx-rPWrW02WaLou_-ZcVDpu2hd6AYmyzpygM for domain: _acme-challenge.pyte.dev
[Fri Jul 18 01:37:20 PM EDT 2025] Getting Session ID
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved Session ID.
[Fri Jul 18 01:37:20 PM EDT 2025] Getting Zoneinfo
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved zone data.
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved Server ID
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved Zone ID
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved SYS User ID.
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved Client ID.
[Fri Jul 18 01:37:20 PM EDT 2025] Added ACME Challenge TXT record to zone.
[Fri Jul 18 01:37:20 PM EDT 2025] The TXT record has been successfully added.
[Fri Jul 18 01:37:20 PM EDT 2025] Adding TXT value: doV_h2pSbyKGWHMvo3YTIzyqhII_e7zZxKODXeRTJaw for domain: _acme-challenge.www.pyte.dev
[Fri Jul 18 01:37:20 PM EDT 2025] Getting Session ID
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved Session ID.
[Fri Jul 18 01:37:20 PM EDT 2025] Getting Zoneinfo
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved zone data.
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved Server ID
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved Zone ID
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved SYS User ID.
[Fri Jul 18 01:37:20 PM EDT 2025] Retrieved Client ID.
[Fri Jul 18 01:37:21 PM EDT 2025] Added ACME Challenge TXT record to zone.
[Fri Jul 18 01:37:21 PM EDT 2025] The TXT record has been successfully added.
[Fri Jul 18 01:37:21 PM EDT 2025] Let's check each DNS record now. Sleeping for 20 seconds first.
[Fri Jul 18 01:37:42 PM EDT 2025] You can use '--dnssleep' to disable public dns checks.
[Fri Jul 18 01:37:42 PM EDT 2025] See: https://github.com/acmesh-official/acme.sh/wiki/dnscheck
[Fri Jul 18 01:37:42 PM EDT 2025] Checking pyte.dev for _acme-challenge.pyte.dev
[Fri Jul 18 01:37:49 PM EDT 2025] Not valid yet, let's wait for 10 seconds then check the next one.

I then updated to 3.3.0p2 and rerun the tests again with the same outcome. Here are the records that the acme.sh added:

 

Very strange!

My API user only has client functions, domaintool functions, dns zone function and dns txt functions checked.
It has been so since the user exists.

Checked all nodes cron-logs in search of any other sql issues but none are logged.

 

This is the check scripted pieced together(I usually use a class based wrapper around the api sorry). You have to edit the clientid, server_id and zone accordingly.

No worries. This is a local dev machine with random generated pws on deploy.

PHP:

<?php
$ispcLocation = 'https://192.168.122.252:8080/remote/index.php';
$ispcUri = 'https://192.168.122.252:8080/remote/';
$ispcUser = 'devapi';
$ispcPass = 'VHcaymR9sss';

$soapClient = new SoapClient(null, array(
    'location' => $ispcLocation,
    'uri' => $ispcUri,
    'stream_context' => stream_context_create(array(
        'ssl' => array(
            'verify_peer' => false,
            'verify_peer_name' => false
        )
    ))
));

try {
    $sessionId = $soapClient->login($ispcUser, $ispcPass);
} catch (SoapFault $e) {
    die('SOAP Login Error: ' . $e->getMessage());
}

$timestamp = date('Y-m-d H:i:s');

$clientId = 1;
$data = array(
    'server_id' => 1,
    'zone' => 1,
    'name' => 'acme',
    'type' => 'txt',
    'data' => '001jSDSAKLDdj12hjq08dih21',
    'aux' => '0',
    'ttl' => '300',
    'active' => 'y',
    'stamp' => $timestamp,
);

try {
    $id = $soapClient->dns_txt_add($sessionId, $clientId, $data);
    echo "ID: " . $id . "<br>";
} catch (SoapFault $e) {
    die('Error adding TXT record: ' . $e->getMessage());
}

if ($soapClient->logout($sessionId)) {
    echo 'Logged out.<br />';
} else {
    echo 'Logout failed.<br />';
}

Anyways what i saw when looking through this thread again is the message:

Can you please post the permission that the API user has configured? Maybe there is something wrong with the api permission checking?

 

Checked again with only reduced permission with the one mentioned by you. Still both tests work properly. Did you ever edit any files of the ISPConfig Installation manualy?

And maybe test if it works with all API permissions enabled.

 

How do I do that?
I just created a remote user ages ago and have set username, password, checked remote access, no remote access ip's so any and the checked 4 function mentioned earlier.
There is nothing about permissions.

None anywhere remotely related to this issue.
Last edit was solving this dnssec issue:
https://git.ispconfig.org/ispconfig/ispconfig3/-/issues/6715
https://forum.howtoforge.com/threads/dnssec-denial-of-existence-warnings.92517/#post-457607

I'll give your script a go.

 

The activated functions is what I was referring to with "permissions". Check all boxes and see if that changes anything related to your issue.

 

Your script is returning the exact same error I posted earlier:

Code:

Error adding TXT record: ALTER command denied to user '<ispc user>'@'<ispc host>' for table `<ispc master db>`.`mail_user` INSERT INTO `dns_rr` (`server_id`, `zone`, `name`, `type`, `data`, `ttl`, `active`, `stamp`, `serial`, `sys_userid`, `sys_groupid`, `sys_perm_user`, `sys_perm_group`, `sys_perm_other`) VALUES ('12', '8', 'acme', 'txt', '001jSDSAKLDdj12hjq08dih21', '300', 'y', '2025-07-18 20:40:14', '0', '2', '2', 'riud', 'riud', '')

Also adding .`mail_user` behind the db name.
I haven't got the faintest idea where it's coming from.

Checking all remote user's functions doesn't change a thing.