Cluster Deployment & Initialization

5. Apply configuration once #

After all XML changes and network rules are in place, restart ClickHouse Server once per existing node, preferably one node at a time:

systemctl restart clickhouse-server

On a new node where ClickHouse has not yet been started:

systemctl start clickhouse-server

Confirm that port 9000 is reachable between nodes. No separate restarts are needed for individual XML files.

6. Verify the cluster #

Run from the NFA node:

clickhouse-client -q "
SELECT shard_num, replica_num, host_name
FROM system.clusters
WHERE cluster = 'nfa'
ORDER BY shard_num, replica_num"

clickhouse-client -q "
SELECT hostName(), version()
FROM clusterAllReplicas('nfa', system.one)
ORDER BY hostName()"

All configured nodes must appear without errors. On each new node, verify that ClickHouse can communicate with Keeper:

clickhouse-client -q "
SELECT name FROM system.zookeeper WHERE path = '/'"

Important: Do not proceed if any node is unreachable, inter-node authentication fails, or Keeper queries fail.

7. Initialize the NFA database #

Run the following only on the NFA node, after cluster connectivity is confirmed:

nfaadm --init ts

This step is mandatory after adding a shard or replica. Wait for successful completion.

8. Basic post-initialization checks #

Confirm the NFA database exists on all nodes:

clickhouse-client -q "
SELECT hostName(), name
FROM clusterAllReplicas('nfa', system.databases)
WHERE name = 'nfa'
ORDER BY hostName()"

Check replica availability:

clickhouse-client -q "
SELECT hostName(), table, is_readonly, active_replicas, total_replicas
FROM clusterAllReplicas('nfa', system.replicas)
WHERE database = 'nfa'
ORDER BY table, hostName()"

Expected: is_readonly = 0 and, once synchronization finishes, active_replicas = total_replicas. Initial synchronization of a new replica may take time.

9. Restart NFA services #

systemctl restart nfa.target

The expanded cluster is ready for normal NFA operation once initialization and the checks above complete successfully.