Orchestrator を Mackerel と組み合わせて幸せを掴むスクリプト

errant transaction の検知

function check_errant {
for cluster in "${CLUSTERS[@]}"
do
replicas=( $( orchestrator -c which-replicas -i "${cluster}") )
for replica in "${replicas[@]}"
do
local result=$( orchestrator -c which-gtid-errant -i "${replica}" )
if [ -n "${result}" ]; then
inject_empty_transaction "${replica}"
fi
done
done
}

レプリカが errant transaction を持っていた場合は、
inject_empty_transaction を実行する。

local result=$( orchestrator -c which-gtid-errant -i "${replica}" )
if [ -n "${result}" ]; then
inject_empty_transaction "${replica}"
fi

errant transaction の修正

function inject_empty_transaction {
local replica="${1}"
orchestrator -c gtid-errant-inject-empty -i "${replica}"
}

なんで問答無用で inject empty transaction してるかというと、、、
次の記事に書く予定。

can-replicate-from-gtid を叩く

function can_replicate {
for cluster in "${CLUSTERS[@]}"
do
replicas=( $( orchestrator -c which-replicas -i "${cluster}") )
for replica_i in "${!replicas[@]}"
do
for other_i in "${!replicas[@]}"
do
if [ ${replica_i} -eq ${other_i} ]; then
continue
fi
source_replica="${replicas[${replica_i}]}"
destination="${replicas[${other_i}]}"
set +e
result=$( orchestrator -c can-replicate-from-gtid -i "${source_replica}" -d "${destination}" )
if [ $? -ne 0 ]; then
echo -e "${source_replica} to ${destination} cannot replicate\nError: ${result}\nCheck errant gtid http://prd-orchestrator01.mogg.qualiarts.vm.tky02a.cycloud.io:3000/web/cluster/alias/prd-dbs07"
exit 2
fi
set -e
done
done
done
}

マスターを除く全てのレプリカ同士でレプリケーションが貼れるかどうかを調べる。