{"id":191,"date":"2010-10-01T09:29:38","date_gmt":"2010-10-01T09:29:38","guid":{"rendered":"http:\/\/10sa.com\/sql_stories\/?p=191"},"modified":"2014-01-10T13:16:28","modified_gmt":"2014-01-10T13:16:28","slug":"my-sphinx-search-replication-master-and-slaves","status":"publish","type":"post","link":"http:\/\/10sa.com\/sql_stories\/?p=191","title":{"rendered":"My Sphinx Search replication (master and slaves)"},"content":{"rendered":"

One instance and one storage with indices is neither secure nor effective.\u00a0 That is why I\u00a0created\u00a0my\u00a0replication\u00a0for Sphinx Search\u00a0<\/a>. To put it shortly, my Sphinx’s architecture constists of tree indices and MVA (multivalue attributes stored in mva_updates_pool<\/a>) – that is why I cannot use RT index.<\/p>\n

My first idea was to create independent Sphix index + searchd(aemon) per machine which would be self-sufficient – but it has tree disadvantagies:
\n– there is one source (i.e. database :));
\n– advanced structure in database (as manager function);
\n– does not conform to KISS;<\/p>\n

Therefore,\u00a0 a known solution from ancient times – one master many slaves – seems to be better.
\nMaster works as usual and its additional requirement is NFS server ON. You should configure your iptables – this is my shot (considering that slave is 192.168.1.51):<\/p>\n

ACCEPT     tcp  --  192.168.1.51            0.0.0.0\/0           tcp dpt:32803\r\nACCEPT     udp  --  192.168.1.51            0.0.0.0\/0           udp dpt:32769\r\nACCEPT     tcp  --  192.168.1.51            0.0.0.0\/0           tcp dpt:2049\r\nACCEPT     udp  --  192.168.1.51            0.0.0.0\/0           udp dpt:2049\r\nACCEPT     tcp  --  192.168.1.51            0.0.0.0\/0           tcp dpt:892\r\nACCEPT     udp  --  192.168.1.51            0.0.0.0\/0           udp dpt:892\r\nACCEPT     tcp  --  192.168.1.51            0.0.0.0\/0           tcp dpt:875\r\nACCEPT     udp  --  192.168.1.51            0.0.0.0\/0           udp dpt:875\r\nACCEPT     udp  --  192.168.1.51            0.0.0.0\/0           udp dpt:111\r\nACCEPT     tcp  --  192.168.1.51            0.0.0.0\/0           tcp dpt:111<\/pre>\n
Share master storage giving adequate writing to \/etc\/exports<\/p>\n
\/storage\/ 192.168.1.51(rw,no_subtree_check,sync,no_root_squash)<\/pre>\n
Now slave, firstly you should mount master through nfs in yout fstab:<\/p>\n
# 192.168.1.50 is master\r\n192.168.1.50:\/storage      \/storage_master         nfs     ro,rsize=8192,hard,intr,nfsvers=3,tcp,noatime,nodev,async 0 0<\/pre>\n
Ok, slave sees master storage, now I would like to explain how my script works:
\n1. Script observes whether master’s index change;
\n2. In case master’s index has been rebuilt it copies all files conform to sphinx rotate requirements (conforming to Sphinx sources);
\n3. It executes special stop\/start file with option rotate (I present this file as \/etc\/init.d\/sphinx)<\/p>\n
\/etc\/init.d\/sphinx<\/h3>\n
How to install<\/h4>\n
On redhat\/centos Os you execute:<\/p>\n
chkconfig --add sphinx<\/pre>\n
Content<\/h4>\n
#!\/bin\/bash\r\n\r\n# inits, open order, close order\r\n# chkconfig: 2345 99 10\r\n# description: Sphinx daemon\r\nHOME_DIR=\"\/usr\/local\/services\/sphinx\/bin\/\"\r\nSTORAGE_DIR=\"\/storage\"\r\nNAME=\"searchd\"\r\nCONFIG=\"\/usr\/local\/services\/sphinx\/etc\/arch.cnf\"\r\niterator=2\r\nMAX_ATTEMPT=5\r\naINDEXES=(daily weekly archive)\r\nINDEX_PID_EXT=\"spl\"\r\n\r\nif [ ! -r $CONFIG ]; then\r\n    echo \"No config file or cannot be read. Script has been stopped\"\r\n    exit 0\r\nfi\r\n\r\nif [ ! -d $HOME_DIR ]; then\r\n    echo \"No home dir. Script has been stopped\"\r\n    exit 0\r\nfi\r\n# Function unlinks files containg Sphinxa indices' pids  (they exists when daemon is ugly killed)\r\ndel_index_pid(){\r\n    element_count=${#aINDEXES[@]}\r\n    index=0\r\n    while [ \"$index\" -lt \"$element_count\" ]\r\n    do\r\n\tINDEX_FILE=${aINDEXES[$index]}\r\n\tINDEX_FILE_PATH=\"$STORAGE_DIR\/$INDEX_FILE.$INDEX_PID_EXT\"\r\n\tif [ -f  \"$INDEX_FILE_PATH\" ]; then\r\n\t    rm $INDEX_FILE_PATH\r\n    \t    if [ -f  \"$INDEX_FILE_PATH\" ]; then\r\n\t\t\"Error: cannot remove PID index file in localization: $INDEX_FILE_PATH\"\r\n\t    else\r\n\t\t\"Warning: Sphinx was not clearly closed. I removed index PID file: $INDEX_FILE_PATH\"\r\n            fi\r\n\t    # echo \"$INDEX_FILE_PATH does not exists.\"\r\n\tfi\r\n        index+=1\r\n    done\r\n}\r\n\r\ncheck_pid(){\r\n    DAEMON_PID=$(\/sbin\/pidof -s -o %PPID \"$HOME_DIR$NAME\")\r\n    DAEMON_PID=$(($DAEMON_PID+0))\r\n    usleep 500000\r\n}\r\n\r\nstart_process() {\r\n    if [ $DAEMON_PID -gt 1 ]; then\r\n        echo \"Process is running ($DAEMON_PID)\"\r\n    else\r\n\tdel_index_pid\r\n\techo \"Starting...\"\r\n\techo \"$HOME_DIR$NAME\" --config \"$CONFIG\"\r\n\t\"$HOME_DIR$NAME\" --config \"$CONFIG\"\r\n\trebuild;\r\n        RETVAL=$?\r\n    fi\r\n}\r\n\r\nkill_process() {\r\n    if [ $DAEMON_PID -lt 1 ]; then\r\n\techo \"Process is not running $DAEMON_PID\"\r\n    else\r\n        if [ $iterator -gt 1 ]; then\r\n\t    echo \"Killing $NAME...\"\r\n\tfi\r\n\tkillall $NAME\r\n\tsleep 1\r\n\tcheck_pid\r\n\tif [ ! $DAEMON_PID  ]; then\r\n\t    echo \"Process killed $DAEMON_PID\"\r\n\telse\r\n\t    if [ $iterator -gt $MAX_ATTEMPT ]; then\r\n\t\techo \"I cannot kill process. I give up after $MAX_ATTEMPT attempts. Sorry, I fucked it.\"\r\n\t    else\r\n\t        iterator=$(( $iterator + 1 ))\r\n\t\tkill_process\r\n\t    fi\r\n\tfi\r\n    fi\r\n}\r\n\r\nfunction rotate {\r\n    if [ $DAEMON_PID -lt 1 ]; then\r\n\techo \"Process is not running $DAEMON_PID\"\r\n    else\r\n\techo \"Process is to be rotated\"\r\n\tkill -HUP $DAEMON_PID;\r\n    fi\r\n}\r\n\r\ncheck_status() {\r\n    if [ $DAEMON_PID -gt 1 ]; then\r\n        echo \"1\"\r\n    else\r\n\tusleep 500000\r\n\tcheck_pid\r\n\tcheck_status2\r\n    fi\r\n}\r\n\r\ncheck_status2() {\r\n    if [ $DAEMON_PID -gt 1 ]; then\r\n        echo \"1\"\r\n    else\r\n\techo \"-1\"\r\n    fi\r\n}\r\n\r\ncheck_pid\r\n\r\ncase \"$1\" in\r\n    start)\r\n\tstart_process\r\n\t;;\r\n    stop)\r\n\tkill_process\r\n\t;;\r\n    restart)\r\n\tkill_process\r\n\tstart_process\r\n    \t;;\r\n    status)\r\n\tcheck_status\r\n\t;;\r\n    rotate)\r\n\trotate\r\n\t;;\r\n    *)\r\n    echo \"Usage: $0 (start|stop|restart|status|rotate)\"\r\n    ;;\r\nesac<\/pre>\n
Ok, now the main script self-descripting “sphinx-slave.sh”<\/p>\n
#!\/bin\/bash\r\n\r\nMASTER_DIR=\"\/storage_master\/arch\/\";\r\nLOCAL_DIR=\"\/storage\/arch\/\";\r\nLOG_ERROR_DIR=\"\/var\/log\/sphinx\/slave\/\";\r\nLOG_FILE=\"\/var\/log\/sphinx\/slave.log\";\r\nLOCK_FILE=\"\/var\/log\/sphinx\/slave.lock\";\r\nUNIQUE_NUMBER=\"\";\r\n# basic file to watch - it is created as last.\r\nFILE_TIMESTAMP=\"sph\"\r\n# Lock file of Sphinx process. File is present while process is working.\r\nFILE_LOCK_SPH=\"spl\"\r\nMAKE_ROTATE=1;\r\nMAKE_RESTART=0;\r\n\r\nSPH_INDICES=('daily' 'weekly' 'archive');\r\n# Basic index which requires to stop searchd(aemon).\r\nSPH_MAIN_INDEX=\"archive\";\r\nSPH_INDEX_ERR=();\r\nSPH_INDEX_NEW=();\r\nSPH_INDICES_QTY=${#SPH_INDICES[@]};\r\nSPH_INDEX_NEW_ITEM_ITERATOR=0;\r\n\r\nFILES_TO_DETECT=('spd' 'sph' 'spi' 'spp');\r\nFILES_TO_DETECT_QTY=${#FILES_TO_DETECT[@]};\r\nFILES_TO_RENAME=('spl' 'spa' 'spd' 'sph' 'spi' 'spk' 'spm' 'spp');\r\nFILES_TO_RENAME_QTY=${#FILES_TO_RENAME[@]};\r\n\r\n# To be set\r\nHOMEDIR=\"\";\r\n\r\n# Block functions\r\n\r\nfunction set_homedir {\r\n    dirExec=`pwd`;\r\n    fileExec=`basename $1`;\r\n    fullPath=\"$dirExec\/$fileExec\";\r\n    if [ -f $fullPath ]; then\r\n        HOMEDIR=$dirExec;\r\n    else\r\n        HOMEDIR=`dirname $0`;\r\n    fi\r\n}\r\n\r\nfunction check_all {\r\n\r\n    if [  -f $LOCK_FILE ]; then\r\n\twrite_to_log \"Lock file exists ($LOCK_FILE)\";\r\n\treturn 0;\r\n\r\n    fi\r\n\r\n    return 1;\r\n}\r\n\r\nfunction compare_int {\r\n    if [ $1 -gt $2 ]; then\r\n\treturn 1;\r\n    else\r\n\treturn 0;\r\n    fi\r\n}\r\n\r\nfunction write_to_log {\r\n    INFO=$1\r\n    TIME=$(date '+%Y.%m.%d|%H:%M');\r\n    echo \"[$UNIQUE_NUMBER] $TIME $INFO\" >> $LOG_FILE\r\n}\r\n\r\n# End block functions\r\n\r\n# Detect homedir\r\nset_homedir $0\r\nUNIQUE_NUMBER=$(date '+%s');\r\n\r\ncheck_all\r\nMAY_CONTINUE=$?;\r\nif [ \"$MAY_CONTINUE\" -eq 0 ]; then\r\n    exit 0;\r\nfi\r\ntouch $LOCK_FILE;\r\n\r\nfor ((INDICES_ITERATOR=0;INDICES_ITERATOR<$SPH_INDICES_QTY;INDICES_ITERATOR++)); do     SPH_INDEX=${SPH_INDICES[${INDICES_ITERATOR}]};     if [ -f $MASTER_DIR$SPH_INDEX.$FILE_TIMESTAMP ] && [ -f $MASTER_DIR$SPH_INDEX.$FILE_LOCK_SPH ]; then \t# Musza istniec obydwa pliki \tMASTER_STAMP=`\/usr\/bin\/stat  -c '%Z' $MASTER_DIR$SPH_INDEX.$FILE_TIMESTAMP`;         MASTER_STAMP=$((MASTER_STAMP + 0));     else \t# No sph|spl files. spl: master does not work, sph: index does not exists \tMASTER_STAMP=0;     fi     if [ -f $LOCAL_DIR$SPH_INDEX.$FILE_TIMESTAMP ]; then \tLOCAL_STAMP=`\/usr\/bin\/stat  -c '%Z' $LOCAL_DIR$SPH_INDEX.$FILE_TIMESTAMP`; \tLOCAL_STAMP=$((LOCAL_STAMP + 0));     else \tLOCAL_STAMP=0;     fi          if [ \"$MASTER_STAMP\" -eq 0 ]; then \twrite_to_log \"Brak indeksu master: $SPH_INDEX\"     else \tINDEX_ERROR_FILE=\"$LOG_ERROR_DIR$SPH_INDEX.err\"; \t# echo \"$MASTER_STAMP for $SPH_INDEX\"; \tcompare_int $MASTER_STAMP $LOCAL_STAMP \tis_new=$? \tif [ \"$is_new\" -eq \"1\" ]; then \t    if [ -f $INDEX_ERROR_FILE ]; then \t\trm $INDEX_ERROR_FILE; \t    fi \t    # write logs \t    write_to_log \"Index '$SPH_INDEX' is to be renewing\"; \t    write_to_log \"Stat for index '$SPH_INDEX' MASTER|SLAVE: $MASTER_STAMP|$LOCAL_STAMP\"; \t     \t    # Sprawdz, czy na pewno nie ma plikow tymczasowych (oznaczajacych, ze indeks jest w trakcie budowy) \t    TMP_FILE_QTY=$(ls -l $MASTER_DIR$SPH_INDEX.*.tmp* 2>\/dev\/null | wc -l);\r\n\t    TMP_FILE_QTY=$(( TMP_FILE_QTY + 0));\r\n\t    if [ \"$TMP_FILE_QTY\" -gt \"0\" ]; then\r\n\t\twrite_to_log \"Temporary files are available [$TMP_FILE_QTY]. Omitting index renewal - '$SPH_INDEX'.\";\r\n\t\tcontinue;\r\n\t    fi\r\n\r\n\t    if [ \"$SPH_MAIN_INDEX\" = \"$SPH_INDEX\" ]; then\r\n\t\twrite_to_log \"Stopping service\"\r\n\t\twrite_to_log \"Copying file with stopwords\"\r\n\t\tcp $MASTER_DIR..\/stopwords-pl $LOCAL_DIR..\/\r\n\t\t\/etc\/init.d\/sphinx stop\r\n\t\twrite_to_log \"Executing command 'sync'\";\r\n\t\tsync;\r\n\t\twrite_to_log \"Writing command to clear cache\";\r\n\t\techo 3 > \/proc\/sys\/vm\/drop_caches;\r\n\t\tsleep 10;\r\n\r\n\t    fi\r\n    \t    # copy files as .new\r\n\t    for ((TO_COPY_ITERATOR=0;TO_COPY_ITERATOR<$FILES_TO_RENAME_QTY;TO_COPY_ITERATOR++)); do \t\tSPH_INDEX_COPY_SRC=$MASTER_DIR$SPH_INDEX\".\"; \t\tSPH_INDEX_COPY_DST=$LOCAL_DIR$SPH_INDEX\".\"; \t\tFILE_TO_COPY=${FILES_TO_RENAME[${TO_COPY_ITERATOR}]}; \t\t# echo $SPH_INDEX_COPY$FILE_TO_COPY; \t\tif [ \"$SPH_MAIN_INDEX\" = \"$SPH_INDEX\" ]; then \t\t    rm_result=`rm -f $SPH_INDEX_COPY_DST$FILE_TO_COPY  2>&1`;\r\n\t\t    res_int=$(( $? + 0 ));\r\n\t\t    rm_result=`echo -n $rm_result | sed 's\/\\s\\n\/\/g'`;\r\n\t\t    rm_no_file=`echo $rm_result | grep -i \"No such file\"`;\r\n\t\t    # the length of STRING is zero\r\n\r\n\t\t    if [ $res_int -gt \"0\" ] && [ -z \"$rm_no_file\" ]; then\r\n\t\t\techo \"Error: -z rm_nofile: $rm_no_file vs. result:  $rm_result vs. $SPH_INDEX_COPY_DST$FILE_TO_COPY\"\r\n\t\t\tSPH_INDEX_ERR[$INDICES_ITERATOR]=1;\r\n\t\t\twrite_to_log \"$SPH_INDEX $rm_result\";\r\n\t\t    fi\r\n\t\t    cp_result=`cp $SPH_INDEX_COPY_SRC$FILE_TO_COPY $SPH_INDEX_COPY_DST$FILE_TO_COPY 2>&1 `;\r\n\t\telse\r\n\t\t    # add extenstion \".new\"\r\n    \t\t    cp_result=`cp $SPH_INDEX_COPY_SRC$FILE_TO_COPY $SPH_INDEX_COPY_DST\"new.\"$FILE_TO_COPY 2>&1 `;\r\n\t\tfi\r\n\r\n\t\tcp_result=`echo -n $cp_result | sed 's\/\\s\\n\/\/g'`;\r\n\t\t# echo $cp_result;\r\n\t\tif [ -n \"$cp_result\" ]; then\r\n\t\t    SPH_INDEX_ERR[$INDICES_ITERATOR]=1;\r\n\t\t    MAKE_ROTATE=0;\r\n\t\t    write_to_log \"$SPH_INDEX $FILE_TO_COPY error: $cp_result.\"\r\n\t\telse\r\n\t\t    write_to_log \"$SPH_INDEX $FILE_TO_COPY successfuly copied.\"\r\n\t\tfi\r\n\t    done\r\n\r\n\t    IS_ERROR=${SPH_INDEX_ERR[${INDICES_ITERATOR}]};\r\n\t    if [ -z $IS_ERROR ]; then\r\n\t\tIS_ERROR=0;\r\n\t\t# set index to renew\r\n\t\tSPH_INDEX_NEW[$SPH_INDEX_NEW_ITEM_ITERATOR]=$SPH_INDEX;\r\n\t\t((SPH_INDEX_NEW_ITEM_ITERATOR++));\r\n\t    fi\r\n\t    if [ \"$IS_ERROR\" -eq \"1\" ] ; then\r\n\t\ttouch $INDEX_ERROR_FILE;\r\n\t    fi\r\n\tfi\r\n    fi\r\ndone \r\n\r\nSPH_INDEX_NEW_QTY=$(( ${#SPH_INDEX_NEW[@]} + 0 ));\r\nSPH_INDEX_ERR_QTY=$(( ${#SPH_INDEX_ERR[@]} + 0 ));\r\n\r\nif [ \"$SPH_INDEX_NEW_QTY\" -gt \"0\" ] && [ \"$SPH_INDEX_ERR_QTY\" -lt \"1\" ]; then\r\n\r\n    write_to_log \"Qty of new indices is larger then 0 and error indices is less then 1 [OK].\";\r\n\r\n    for ((INDICES_ITERATOR2=0;INDICES_ITERATOR2<$SPH_INDEX_NEW_QTY;INDICES_ITERATOR2++)); do\r\n\tSPH_INDEX_NAME=${SPH_INDEX_NEW[${INDICES_ITERATOR2}]};\r\n\tif [ \"$SPH_INDEX_NAME\" = $SPH_MAIN_INDEX ]; then\r\n\t    MAKE_RESTART=1;\r\n\telse\r\n\t    continue;\r\n\t    # for IDX_FILE_TMP in \"$LOCAL_DIR$SPH_INDEX_NAME\"*\".new\"; do\r\n\t    #\tIDX_FILE=`echo $IDX_FILE_TMP | sed 's\/\\.tmp$\/\/g'`;\r\n\t    #\tmv $IDX_FILE_TMP $IDX_FILE;\r\n\t    # done\r\n\tfi\r\n    done\r\n\r\n    if [ \"$MAKE_RESTART\" -gt \"0\" ]; then\r\n\twrite_to_log \"Starting service\"\r\n\t\/etc\/init.d\/sphinx start\r\n    else\r\n\tif [ \"$MAKE_ROTATE\" -gt \"0\" ]; then\r\n\t    write_to_log \"Rotating indexes\"\r\n\t    \/etc\/init.d\/sphinx rotate\r\n\telse\r\n\t    write_to_log \"Not allowed to rotate indices\";\r\n\tfi\r\n    fi\r\n\r\n# else\r\n#    write_to_log \"Qty of new indices [$SPH_INDEX_NEW_QTY] is less then 0 OR error indices [$SPH_INDEX_ERR_QTY] is greater then 0.\";\r\n\r\nfi                    \r\n\r\nrm $LOCK_FILE;\r\n\r\nexit 0<\/pre>\n","protected":false},"excerpt":{"rendered":"
One instance and one storage with indices is neither secure nor effective.\u00a0 That is why I\u00a0created\u00a0my\u00a0replication\u00a0for Sphinx Search\u00a0. To put it shortly, my Sphinx’s architecture constists of tree indices and MVA (multivalue attributes stored in mva_updates_pool) – that is why I cannot use RT index. My first idea was to create independent Sphix index + […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[11,12],"tags":[10,13],"_links":{"self":[{"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=\/wp\/v2\/posts\/191"}],"collection":[{"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=191"}],"version-history":[{"count":19,"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=\/wp\/v2\/posts\/191\/revisions"}],"predecessor-version":[{"id":692,"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=\/wp\/v2\/posts\/191\/revisions\/692"}],"wp:attachment":[{"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=191"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=191"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/10sa.com\/sql_stories\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=191"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}