User Tools

Site Tools


marfeel:test

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
marfeel:test [2020/03/17 07:58] – [Things I forgot] dodgermarfeel:test [2020/10/23 09:50] (current) – removed dodger
Line 1: Line 1:
-====== Marfeel Test ====== 
- 
-====== App server ====== 
- 
-===== Why do you think different cache times for the nginx cache and for the browser were defined? ===== 
-General answer: Caching is mandatory for performance.  
- 
-  * Particular answer to proxy cache: 
-    * I've configured nginx proxy_cache on ''/dev/shm'' for performance (on ram cache). 1 minute for proxy_cache should be fine if the content is changing constantly, cause nginx will "rebuild/compile" the cache after 1 minute but inside that minute, it will use the already compiled version of the file. 
-  * Particular answer to static content: 
-    * Static content to 1 hour maybe is aggressive, normally I set it up to 1day or more, also depends on the type of application. 
-  * Particular answer to proxied content: 
-    * I Think we must include a header like "If-Modified-Since" to invalidate local cache if the file has been changed on the server. 
- 
-====== Auto-scaling ====== 
-===== Ami and UserData ===== 
-I did some mistakes, so I create some versions of it: 
- 
-| **name** | ''dodger_marfeel_test_003'' 
-| **id** | ''ami-0bc1be25784321fc1'' | 
- 
-\\ 
-==== nginx ==== 
-I've configured nginx with a bare minimal setup: 
-<file config /etc/nginx/nginx.conf> 
-user www-data; 
-worker_processes auto; 
-pid /run/nginx.pid; 
-events { 
-        worker_connections 768; 
-} 
-http { 
-        server_tokens off; 
-        sendfile on; 
-        tcp_nopush on; 
-        tcp_nodelay on; 
- 
-        ## Start: Timeouts ## 
-        client_body_timeout   10; 
-        client_header_timeout 10; 
-        keepalive_timeout     5 5; 
-        send_timeout          10; 
-        ## End: Timeouts ## 
- 
-        types_hash_max_size 2048; 
-        include /etc/nginx/mime.types; 
-        default_type application/octet-stream; 
-        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE 
-        ssl_prefer_server_ciphers on; 
-        access_log /var/log/nginx/access.log; 
-        error_log /var/log/nginx/error.log; 
- 
-        # compress everything 
-        gzip on; 
-        # disabled by marfeel request: 
-        #       Enable gzip for all the requests (proxied request included) 
-        #       you have chosen it :-) 
-        #gzip_disable "msie6"; 
- 
-        #include /etc/nginx/conf.d/*.conf; 
-        include /etc/nginx/conf.d/proxy_cache.conf; 
-        include /etc/nginx/sites-enabled/*; 
-} 
-</file> 
- 
-Nginx proxy cache setup: 
-<file config  /etc/nginx/conf.d/proxy_cache.conf> 
-proxy_cache_path /dev/shm levels=1:2 keys_zone=marfeel:1m; 
-proxy_cache                     marfeel; 
-#proxy_cache_background_update  on; 
-proxy_cache_key $scheme$proxy_host$request_uri; 
-proxy_cache_revalidate          on; 
-</file> 
- 
-Default site: 
-<file config /etc/nginx/sites-available/default.conf> 
-server { 
-        listen 80 default_server; 
-        listen [::]:80 default_server; 
-        #root /opt/test/www; 
-        root /opt/test/Marfeel-appserverpythontestapp-2937d4f8673c; 
-        index index.html index.htm ; 
-        server_name _; 
- 
-        # compress all proxy requests 
-        include conf.d/proxy_compression.conf; 
-        include conf.d/static_files.conf; 
- 
-        location ^~ /cgi-bin { 
-                proxy_pass http://localhost:8080; 
-                proxy_set_header Host $http_host; 
-                proxy_cache_valid any      1m; 
-                expires 10m; 
-        } 
-} 
-</file> 
- 
-Proxy compression config: 
-<file config /etc/nginx/conf.d/proxy_compression.conf> 
-# Enable gzip but do not remove ETag headers 
-gzip on; 
-gzip_vary on; 
-gzip_comp_level 4; 
-gzip_min_length 256; 
-gzip_proxied any; 
-#gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy; 
-gzip_types *; 
-</file> 
- 
-Static files ''expires'': 
-<file config /etc/nginx/conf.d/static_files.conf> 
-# serve static files directly 
-# The ?: prefix is a 'non-capturing' mark, meaning we do not require 
-# the pattern to be captured into $1 which should help improve performance 
-location ~* ^.+\.(?:jpg|jpeg|gif|css|png|js|ico|txt)$ { 
-    #access_log        off; 
-    expires           1h; 
-} 
-</file> 
- 
- 
-==== Python http server ==== 
-I've setup a ''.service'' for systemd named ''marfeel_test'' which is inside the AMI: 
-<file yaml /etc/systemd/system/marfeel_test.service> 
-[Unit] 
-Description=marfeel test service 
-After=auditd.service systemd-user-sessions.service time-sync.target 
-  
-[Service] 
-User=marfeel 
-TimeoutStartSec=0 
-Type=simple 
-KillMode=control-group 
-WorkingDirectory=/opt/test/Marfeel-appserverpythontestapp-2937d4f8673c 
-ExecStart=/usr/bin/python3 -m http.server --cgi 8080 
-Restart=no 
-  
-[Install] 
-WantedBy=multi-user.target 
-</file> 
-===== Which code have you added to the user-data on launching the instance? ===== 
-<code yaml> 
-#cloud-config 
- 
-runcmd: 
-  - [ mkdir, -p, /opt/test ] 
-  - [ wget, -O/opt/test/master.tar.gz, "https://bitbucket.org/Marfeel/appserverpythontestapp/get/master.tar.gz" ] 
-  - [ tar, xzfv, /opt/test/master.tar.gz, -C, /opt/test/ ] 
-  - [ systemctl, start, marfeel_test ] 
-  - [ systemctl, restart, nginx ] 
-</code> 
- 
-===== Security group ===== 
-I create a security group (ending with 001) with https, then I remove it in the 2nd version: 
-| **name** | ''dodger_launch_002'' | 
- 
-===== auto-scaling Group ===== 
- 
-| **name** | ''marfeel_scalinggroup_001'' | 
- 
- 
-====== Load balancer ====== 
- 
-===== Dynamic set up ===== 
- 
-Done as required with a very simple bash script.\\ 
-Dependencies for running the script: 
-  * ''aws-cli'' 
-  * ''jq'' 
- 
- 
-<WRAP center round info 60%> 
-''aws-cli'' must be configured!!! 
-</WRAP> 
- 
-\\ 
-This script should be run by root. 
- 
-Code: 
-<file bash /root/marfeel_auto_updater/make_it_cool.sh> 
-#!/bin/bash 
- 
-# Exit codes: 
-#  1 : 
-#  2 : 
-#  3 : 
-#  4 : 
- 
- 
-######################################################################## 
-# INIT 
-######################################################################## 
-CONFIGFILE="$(dirname $(readlink -f $0))/$(basename $(readlink -f $0) .sh).config" 
- 
-######################################################################## 
-#/INIT 
-######################################################################## 
- 
-######################################################################## 
-# 
-# CONSTANTS 
-# 
-######################################################################## 
- 
-# colors 
-BOLD="\e[1m" 
-GREEN="\e[32m" 
-LIGHTGREEN="${BOLD}${GREEN}" 
-RED="\033[1;31m" 
-LIGHTRED="\033[1;31m" 
-BLUE="\e[34m" 
-LIGHTBLUE="${BOLD}${BLUE}" 
-YELLOW="\e[33m" 
-LIGHTYELLOW="${BOLD}${YELLOW}" 
-WHITE="\033[0;37m" 
-RESET="\033[0;00m" 
- 
-NOW="$(date +%Y%m%d%H%M%S)" 
- 
- 
-######################################################################## 
-# 
-# / CONSTANTS 
-# 
-######################################################################## 
- 
- 
- 
- 
-######################################################################## 
-# 
-# VARIABLES 
-# 
-######################################################################## 
- 
- 
-SCRIPTLOG="$(dirname `readlink -f $0`)/logs/$(basename $0 .sh)_script_${NOW}.log" 
-SCRIPTLOGERR="$(dirname `readlink -f $0`)/logs/$(basename $0 .sh)_script_${NOW}.err" 
- 
-TMPFILE=$(mktemp) 
-HADYNAMICCFG=$(mktemp) 
- 
-######################################################################## 
-# 
-# / VARIABLES 
-# 
-######################################################################## 
- 
- 
-########################################################################                                                                                                                                [49/5875] 
-#                                                                                                                                                                                                                 
-# FUNCTIONS 
-# 
-######################################################################## 
- 
- 
-usage() 
-{ 
-    printf "%s${LIGHTRED}USAGE:${RESET} 
-    $0 
-      
- Read documentation here: 
- https://wiki.ciberterminal.net/doku.php?id=marfeel:test#dynamic_set_up" 
-    # VERY INITIAL CHECKS 
-} 
- 
-printmsg() 
-{ 
-   echo -e "$*" 
-} 
- 
-output_log() 
-{ 
-    if [[ "${QUIETOUTPUT}" == true ]]; then 
-        printmsg "$*" >> ${OUTPUTFILE} 
-    else 
-        printmsg "$*" | tee -a "${OUTPUTFILE}" 
-    fi 
-} 
- 
-abort_message() 
-{ 
-    printmsg "ERROR: $*" 
-    exit 1 
-} 
- 
-# debug_me uses variable ${DEBUGME} 
-debug_me() 
-{ 
-    if [[ "${DEBUGME}" && ${DEBUGME} -eq 0 ]] ; then  
-        echo -e "${LIGHTBLUE}DEBUG: ${RESET}$*" 
-    fi 
-} 
- 
- 
-######################################################################## 
-# 
-# / FUNCTIONS 
-# 
-######################################################################## 
- 
-######################################################################## 
-# 
-# MAIN 
-# 
-######################################################################## 
- 
-[[ ! -d $(dirname ${SCRIPTLOG}) ]] && mkdir -p $(dirname ${SCRIPTLOG}) 
- 
-# DETECTING if the script is run by cron 
-if [[ "$(tty)" = "not a tty" ]] ; then 
- set -x 
- exec > ${SCRIPTLOG} 
- exec 2> ${SCRIPTLOGERR} 
-fi 
- 
-if [[ ${DEBUG} -eq 0 ]] ; then 
-    echo -e "${BLUE}DEBUGMODE${RESET} is on" 
-    echo -e "\t SCRIPTLOG will be ${SCRIPTLOG}" 
-    echo -e "\t SCRIPTLOGERR will be ${SCRIPTLOGERR}" 
-    set -x 
-    exec 2> ${SCRIPTLOGERR} 
-fi 
- 
- 
-[[ ! -f ${CONFIGFILE} ]] && echo -e "${LIGHTRED} CONFIGFILE ${CONFIGFILE} NOT FOUND${RESET}" && exit 1 
-. ${CONFIGFILE} 
- 
-[[ ${DEBUG} -eq 0 ]] && DEBUGME="bash -x" 
- 
-${AWSCLI} ec2 describe-instances --filters Name=instance-type,Values=t2.nano Name=instance-state-name,Values=running > ${TMPFILE} 
-let x=0 
-for PRIVATEIP in $(cat ${TMPFILE} | ${JQ} '.Reservations[] | .Instances[] | "\(.PrivateIpAddress) \(.PublicIpAddress) \(.Tags)"' | egrep "${SCALINGTAG}" | awk 'BEGIN{FS="[ \"]"}{print $2}') ; do 
- echo -e "\tserver marfeel_nginx_${x} ${PRIVATEIP}:80 check maxconn 512" >> ${HADYNAMICCFG} 
- let x++ 
-done 
- 
-if [[ $(cat ${HADYNAMICCFG} | wc -l) -ge ${MINBACKENDS} ]] ; then 
- # ok, overwriting config 
- echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" > ${HAPROXYCONFIG} 
- cat ${HAPROXYTEMPLATE} >> ${HAPROXYCONFIG} 
- cat ${HADYNAMICCFG} >> ${HAPROXYCONFIG} 
- echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" >> ${HAPROXYCONFIG} 
- systemctl reload haproxy 
-fi 
- 
-rm -f ${TMPFILE} 
-rm -f ${HADYNAMICCFG} 
- 
-exit ${EXITCODE} 
- 
-######################################################################## 
-# 
-# / MAIN 
-# 
-######################################################################## 
- 
-</file> 
-\\ 
-Config file: 
-<file config /root/marfeel_auto_updater/make_it_cool.config> 
- 
-# Our template for re-generate the config 
-HAPROXYTEMPLATE=/etc/haproxy/haproxy.cfg.TMPL 
-# The configfile itself 
-HAPROXYCONFIG=/etc/haproxy/haproxy.cfg 
-# Minimum amount of backends that must be running, less than this, the config file won't be changed 
-MINBACKENDS=2 
- 
-# Tag for the scaling group (backend servers will be filtered by this tag) 
-SCALINGTAG="marfeel_scalinggroup_001" 
- 
-# generic setup 
-AWSCLI=/usr/bin/aws 
-JQ=/usr/bin/jq 
- 
-# SET to 0 for DEBUG 
-DEBUG=0 
-</file> 
- 
-\\ 
-Setup crontab with desired frequency for refresh, for example: 
-<code crontab> 
-*/3 *   * * *   root    /root/marfeel_auto_updater/make_it_cool.sh 
-</code> 
- 
- 
-====== Things I forgot ====== 
- 
-  * Purge log files from ''make_it_cool.sh'' in HA proxy node 
-  * A lot of additional checks for the script... 
  
marfeel/test.1584431890.txt.gz · Last modified: 2020/03/17 07:58 by dodger