This is an old revision of the document!
Table of Contents
Marfeel Test
App server
Why do you think different cache times for the nginx cache and for the browser were defined?
General answer: Caching is mandatory for performance.
- Particular answer to proxy cache:
- I've configured nginx proxycache on
/dev/shm
for performance (on ram cache). 1 minute for proxycache should be fine if the content is changing constantly, cause nginx will “rebuild/compile” the cache after 1 minute but inside that minute, it will use the already compiled version of the file.
- Particular answer to static content:
- Static content to 1 hour maybe is aggressive, normally I set it up to 1day or more, also depends on the type of application.
- Particular answer to proxied content:
- I Think we must include a header like “If-Modified-Since” to invalidate local cache if the file has been changed on the server.
Auto-scaling
Ami and UserData
I did some mistakes, so I create some versions of it:
- name :
dodgermarfeeltest_003
- id :
ami-0bc1be25784321fc1
nginx
I've configured nginx with a bare minimal setup:
- /etc/nginx/nginx.conf
user www-data; worker_processes auto; pid /run/nginx.pid; events { worker_connections 768; } http { server_tokens off; sendfile on; tcp_nopush on; tcp_nodelay on; ## Start: Timeouts ## client_body_timeout 10; client_header_timeout 10; keepalive_timeout 5 5; send_timeout 10; ## End: Timeouts ## types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE ssl_prefer_server_ciphers on; access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; # compress everything gzip on; # disabled by marfeel request: # Enable gzip for all the requests (proxied request included) # you have chosen it :-) #gzip_disable "msie6"; #include /etc/nginx/conf.d/*.conf; include /etc/nginx/conf.d/proxy_cache.conf; include /etc/nginx/sites-enabled/*; }
Python http server
I've setup a .service
for systemd named marfeeltest
which is inside the AMI:
<file yaml /etc/systemd/system/marfeel_test.service>
[Unit]
Description=marfeel test service
After=auditd.service systemd-user-sessions.service time-sync.target
[Service]
User=marfeel
TimeoutStartSec=0
Type=simple
KillMode=control-group
WorkingDirectory=/opt/test/Marfeel-appserverpythontestapp-2937d4f8673c
ExecStart=/usr/bin/python3 -m http.server –cgi 8080
Restart=no
[Install]
WantedBy=multi-user.target
</file>
===== Which code have you added to the user-data on launching the instance? =====
<code yaml>
#cloud-config
runcmd:
- [ mkdir, -p, /opt/test ]
- [ wget, -O/opt/test/master.tar.gz, “https://bitbucket.org/Marfeel/appserverpythontestapp/get/master.tar.gz” ]
- [ tar, xzfv, /opt/test/master.tar.gz, -C, /opt/test/ ]
- [ systemctl, start, marfeel_test ]
- [ systemctl, restart, nginx ]
</code>
===== Security group =====
I create the 001 group with https, then I remove it in the 2nd version:
* name:
dodgerlaunch_002
auto-scaling Group
* name: ''marfeel_scalinggroup_001''
Load balancer
Dynamic set up
Done as required with a very simple bash script.
Dependencies for running the script:
aws-cli
jq
aws-cli
must be configured!!!
This script should be run by root.
Code:
- make_it_cool.sh
#!/bin/bash # Exit codes: # 1 : # 2 : # 3 : # 4 : ######################################################################## # INIT ######################################################################## CONFIGFILE="$(dirname $(readlink -f $0))/$(basename $(readlink -f $0) .sh).config" ######################################################################## #/INIT ######################################################################## ######################################################################## # # CONSTANTS # ######################################################################## # colors BOLD="\e[1m" GREEN="\e[32m" LIGHTGREEN="${BOLD}${GREEN}" RED="\033[1;31m" LIGHTRED="\033[1;31m" BLUE="\e[34m" LIGHTBLUE="${BOLD}${BLUE}" YELLOW="\e[33m" LIGHTYELLOW="${BOLD}${YELLOW}" WHITE="\033[0;37m" RESET="\033[0;00m" NOW="$(date +%Y%m%d%H%M%S)" ######################################################################## # # / CONSTANTS # ######################################################################## ######################################################################## # # VARIABLES # ######################################################################## SCRIPTLOG="$(dirname `readlink -f $0`)/logs/$(basename $0 .sh)_script_${NOW}.log" SCRIPTLOGERR="$(dirname `readlink -f $0`)/logs/$(basename $0 .sh)_script_${NOW}.err" TMPFILE=$(mktemp) HADYNAMICCFG=$(mktemp) ######################################################################## # # / VARIABLES # ######################################################################## ######################################################################## [49/5875] # # FUNCTIONS # ######################################################################## usage() { printf "%s${LIGHTRED}USAGE:${RESET} $0 Read documentation here: https://wiki.ciberterminal.net/doku.php?id=marfeel:test#dynamic_set_up" # VERY INITIAL CHECKS } printmsg() { echo -e "$*" } output_log() { if [[ "${QUIETOUTPUT}" == true ]]; then printmsg "$*" >> ${OUTPUTFILE} else printmsg "$*" | tee -a "${OUTPUTFILE}" fi } abort_message() { printmsg "ERROR: $*" exit 1 } # debug_me uses variable ${DEBUGME} debug_me() { if [[ "${DEBUGME}" && ${DEBUGME} -eq 0 ]] ; then echo -e "${LIGHTBLUE}DEBUG: ${RESET}$*" fi } ######################################################################## # # / FUNCTIONS # ######################################################################## ######################################################################## # # MAIN # ######################################################################## [[ ! -d $(dirname ${SCRIPTLOG}) ]] && mkdir -p $(dirname ${SCRIPTLOG}) # DETECTING if the script is run by cron if [[ "$(tty)" = "not a tty" ]] ; then set -x exec > ${SCRIPTLOG} exec 2> ${SCRIPTLOGERR} fi if [[ ${DEBUG} -eq 0 ]] ; then echo -e "${BLUE}DEBUGMODE${RESET} is on" echo -e "\t SCRIPTLOG will be ${SCRIPTLOG}" echo -e "\t SCRIPTLOGERR will be ${SCRIPTLOGERR}" set -x exec 2> ${SCRIPTLOGERR} fi [[ ! -f ${CONFIGFILE} ]] && echo -e "${LIGHTRED} CONFIGFILE ${CONFIGFILE} NOT FOUND${RESET}" && exit 1 . ${CONFIGFILE} [[ ${DEBUG} -eq 0 ]] && DEBUGME="bash -x" ${AWSCLI} ec2 describe-instances --filters Name=instance-type,Values=t2.nano Name=instance-state-name,Values=running > ${TMPFILE} let x=0 for PRIVATEIP in $(cat ${TMPFILE} | ${JQ} '.Reservations[] | .Instances[] | "\(.PrivateIpAddress) \(.PublicIpAddress) \(.Tags)"' | egrep "${SCALINGTAG}" | awk 'BEGIN{FS="[ \"]"}{print $2}') ; do echo -e "\tserver marfeel_nginx_${x} ${PRIVATEIP}:80 check maxconn 512" >> ${HADYNAMICCFG} let x++ done if [[ $(cat ${HADYNAMICCFG} | wc -l) -ge ${MINBACKENDS} ]] ; then # ok, overwriting config echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" > ${HAPROXYCONFIG} cat ${HAPROXYTEMPLATE} >> ${HAPROXYCONFIG} cat ${HADYNAMICCFG} >> ${HAPROXYCONFIG} echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" >> ${HAPROXYCONFIG} systemctl reload haproxy fi rm -f ${TMPFILE} rm -f ${HADYNAMICCFG} exit ${EXITCODE} ######################################################################## # # / MAIN # ########################################################################
Config file:
- make_it_cool.config
# Our template for re-generate the config HAPROXYTEMPLATE=/etc/haproxy/haproxy.cfg.TMPL # The configfile itself HAPROXYCONFIG=/etc/haproxy/haproxy.cfg # Minimum amount of backends that must be running, less than this, the config file won't be changed MINBACKENDS=2 # Tag for the scaling group (backend servers will be filtered by this tag) SCALINGTAG="marfeel_scalinggroup_001" # generic setup AWSCLI=/usr/bin/aws JQ=/usr/bin/jq # SET to 0 for DEBUG DEBUG=0
Setup crontab with desired frequency for refresh, for example:
*/3 * * * * root /root/marfeel_auto_updater/make_it_cool.sh