marfeel:test
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revision | |||
marfeel:test [2020/03/17 07:58] – [Things I forgot] dodger | marfeel:test [2020/10/23 09:50] (current) – removed dodger | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== Marfeel Test ====== | ||
- | |||
- | ====== App server ====== | ||
- | |||
- | ===== Why do you think different cache times for the nginx cache and for the browser were defined? ===== | ||
- | General answer: Caching is mandatory for performance. | ||
- | |||
- | * Particular answer to proxy cache: | ||
- | * I've configured nginx proxy_cache on ''/ | ||
- | * Particular answer to static content: | ||
- | * Static content to 1 hour maybe is aggressive, normally I set it up to 1day or more, also depends on the type of application. | ||
- | * Particular answer to proxied content: | ||
- | * I Think we must include a header like " | ||
- | |||
- | ====== Auto-scaling ====== | ||
- | ===== Ami and UserData ===== | ||
- | I did some mistakes, so I create some versions of it: | ||
- | |||
- | | **name** | '' | ||
- | | **id** | '' | ||
- | |||
- | \\ | ||
- | ==== nginx ==== | ||
- | I've configured nginx with a bare minimal setup: | ||
- | <file config / | ||
- | user www-data; | ||
- | worker_processes auto; | ||
- | pid / | ||
- | events { | ||
- | worker_connections 768; | ||
- | } | ||
- | http { | ||
- | server_tokens off; | ||
- | sendfile on; | ||
- | tcp_nopush on; | ||
- | tcp_nodelay on; | ||
- | |||
- | ## Start: Timeouts ## | ||
- | client_body_timeout | ||
- | client_header_timeout 10; | ||
- | keepalive_timeout | ||
- | send_timeout | ||
- | ## End: Timeouts ## | ||
- | |||
- | types_hash_max_size 2048; | ||
- | include / | ||
- | default_type application/ | ||
- | ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE | ||
- | ssl_prefer_server_ciphers on; | ||
- | access_log / | ||
- | error_log / | ||
- | |||
- | # compress everything | ||
- | gzip on; | ||
- | # disabled by marfeel request: | ||
- | # | ||
- | # you have chosen it :-) | ||
- | # | ||
- | |||
- | #include / | ||
- | include / | ||
- | include / | ||
- | } | ||
- | </ | ||
- | |||
- | Nginx proxy cache setup: | ||
- | <file config | ||
- | proxy_cache_path /dev/shm levels=1:2 keys_zone=marfeel: | ||
- | proxy_cache | ||
- | # | ||
- | proxy_cache_key $scheme$proxy_host$request_uri; | ||
- | proxy_cache_revalidate | ||
- | </ | ||
- | |||
- | Default site: | ||
- | <file config / | ||
- | server { | ||
- | listen 80 default_server; | ||
- | listen [::]:80 default_server; | ||
- | #root / | ||
- | root / | ||
- | index index.html index.htm ; | ||
- | server_name _; | ||
- | |||
- | # compress all proxy requests | ||
- | include conf.d/ | ||
- | include conf.d/ | ||
- | |||
- | location ^~ /cgi-bin { | ||
- | proxy_pass http:// | ||
- | proxy_set_header Host $http_host; | ||
- | proxy_cache_valid any 1m; | ||
- | expires 10m; | ||
- | } | ||
- | } | ||
- | </ | ||
- | |||
- | Proxy compression config: | ||
- | <file config / | ||
- | # Enable gzip but do not remove ETag headers | ||
- | gzip on; | ||
- | gzip_vary on; | ||
- | gzip_comp_level 4; | ||
- | gzip_min_length 256; | ||
- | gzip_proxied any; | ||
- | #gzip_types application/ | ||
- | gzip_types *; | ||
- | </ | ||
- | |||
- | Static files '' | ||
- | <file config / | ||
- | # serve static files directly | ||
- | # The ?: prefix is a ' | ||
- | # the pattern to be captured into $1 which should help improve performance | ||
- | location ~* ^.+\.(?: | ||
- | # | ||
- | expires | ||
- | } | ||
- | </ | ||
- | |||
- | |||
- | ==== Python http server ==== | ||
- | I've setup a '' | ||
- | <file yaml / | ||
- | [Unit] | ||
- | Description=marfeel test service | ||
- | After=auditd.service systemd-user-sessions.service time-sync.target | ||
- | |||
- | [Service] | ||
- | User=marfeel | ||
- | TimeoutStartSec=0 | ||
- | Type=simple | ||
- | KillMode=control-group | ||
- | WorkingDirectory=/ | ||
- | ExecStart=/ | ||
- | Restart=no | ||
- | |||
- | [Install] | ||
- | WantedBy=multi-user.target | ||
- | </ | ||
- | ===== Which code have you added to the user-data on launching the instance? ===== | ||
- | <code yaml> | ||
- | # | ||
- | |||
- | runcmd: | ||
- | - [ mkdir, -p, /opt/test ] | ||
- | - [ wget, -O/ | ||
- | - [ tar, xzfv, / | ||
- | - [ systemctl, start, marfeel_test ] | ||
- | - [ systemctl, restart, nginx ] | ||
- | </ | ||
- | |||
- | ===== Security group ===== | ||
- | I create a security group (ending with 001) with https, then I remove it in the 2nd version: | ||
- | | **name** | '' | ||
- | |||
- | ===== auto-scaling Group ===== | ||
- | |||
- | | **name** | '' | ||
- | |||
- | |||
- | ====== Load balancer ====== | ||
- | |||
- | ===== Dynamic set up ===== | ||
- | |||
- | Done as required with a very simple bash script.\\ | ||
- | Dependencies for running the script: | ||
- | * '' | ||
- | * '' | ||
- | |||
- | |||
- | <WRAP center round info 60%> | ||
- | '' | ||
- | </ | ||
- | |||
- | \\ | ||
- | This script should be run by root. | ||
- | |||
- | Code: | ||
- | <file bash / | ||
- | #!/bin/bash | ||
- | |||
- | # Exit codes: | ||
- | # 1 : | ||
- | # 2 : | ||
- | # 3 : | ||
- | # 4 : | ||
- | |||
- | |||
- | ######################################################################## | ||
- | # INIT | ||
- | ######################################################################## | ||
- | CONFIGFILE=" | ||
- | |||
- | ######################################################################## | ||
- | #/INIT | ||
- | ######################################################################## | ||
- | |||
- | ######################################################################## | ||
- | # | ||
- | # CONSTANTS | ||
- | # | ||
- | ######################################################################## | ||
- | |||
- | # colors | ||
- | BOLD=" | ||
- | GREEN=" | ||
- | LIGHTGREEN=" | ||
- | RED=" | ||
- | LIGHTRED=" | ||
- | BLUE=" | ||
- | LIGHTBLUE=" | ||
- | YELLOW=" | ||
- | LIGHTYELLOW=" | ||
- | WHITE=" | ||
- | RESET=" | ||
- | |||
- | NOW=" | ||
- | |||
- | |||
- | ######################################################################## | ||
- | # | ||
- | # / CONSTANTS | ||
- | # | ||
- | ######################################################################## | ||
- | |||
- | |||
- | |||
- | |||
- | ######################################################################## | ||
- | # | ||
- | # VARIABLES | ||
- | # | ||
- | ######################################################################## | ||
- | |||
- | |||
- | SCRIPTLOG=" | ||
- | SCRIPTLOGERR=" | ||
- | |||
- | TMPFILE=$(mktemp) | ||
- | HADYNAMICCFG=$(mktemp) | ||
- | |||
- | ######################################################################## | ||
- | # | ||
- | # / VARIABLES | ||
- | # | ||
- | ######################################################################## | ||
- | |||
- | |||
- | ######################################################################## | ||
- | # | ||
- | # FUNCTIONS | ||
- | # | ||
- | ######################################################################## | ||
- | |||
- | |||
- | usage() | ||
- | { | ||
- | printf " | ||
- | $0 | ||
- | |||
- | Read documentation here: | ||
- | https:// | ||
- | # VERY INITIAL CHECKS | ||
- | } | ||
- | |||
- | printmsg() | ||
- | { | ||
- | echo -e " | ||
- | } | ||
- | |||
- | output_log() | ||
- | { | ||
- | if [[ " | ||
- | printmsg " | ||
- | else | ||
- | printmsg " | ||
- | fi | ||
- | } | ||
- | |||
- | abort_message() | ||
- | { | ||
- | printmsg " | ||
- | exit 1 | ||
- | } | ||
- | |||
- | # debug_me uses variable ${DEBUGME} | ||
- | debug_me() | ||
- | { | ||
- | if [[ " | ||
- | echo -e " | ||
- | fi | ||
- | } | ||
- | |||
- | |||
- | ######################################################################## | ||
- | # | ||
- | # / FUNCTIONS | ||
- | # | ||
- | ######################################################################## | ||
- | |||
- | ######################################################################## | ||
- | # | ||
- | # MAIN | ||
- | # | ||
- | ######################################################################## | ||
- | |||
- | [[ ! -d $(dirname ${SCRIPTLOG}) ]] && mkdir -p $(dirname ${SCRIPTLOG}) | ||
- | |||
- | # DETECTING if the script is run by cron | ||
- | if [[ " | ||
- | set -x | ||
- | exec > ${SCRIPTLOG} | ||
- | exec 2> ${SCRIPTLOGERR} | ||
- | fi | ||
- | |||
- | if [[ ${DEBUG} -eq 0 ]] ; then | ||
- | echo -e " | ||
- | echo -e "\t SCRIPTLOG will be ${SCRIPTLOG}" | ||
- | echo -e "\t SCRIPTLOGERR will be ${SCRIPTLOGERR}" | ||
- | set -x | ||
- | exec 2> ${SCRIPTLOGERR} | ||
- | fi | ||
- | |||
- | |||
- | [[ ! -f ${CONFIGFILE} ]] && echo -e " | ||
- | . ${CONFIGFILE} | ||
- | |||
- | [[ ${DEBUG} -eq 0 ]] && DEBUGME=" | ||
- | |||
- | ${AWSCLI} ec2 describe-instances --filters Name=instance-type, | ||
- | let x=0 | ||
- | for PRIVATEIP in $(cat ${TMPFILE} | ${JQ} ' | ||
- | echo -e " | ||
- | let x++ | ||
- | done | ||
- | |||
- | if [[ $(cat ${HADYNAMICCFG} | wc -l) -ge ${MINBACKENDS} ]] ; then | ||
- | # ok, overwriting config | ||
- | echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" | ||
- | cat ${HAPROXYTEMPLATE} >> ${HAPROXYCONFIG} | ||
- | cat ${HADYNAMICCFG} >> ${HAPROXYCONFIG} | ||
- | echo -e "#### WARNING THIS CONFIG WILL BE REWRITTEN BY CRONJOB" | ||
- | systemctl reload haproxy | ||
- | fi | ||
- | |||
- | rm -f ${TMPFILE} | ||
- | rm -f ${HADYNAMICCFG} | ||
- | |||
- | exit ${EXITCODE} | ||
- | |||
- | ######################################################################## | ||
- | # | ||
- | # / MAIN | ||
- | # | ||
- | ######################################################################## | ||
- | |||
- | </ | ||
- | \\ | ||
- | Config file: | ||
- | <file config / | ||
- | |||
- | # Our template for re-generate the config | ||
- | HAPROXYTEMPLATE=/ | ||
- | # The configfile itself | ||
- | HAPROXYCONFIG=/ | ||
- | # Minimum amount of backends that must be running, less than this, the config file won't be changed | ||
- | MINBACKENDS=2 | ||
- | |||
- | # Tag for the scaling group (backend servers will be filtered by this tag) | ||
- | SCALINGTAG=" | ||
- | |||
- | # generic setup | ||
- | AWSCLI=/ | ||
- | JQ=/ | ||
- | |||
- | # SET to 0 for DEBUG | ||
- | DEBUG=0 | ||
- | </ | ||
- | |||
- | \\ | ||
- | Setup crontab with desired frequency for refresh, for example: | ||
- | <code crontab> | ||
- | */3 * * * * | ||
- | </ | ||
- | |||
- | |||
- | ====== Things I forgot ====== | ||
- | |||
- | * Purge log files from '' | ||
- | * A lot of additional checks for the script... | ||
marfeel/test.1584431890.txt.gz · Last modified: 2020/03/17 07:58 by dodger