Why BootDev — Controlling Cache headers

From AWS document, when u want to cache objects at browser, From S3 or from Cloudfront, at the same time support CORs resources like font, You can use a parameter MaxAgeSeconds: http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html

With all the tests i tried Chrome don’t really respect MaxAgeSeconds , you still need traditional Cache-Control: Max-age=xxx AND Expires: into header. When using AWS Cloudfront as your edge cache / CDN, and especially adding S3 as your origin, you take special care of your cache headers.

You can use API / CLI / UI to change the cache header at the metadata session of S3.

Screen Shot 2015-07-10 at 3.41.24 PM

And at your bucket policy’s permission, set CORs

Screen Shot 2015-07-10 at 3.43.35 PM

Once you success in setting up those things, you can curl -I -H to test your settings. If you use Chrome to test, REMEMBER

  1. DONT click refresh
  2. DONT click CMD +R
  3. Click another link in your website to test 

Otherwise, you will end up in lots of confusion !

run command:

curl -I http://xxxxxx.example.com/fonts/Neutra2Display-Titling.woff -H “Origin: xxxx.example.com”

Screen Shot 2015-07-10 at 3.47.35 PMScreen Shot 2015-07-10 at 3.47.59 PM

first time u will see “Miss From Cloudfront”, if it is your production site url, you may ask why ? You should have many people visiting this obejcy. As the header is different than normal browser, Cloudfront treat it as a new object. So, no worry.

At the second time you curl, you will see “HIT from cloudfront”. So with this setup your resource (this time is font), will be cached on Cloudfront a long time and then once downloaded to browser, it will locally cached as the Cache-control: max-age set.

P.S. Cloudfront respect Cache-Control, so how long your browser will cache = how long your object will stay on Cloudfront.

With MaxAgeSeconds only, your resource can keep at browser with 304.

With Cache-control and expires header, your resource can keep at 200, from cache.

Question: So what does MaxAgeSeconds do here ? Any special require that we always want 304 but not 200, from cache ? I need someone to answer me as well 🙂

Why Bootdev — Dynamic CDN

In the old days, we put images, css / js, woff etc any assets to CDN, so that clients can download somewhere that geographically optimised.

Around end of 2012 – early 2013, new idea comes out like we should CDN everything, so that we can reduce the complex architecture with memcache, page cache cluster (Varnish) or even microccache etc. Just 1 layer cache and having everything to CDN. Like architecture below.

dnamic cdn

Your website domain name will directly point to CDN with CNAME. And then the CDN will point to your load balancer address or your web server. So it is like a proxy. When you do create, it will 100% bypass CDN and goes to web server, when u do UPDATE, the CDN will update from web server then invalidate itself, when you do DELETE, the CDN will bypass request to web server and invalidate its cache. When you read, it read from the CDN, not from web server. So the CRUD action can be completed.

You will need your own cache invalidation strategy, like send update to cloudfront / using versioning object or url.

Here is a sample conf of how we bypass some URLs to go to web server, and making Drupal works.

E1EB6A9C-17EC-446D-AD59-80B471A4F962 62367506-DDC3-4E5C-8F05-24E2D20DBBBB

With AWS cloudfront, you can bypass header ORIGIN, so that you can preform CORs actions. Also you can use similar header bypass feature to detect mobile/PC. With such architecture well setup, theoretically, you can have unlimited PV, as your server wont be really hitted. Your bound will be write DB bound only, which is not a concern in most case.

If you don’t want to understand all these, but want to lower your cost and have higher traffic and faster response, contact bootdev at founders@bootdev.com ! We can deploy Dynamic CDN to you in minutes no matter you are using AWS or not. We can point our CloudFront account to your server, it can be Azure, Linode, or any bare-meter. It just need to be Drupal, and you can enjoy the best performance ever.

ref: https://media.amazonwebservices.com/blog/cloudfront_dynamic_web_sites_full_1.jpg

fastcgi with apache, virtualhost per user setup

Im showing a working conf for fastcgi per user virtualhost & php.ini on apache2

Googled some reference with lots of apache like putting FcgidInitialEnv into virtualhost does not work. Tested out a working conf for fastcgi with apache, virtualhost per user by suExec setup.

About installing dependencies on CentOS 6, reference below ***Remember to add the epel repo

http://ruleoftech.com/2013/using-php-fpm-with-apache-2-on-centos
http://mcdee.com.au/fastcgi-with-php-apache-centos6/
http://kuaileyongshi.blog.51cto.com/1480649/648789

Beware suexec not allow to set uid < 500 ===> you cannot use apache (CentOS) www-data(Ubuntu) in SuexecUserGroup

Just add below apache conf in /etc/httpd/conf.d/anyfilename.conf, this conf run a php wrapper by FCGIWrapper

Then the wrapper can specify which php.ini to run

<IfModule fcgid_module>
  FcgidInitialEnv PHPRC "/php"
</IfModule>

# using mod_fcgid and user specific php.ini
# You can copy this part, add user and create more virtualhost on demand
<VirtualHost *:80>
  ServerName fcgi1.keithyau.com
  DocumentRoot "/var/www/keithyau"
  SuexecUserGroup keithyau keithyau

  <Directory /var/www/keithyau>
    Options FollowSymLinks +ExecCGI
    AllowOverride All
    AddHandler fcgid-script .php
    FCGIWrapper /var/www/change-this/fcgi1/fcgi1-wrapper .php
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>

The fcgi1-wrapper is something like this, PHPRC is the folder to contain custom php.ini you can add more settings.

  #!/bin/sh
  PHPRC=/var/www/keithyau/fcgi1
  export PHPRC
  export PHP_FCGI_MAX_REQUESTS=5000
  export PHP_FCGI_CHILDREN=8
  exec /var/www/html/fastcgi-bin/php-cgi

Screen Shot 2014-03-12 at 6.20.10 PM

 

We can config like disable_function in this php.ini. Finally, check the loaded phpinfo

淺談叢集式電腦 ( Clustering )

當你開始接觸 clustering 時就會發覺, 原來沒學 clustering 根本等於沒有學 Unix / Linux. 
我們一般情況用到的 SERVER / Client 只是皮毛. Clustering 一個字, 內裹包含了多種技術, 
以下例子是常用的
  1. HPC – High performance computing
  2. Server Load balancing 負載平衡
    1. DNS round robin
    2. LVS – ipvsadm
  3. HA – High availability
    1. XEN cluster
    2. Heartbeat
    3. DRBD - http://www.drbd.org/
  4. Parallel computing
    1. http://en.wikipedia.org/wiki/MPICH

今天所談的是入門級的 clustering, LVS

LVS – Linux virtual server主要是用作負載平衡 與 HA,

What is the Linux Virtual Server?

The Linux Virtual Server is a highly scalable and highly available server built on a cluster of real servers, with the load balancer running on the Linux operating system. The architecture of the server cluster is fully transparent to end users, and the users interact as if it were a single high-performance virtual server.

SOURCE: http://www.linuxvirtualserver.org/ 

負載平衡的意思是把一部服務器的工作分到多部服務器之上. 由一個服務器為主人 ( Master ), 其他為幫手 ( node ). 如下圖

more_on_clustering

更可以制定負載平衡的策略

詳程參考, http://linux.die.net/man/8/ipvsadm

    1. rr – Robin Robin: distributes jobs equally amongst the available real servers.
    2. wrr – Weighted Round Robin: assigns jobs to real servers proportionally to there real servers’ weight. Servers with higher weights receive new jobs first and get more jobs than servers with lower weights. Servers with equal weights get an equal distribution of new jobs.
    3. lc – Least-Connection: assigns more jobs to real servers with fewer active jobs.
    4. wlc – Weighted Least-Connection: assigns more jobs to servers with fewer jobs and relative to the real servers’ weight (Ci/Wi). This is the default.
    5. lblc – Locality-Based Least-Connection: assigns jobs destined for the same IP address to the same server if the server is not overloaded and available; otherwise assign jobs to servers with fewer jobs, and keep it for future assignment.
    6. lblcr – Locality-Based Least-Connection with Replication: assigns jobs destined for the same IP address to the least-connection node in the server set for the IP address. If all the node in the server set are over loaded, it picks up a node with fewer jobs in the cluster and adds it in the sever set for the target. If the server set has not been modified for the specified time, the most loaded node is removed from the server set, in order to avoid high degree of replication.
    7. dh – Destination Hashing: assigns jobs to servers through looking up a statically assigned hash table by their destination IP addresses.
    8. sh – Source Hashing: assigns jobs to servers through looking up a statically assigned hash table by their source IP addresses.
    9. sed – Shortest Expected Delay: assigns an incoming job to the server with the shortest expected delay. The expected delay that the job will experience is (Ci + 1) / Ui if sent to the ith server, in which Ci is the number of jobs on the the ith server and Ui is the fixed service rate (weight) of the ith server.
    10. nq – Never Queue: assigns an incoming job to an idle server if there is, instead of waiting for a fast one; if all the servers are busy, it adopts the Shortest Expected Delay policy to assign the job.

source: Man page of ipvsadm

負載平衡方法大致上有

  1. NAT (Network address translation)
  2. DR – direct routing
  3. Tunneling

以下是 LVS – NAT LOAD BALANCING 的參考

原意圖

VS-NAT

VS via NAT, source: http://www.linuxvirtualserver.org/VS-NAT.html

指令

Master (NAT)

echo 1 > /proc/sys/net/ipv4/ip_forward

ipvsadm -A -t 192.168.7.200:80 -s rr

ipvsadm -a -t 192.168.7.200:80 -r 192.168.7.201:80 -m

ipvsadm -a -t 192.168.7.200:80 -r 192.168.7.202:80 -m

ipvsadm -a -t 192.168.7.200:80 -r 192.168.7.203:80 -m

ipvsadm -L –stats (現時的連接情況)

Nodes:

Set default route to 192.168.7.200

route add default gw 192.168.7.000

ip 地址

名稱 意思
192.168.7.200 Master
192.168.7.201 Node
192.168.7.202 Node
192.168.7.203 Node
192.168.7.1 Default GW
80 Www – web 服務器的 port number
rr Round robin 策略
-m LVS NAT
-a ADD SERVER
-A ADD SERVICE

Thanks, 有興趣多討論請跟我連絡 keithyau@yubis.net !!

淺談網頁服務器群的設計 (A group of web server)

前陣子, 在師兄那邊聽了一個他的經驗, 一間寄存公司其中一個網頁服務器壞了的經驗.

話說一個裝有 800 客戶的網頁服務器只用 raid 5 作備份, 沒有備份服務器 (HA / heartbeat)的低成本制作,
某天一下子死掉兩個硬碟. 之後的 18 小時, 客戶服務部就 ...

在 HSBC 中學得的服務器管理知識, 告訴我任何作實戰 production 的服務器必需有備份,
而且是能夠在 15 分鐘內回復的備份. 不過, 現實情況是一些中小型機構, 沒有兩倍的成本去做備份.
所以回復時間在 18-24小時內都可以接受.

現時中小企的 IT 成本, 軟件比硬件高得多. 所以如果選用開源軟件,
再買一個 HKD $3000 的小服務器就安全得多了.

本文是以一個案例作介紹以虛擬技術把服務器模組化, 令備份 / 資源調動時方便得多.
那我師兄的故事, 如果是我的話怎樣處理 ? 先發現問題所在

問題 1: 裝有 800 客戶的服務器沒有虛擬化, 在救援 / 備份時造成不便
問題 2: 沒有備用服務器, 服務器停止的 18 小時中客戶服務部的開支比買一個小型服務器更高
問題 3: 除 raid 5 以外, 沒有資料備份. 原因可能是沒有虛擬化令備份程序複雜

所以根本問題是服務器的設計, 與技術人員的能耐無關 !

解決方法: 沒有人說備份服務器的比例一定是 1:1 的, 以一部備份多部 production 亦可.
因成本與客戶要求而定, 備份服務器不一定要有 raid, 不一定要有足夠的計算能力 (cpu power / cpw).
因此, 一部 HKD $3000 的服務器, 只要能夠作備份就足夠有餘了. 

但出現的問題是, 

1) 如果我把所有客戶的資料都備份, 資源需求很大. 這時可考慮 rsync, rsync 是增加式資料備份,
只備份已更改 / 新增檔案

以下是 rysnc 的示範, 在linux備份服務器輸入:
sudo rsync -avl --delete --exclude=/不包括的資料夾1 --exclude=/不包括的資料夾2 
來源地址:/要備份的資料夾 /本地儲存資料的位置

2) CPU 已經用來服務客人了, 怎樣做備份
這時需要 at 指令 / crontab, 可以設定只在特定時間作備份. 以前在 HSBC 時, 備份會在股市收市後進行.
以 at command 作示範

at 1900     ← 意思是當天下午 7:00
> 備份指令 (如以上的 rsync)
> 完成後 CTRL + D

除此之外, 把服務器虛擬化, 有些選擇是把服務器放到一個 img 檔案中.
這樣, 備份時把整個 img 檔案備份就能佔用較小資源.

3) 出事了, 怎樣啟動備用服務器 ?

你可以選用 HA (high availability) 設計, 並以 heartbeat 啟動, 這樣就可以作即時回復. 但成本較高.
在這文中不作詳談. 一般來說, 可以手動. 但先要為服務器作一個好的設計.

以下是一個網頁服務器群的設計

所謂服務器模組化就是一個服務器只作一種服務. 以前的日子, 成本會很高. 但在虛擬化之下就不算問題. 參考虛擬化制作: http://www.yubis.net/blog/keith/summary-virtualbox-vmware-xen-351?page=2

在服務器模組化出名的有 MICROSOFT VIRTUAL SERVER, 但使用 xen 的話更好. 在使用服務器群時會出現一個重大問題. 就是同一種服務不能有兩個服務器, 因為同一個 ip 地址 不能用同一個 port. 例如 port 80, 路由器只能指向一個服務器. 這時, 模組化就會不攻自破.

以下是一個解決方法

web_server_group1

上圖中有兩個網頁服務器. External 是給客人用的. 但如果想把公司的網頁都用同一個 ip 地址放出去, 那就要 proxy 服務器的幫助. 先在 External 中開動 apache 的 proxy modules

# a2enmod proxy*

#/etc/init.d/apache2 restart

例如公司的網頁是 abc.xyz.org, 那分別在 dns 服務器, External & Internal 中加上圖中的設定, abc.xyz.org 就可以提供服務了.

以上說的與師兄的案例有什麼關連 ?

試把圖中 internal web server 當成師兄故事中所死掉的服務器, 那不就把 External 中的設定改一下, 由 死掉的服務器指向備份中的服務器就成了嗎 ? 同樣方法可以加上更多的網頁服務器  !

如何解決MySQL 開動不到的問題 MySQL start fail

昨天在服務器搬遷時遇到問題, 就是搬遷後, Mysql Start Fail – MySQL 資料庫不能開始. 找了很久都沒有答案, 檢查程序如下

以下的例子都是在 Ubuntu 下執行, 最後就是我的解決方法

檢查MySQL 的設定檔

Sudo vi /etc/mysql/my.cnf

[client]

port = 3306

socket = /var/run/mysqld/mysqld.sock

[client]

port = 3306

socket = /var/run/mysqld/mysqld.sock

# This was formally known as [safe_mysqld]. Both versions are currently parsed.

[mysqld_safe]

socket = /var/run/mysqld/mysqld.sock

nice = 0

[mysqld]

user = mysql

pid-file = /var/run/mysqld/mysqld.pid

socket = /var/run/mysqld/mysqld.sock

port = 3306

basedir = /usr

datadir = /var/lib/mysql # 確定 folder 是否存在

tmpdir = /tmp

language = /usr/share/mysql/english

skip-external-locking

bind-address = 127.0.0.1# 如果這裡是其他 ip 地址, 有可能造成 MySQL start fail

key_buffer = 16M

max_allowed_packet = 16M

thread_stack = 128K

thread_cache_size = 8

query_cache_limit = 1M

query_cache_size = 16M

log_bin = /var/log/mysql/mysql-bin.log

# WARNING: Using expire_logs_days without bin_log crashes the server! See README.Debian!

expire_logs_days = 10

max_binlog_size = 100M

skip-bdb

[mysqldump]

quick

quote-names

max_allowed_packet = 16M

[mysql]

#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]

key_buffer = 16M

!includedir /etc/mysql/conf.d/

檢查MySQL 資料夾的權限

權限分別為

keithyau@Yubis-production:/var/lib$ sudo ls -l /var/lib/mysql

total 20612

-rw-r–r– 1 root root 0 2008-03-16 22:01 debian-5.0.flag

drwx—— 2 mysql mysql 12288 2008-08-16 02:29 drupal

-rw-rw—- 1 mysql mysql 10485760 2009-02-26 21:00 ibdata1

-rw-rw—- 1 mysql mysql 5242880 2009-02-26 21:08 ib_logfile0

-rw-rw—- 1 mysql mysql 5242880 2008-03-16 22:01 ib_logfile1

drwxr-xr-x 2 mysql root 4096 2002-01-13 17:16 mysql

-rw——- 1 root root 6 2008-03-16 22:01 mysql_upgrade_info

keithyau@Yubis-production:/var/lib$ sudo ls -l /var/lib

total 176

drwxr-xr-x 4 root root 4096 2008-03-14 20:39 apt

drwxr-xr-x 2 root root 4096 2008-03-14 20:41 aptitude

drwxr-xr-x 2 root root 4096 2002-01-26 19:48 dhcp3

drwxr-xr-x 4 root root 4096 2008-03-14 20:52 dictionaries-common

drwxr-xr-x 2 root root 4096 2006-10-24 03:27 discover

drwxr-xr-x 7 root root 4096 2009-02-26 20:45 dpkg

drwxr-xr-x 3 root root 4096 2008-03-14 20:53 gstreamer

drwxr-xr-x 2 root root 4096 2008-03-14 20:39 initramfs-tools

drwxr-xr-x 2 root root 4096 2007-01-31 06:27 initscripts

drwxr-xr-x 2 root root 4096 2009-02-26 19:17 lib_mysql

drwxr-xr-x 2 root root 4096 2008-03-14 22:32 logrotate

drwxr-xr-x 6 root root 4096 2009-01-03 14:12 menu-xdg

drwxr-xr-x 2 root root 4096 2006-10-28 22:06 misc

drwxr-xr-x 10 mysql mysql 4096 2009-02-26 21:08 mysql

drwxr-xr-x 2 root root 4096 2008-01-27 19:46 mysql-cluster

檢查後依然不能開始MySQL 服務器 ……

打開 /var/log/syslog 看看有沒有以下一行

Feb 26 20:45:59 Yubis-production /etc/init.d/mysql[6254]: ^G/usr/bin/mysqladmin: connect to server at ‘localhost’ failed

Feb 26 20:45:59 Yubis-production /etc/init.d/mysql[6254]: error: ‘Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)’

Feb 26 20:45:59 Yubis-production /etc/init.d/mysql[6254]: Check that mysqld is running and that the socket: ‘/var/run/mysqld/mysqld.sock’ exists!

Feb 26 20:45:59 Yubis-production /etc/init.d/mysql[6254]:

Feb 26 20:46:04 Yubis-production mysqld_safe[6319]: started

Feb 26 20:46:04 Yubis-production mysqld[6328]: 090226 20:46:04 InnoDB: Started; log sequence number 0 43665

Feb 26 20:46:04 Yubis-production mysqld[6328]: 090226 20:46:04 [ERROR] Binlog has bad magic number; It’s not a binary log file that can be used by this version of MySQL

如果存在的話, 應該是你的 MySQL 服務器沒有正常關上, 那就 …

打開 /var/log/mysql

keithyau@Yubis-production:/var/lib$ sudo ls -l /var/log/mysql/
total 295696
-rw-rw—- 1 mysql adm 104885450 2009-02-18 12:36 mysql-bin.000077
-rw-rw—- 1 mysql adm 105175310 2009-02-24 11:27 mysql-bin.000078
-rw-rw—- 1 mysql adm 83164522 2009-02-26 19:24 mysql-bin.000079
-rw-rw—- 1 mysql adm 98 2009-02-26 19:56 mysql-bin.000080
-rw-rw—- 1 mysql adm 9233377 2009-02-27 16:10 mysql-bin.000081
-rw-rw—- 1 mysql adm 160 2009-02-26 21:08 mysql-bin.index

把最後一個 bin 檔案刪除

sudo rm /var/log/mysql/mysql-bin.00081

再打開 mysql-bin.index 把最後一行刪除

sudo vi /var/log/mysql/mysql-bin.index

/var/log/mysql/mysql-bin.000077

/var/log/mysql/mysql-bin.000078

/var/log/mysql/mysql-bin.000079

/var/log/mysql/mysql-bin.000080

/var/log/mysql/mysql-bin.000081

重新打開 MySQL 資料庫就完成了

Sudo /etc/init.d/mysql restart