Commit c5f1b3224fbde8e561e240f85860dc15c88f890c | zig

author	Andrew Kelley <andrew@ziglang.org> 2022-08-31 13:15:42-07:00
committer	Andrew Kelley <andrew@ziglang.org> 2022-08-31 13:15:42-07:00
log	c5f1b3224fbde8e561e240f85860dc15c88f890c
tree	f528db7782aa89970d504f357752f7d24ade7cff
parent	9a0cb34c734d83d42c70e03e4cc292860af0a17b

CI: update_download_page: change docs deployment

The original impetus for making a change here was a typo in --add-header
causing the script to fail. However, upon inspection, I was alarmed that
we were making a --recursive upload to the *root directory* of
ziglang.org. This could result in garbage files being uploaded to the
website, or important files being overwritten. As I addressed this concern,
I decided to take on file compression as well.

Removed compression prior to sending to S3. I am vetoing pre-compressing
objects for the following reasons:
 * It prevents clients from working which do not support gzip encoding.
 * It breaks a premise that objects on S3 are stored 1-to-1 with what is
   on disk.
 * It prevents Cloudflare from using a more efficient encoding, such as
   brotli, which they have started doing recently.

These systems such as Cloudflare or Fastly already do compression on
the fly, and we should interop with these systems instead of fighting them.

Cloudfront has an arbitrary limit of 9.5 MiB for auto-compression. I looked
and did not see a way to increase this limit. The data.js file is currently
16 MiB. In order to fix this problem, we need to do one of the following things:

 * Reduce the size of data.js to less than 9.5 MiB.
 * Figure out how to adjust the Cloudfront settings to increase the max size
   for auto-compressed objects.
 * Migrate to Fastly. Fastly appears to not have this limitation. Note
   that we already plan to migrate to Fastly for the website.

1 files changed, 32 insertions(+), 21 deletions(-)

ci
- srht
  - update_download_page

ci/srht/update_download_page+32-21

...	@@ -101,32 +101,43 @@ CIDIR="$(pwd)"	...	@@ -101,32 +101,43 @@ CIDIR="$(pwd)"
101		101
102	cd "$HOME"	102	cd "$HOME"
103		103
104	# Upload new stdlib autodocs	104	# Update autodocs and langref directly to S3 in order to prevent the
105	mkdir -p docs_to_upload/documentation/master/std/	105	# www.ziglang.org git repo from growing too big.
106	gzip -c -9 "$ZIGDIR/docs/std/index.html" > docs_to_upload/documentation/master/std/index.html	106
107	gzip -c -9 "$ZIGDIR/docs/std/data.js" > docs_to_upload/documentation/master/std/data.js	107	# Please do not edit this script to pre-compress the artifacts before they hit
108	gzip -c -9 "$ZIGDIR/docs/std/main.js" > docs_to_upload/documentation/master/std/main.js	108	# S3. This prevents the website from working on browsers that do not support gzip
109	gzip -c -9 "$LANGREF" > docs_to_upload/documentation/master/index.html	109	# encoding. Cloudfront will automatically compress files if they are less than
110	$S3CMD put -P --no-mime-magic --recursive --add-header="Content-Encoding:gzip" --add-header="Cache-Control: max-age=0, must-revalidate" "docs_to_upload/" s3://ziglang.org/	110	# 9.5 MiB, and the client advertises itself as capable of decompressing.
111		111	# The data.js file is currently 16 MiB. In order to fix this problem, we need to do
112	mkdir -p docs_src_to_upload/documentation/master/std/	112	# one of the following things:
113	cp -r "$ZIGDIR/docs/std/src" docs_src_to_upload/documentation/master/std/	113	# * Reduce the size of data.js to less than 9.5 MiB.
114	$S3CMD put -P --no-mime-magic --recursive --add-header:"Content-Type:text/html" --add-header="Cache-Control: max-age=0, must-revalidate" "docs_src_to_upload/" s3://ziglang.org/	114	# * Figure out how to adjust the Cloudfront settings to increase the max size for
115		115	# auto-compressed objects.
116	## Copy without compression:	116	# * Migrate to fastly.
117	# mkdir -p docs_to_upload/documentation/master/std/	117	$S3CMD put -P --no-mime-magic \
118	# cp "$ZIGDIR/docs/std/index.html" docs_to_upload/documentation/master/std/index.html	118	--add-header="Cache-Control: max-age=0, must-revalidate" \
119	# cp "$ZIGDIR/docs/std/data.js" docs_to_upload/documentation/master/std/data.js	119	"$LANGREF" s3://ziglang.org/documentation/master/index.html
120	# cp "$ZIGDIR/docs/std/main.js" docs_to_upload/documentation/master/std/main.js	120
121	# cp "$LANGREF" docs_to_upload/documentation/master/index.html	121	$S3CMD put -P --no-mime-magic \
122	# $S3CMD put -P --no-mime-magic --recursive --add-header="Cache-Control: max-age=0, must-revalidate" "docs_to_upload/" s3://ziglang.org/	122	--add-header="Cache-Control: max-age=0, must-revalidate" \
		123	"$ZIGDIR/docs/std/index.html" s3://ziglang.org/documentation/master/std/index.html
		124
		125	$S3CMD put -P --no-mime-magic \
		126	--add-header="Cache-Control: max-age=0, must-revalidate" \
		127	"$ZIGDIR/docs/std/main.js" s3://ziglang.org/documentation/master/std/main.js
		128
		129	$S3CMD put -P --no-mime-magic \
		130	--add-header="Cache-Control: max-age=0, must-revalidate" \
		131	"$ZIGDIR/docs/std/data.js" s3://ziglang.org/documentation/master/std/data.js
		132
		133	$S3CMD put -P --no-mime-magic \
		134	--add-header="cache-control: public, max-age=31536000, immutable" \
		135	"$HOME/$SRC_TARBALL" s3://ziglang.org/builds/
123		136
124	git clone --depth 1 git@github.com:ziglang/www.ziglang.org.git	137	git clone --depth 1 git@github.com:ziglang/www.ziglang.org.git
125	cd www.ziglang.org	138	cd www.ziglang.org
126	WWWDIR="$(pwd)"	139	WWWDIR="$(pwd)"
127		140
128	$S3CMD put -P --no-mime-magic --add-header="cache-control: public, max-age=31536000, immutable" "$HOME/$SRC_TARBALL" s3://ziglang.org/builds/
129
130	cd "$WWWDIR"	141	cd "$WWWDIR"
131	cp "$CIDIR/out/index.json" data/releases.json	142	cp "$CIDIR/out/index.json" data/releases.json
132	git add data/releases.json	143	git add data/releases.json