cfebs/blog
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

build_a_blog: bench, main errors

Browse source
This commit is contained in:
Collin Lefeber 2024-06-19 15:34:10 -04:00
parent d2b810441d
commit b44ec21007
3 changed files with 20 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
bench.sh
View file

@ -16,6 +16,11 @@ echo "INFO: number of *.md files $(find ./posts/ -iname '*.md' | wc -l)" 1>&2
echo "INFO: number of *.html files $(find ./posts/ -iname '*.html' | wc -l)" 1>&2 echo "INFO: number of *.html files $(find ./posts/ -iname '*.html' | wc -l)" 1>&2
echo "INFO: running" 1>&2 echo "INFO: running" 1>&2
time -p python main.py 2>/dev/null time -p python main.py 2>/dev/null
rc=$?
if [[ "$rc" != "0" ]]; then
echo "ERROR: program exited with $rc" 1>&2
exit 1
fi
echo "INFO: number of *.html files $(find ./posts/ -iname '*.html' | wc -l)" 1>&2 echo "INFO: number of *.html files $(find ./posts/ -iname '*.html' | wc -l)" 1>&2
echo "INFO: cleanup __bench files" 1>&2 echo "INFO: cleanup __bench files" 1>&2
rm -f ./posts/*__bench* rm -f ./posts/*__bench*

2
main.py
View file

@ -59,7 +59,7 @@ def render_posts():
with Pool(processes=cpu_count) as pool: with Pool(processes=cpu_count) as pool:
posts = pool.map(render_post, files) posts = pool.map(render_post, files)
logging.info("render_posts result: %s", res) logging.info("render_posts result: %s", posts)
return posts return posts
def posts_list_html(posts): def posts_list_html(posts):

25
posts/build_a_blog.md
View file

@ -757,6 +757,11 @@ echo "INFO: number of *.md files $(find ./posts/ -iname '*.md' | wc -l)" 1>&2
echo "INFO: number of *.html files $(find ./posts/ -iname '*.html' | wc -l)" 1>&2 echo "INFO: number of *.html files $(find ./posts/ -iname '*.html' | wc -l)" 1>&2
echo "INFO: running" 1>&2 echo "INFO: running" 1>&2
time -p python main.py 2>/dev/null time -p python main.py 2>/dev/null
rc=$?
if [[ "$rc" != "0" ]]; then
echo "ERROR: program exited with $rc" 1>&2
exit 1
fi
echo "INFO: number of *.html files $(find ./posts/ -iname '*.html' | wc -l)" 1>&2 echo "INFO: number of *.html files $(find ./posts/ -iname '*.html' | wc -l)" 1>&2
echo "INFO: cleanup __bench files" 1>&2 echo "INFO: cleanup __bench files" 1>&2
rm -f ./posts/*__bench* rm -f ./posts/*__bench*
@ -807,7 +812,7 @@ So here is the diff to make that happen:
```diff ```diff
diff --git a/main.py b/main.py diff --git a/main.py b/main.py
index 52ce57b..91d852b 100644 index 52ce57b..95b650d 100644
--- a/main.py --- a/main.py
+++ b/main.py +++ b/main.py
@@ -1,9 +1,11 @@ @@ -1,9 +1,11 @@
@ -842,7 +847,7 @@ index 52ce57b..91d852b 100644
+ with Pool(processes=cpu_count) as pool: + with Pool(processes=cpu_count) as pool:
+ posts = pool.map(render_post, files) + posts = pool.map(render_post, files)
+ logging.info("render_posts result: %s", res) + logging.info("render_posts result: %s", posts)
return posts return posts
def posts_list_html(posts): def posts_list_html(posts):
@ -857,9 +862,9 @@ INFO: removing old __bench files
INFO: number of *.md files 102 INFO: number of *.md files 102
INFO: number of *.html files 2 INFO: number of *.html files 2
INFO: running INFO: running
real 0.21 real 0.27
user 1.64 user 1.82
sys 0.14 sys 0.15
INFO: number of *.html files 102 INFO: number of *.html files 102
INFO: cleanup __bench files INFO: cleanup __bench files
@ -868,16 +873,14 @@ INFO: removing old __bench files
INFO: number of *.md files 1002 INFO: number of *.md files 1002
INFO: number of *.html files 2 INFO: number of *.html files 2
INFO: running INFO: running
real 1.12 real 1.25
user 15.69 user 16.68
sys 0.24 sys 0.52
INFO: number of *.html files 1002 INFO: number of *.html files 1002
INFO: cleanup __bench files INFO: cleanup __bench files
``` ```
So down to <1s for 1000 posts with 16 cores (`multiprocessing.Pool` uses cpu core count by default if not passed) . Down to 1.25s for 1000 posts 🎉
🎉 that's cool!
[1]: https://crystal-lang.org/ [1]: https://crystal-lang.org/
[2]: https://github.com/crystal-lang/crystal/releases/tag/0.31.0 [2]: https://github.com/crystal-lang/crystal/releases/tag/0.31.0