I want a drink and it’s not even 0700 yet

On this here blog I use a few things to help secure everything down and avoid issues, namely, nginx location blocks disallowing access to resources, fail2ban tracking nginx logs to prevent people hammering server or trying to do improper things, and the “Limit Login Attempts” WP plugin.

A combination of all these broke access with the wordpress mobile app. Ended up having to disable the wordpress fail2ban jail and altering some of the nginx directives.

This is going to be a pain in the ass to debug cos the wordpress app doesn’t have any kind of proper error messaging, urgh.

Wiki.js 2 with Nginx Installation

For the past few months I’ve been using Tiddlywiki as a memory dump but been having some issues. First started with the dreaded XMLHttpRequest error:

Error retrieving skinny tiddler list: XMLHttpRequest error code: 404

Which the available documentation offers no help with and the developers just shrug at. Then it just ate a fucken shotgun shell deep down its throat:

Internal JavaScript Error: TypeError: etag is null

We en’t here for that shit so on we went looking for an alternative that treats markdown as a first-class white citizen in apartheid america. Found wiki.js, which seems to have that, and here we are.

What follows is a guide written after a week of bashing our head against multiple desks because devlopers are morons who don’t know how to write documentation, if they even bother writing any. What is available for wiki.js is fucken laughable or only applies to the 1.x series. Real developers are extinct, by the way.

This is what worked for us on Debian 9. You will have to adapt this for your own OS and hosting configuration. We’re not at fault if the results eat your pet, fuck your significant other, and make your mom call them daddy.


This assumes DNS is already routing properly, outgoing mail works, and you’ve already dealt with your firewall. This setup gets you a wiki.js installation with nginx as a reverse proxy running security.

All commands are executed as root.


Install what you need

# aptitude install nginx-extras postgresql postgresql-contrib pgcli nodejs certbot python-3-certbot-nginx

Download and extract wiki.js (assuming we’re at /var/www) like the documentation says:

# wget
# mkdir wiki
# tar xzf wiki-js.tar.gz -C ./wiki
# cd ./wiki
# mv config.sample.yml config.yml



Edit your configuration file for nginx so it passes everything to the wiki cleanly through nginx. The original configuration was generated by and incorporates stuff from the official documentation

As of right now (2020-05-16_14-28) they are valid and working server blocks

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name wiki.domain.invalid;

    # SSL
    ssl_certificate /etc/letsencrypt/live/wiki.domain.invalid/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/wiki.domain.invalid/privkey.pem; #managed by Certbot
    ssl_trusted_certificate /etc/letsencrypt/live/wiki.domain.invalid/chain.pem;

    # security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    #add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
    add_header Strict-Transport-Security "max-age=0" always;

    # . files
    location ~ /\.(?!well-known) {
        deny all;

    # logging
    access_log /var/log/nginx/wiki.domain.invalid.access.log;
    error_log /var/log/nginx/wiki.domain.invalid.error.log warn;

    # reverse proxy
    location / {
        proxy_http_version                  1.1;
        #proxy_cache_bypass                  http_upgrade;
        proxy_set_header Upgradehttp_upgrade;
        proxy_set_header Connection         "upgrade";
        proxy_set_header Host               http_host;
        proxy_set_header X-Real-IPremote_addr;
        #proxy_set_header X-Forwarded-For    proxy_add_x_forwarded_for;
        #proxy_set_header X-Forwarded-Protoscheme;
        #proxy_set_header X-Forwarded-Host   host;
        #proxy_set_header X-Forwarded-Portserver_port;
                proxy_next_upstream error timeout http_502 http_503 http_504;

    # gzip
    gzip on;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;

    # HTTP redirect
server {
    listen 80;
    listen [::]:80;

    server_name wiki.domain.invalid;

    # ACME-challenge
    location ^~ /.well-known/acme-challenge/ {
        root /var/www/_letsencrypt;

    location / {
        return 301 https://wiki.domain.invalid$request_uri;



Using Let’s Encrypt SSL certificates:

# certbot

Go through the wizard and it will automatically fix the SSL entries on your server blocks. You could also do this if you know what you’re doing and don’t want certbot to mess around with your files:

# certbot certonly --webroot -d wiki.domain.invalid --email mail@domain.invalid -w /var/www/_letsencrypt -n --agree-to-tos

Nginx Testing

Test and reload your configuration:

# nginx -t
# service nginx reload

Watch out for any errors, as usual. At this point Nginx will be serving files but as wiki.js isn’t setup yet you’ll get HTTP 502 errors if you try to visit the site on a browser. This configuration plays well with other sites being hosted on the same server.


Secure your Postgres installation:

# sudo su postgres
$ passwd

Then setup your database. pgcli has smart completions turned on by default and looks pretty.

$ pgcli

> create DATABASE wikijs;
> create USER wikijs_user with ENCRYPTED PASSWORD 'Strong password';
> grant ALL PRIVILEGES on DATABASE wikijs to wikijs_user;
> \c wikijs
> exit

$ exit


Edit config.yml and make the appropriate changes:

  • Port should match what was configured in the nginx https server block (3000)
  • In db section, enter your database credentials
  • Do not enable SSL unless you are not to run this behind a proxy. This might work on a developer workstation but on the public internet you’re asking to get it up the ass, no lube.

Once this is done, start the application and watch for any errors

# node server

At this point you can visit your site and go through the installation wizard.


There are a bunch of things the official wiki.js documentation only mentions offhandedly, or that you’ll only find out if you go rooting around in the issues tracker.

Home Page

You can name it anything you want but if you make the path anything other than /home wiki.js will freak out on you and send you on a loop.

File Storage

By default wiki.js will keep all its shit on the DB, which is a fucken stupid bad decision. We like making good decisions so we need to tell wiki.js to keep its shit in the filesystem:

  1. Go to Administration > Storage
  2. Enter the desired absolute path for your stuff, like /var/www/wiki.domain.invalid/wiki-content
  3. Enable the target
  4. Apply the changes

We’re unsure if this means wiki.js will actually use file storage to begin with, but at least you’ll be able to create quick backups of all your stuff. You have backups and you test them, right?

Search Engine

The default search is slow AF, so we’re going to use something better

  1. Go to Administration > Search Engine
  2. Select Database – PostgreSQL
  3. Apply the changes

Finishing thoughts

This thing has potential but it’s got a long way to go before it can look up to MediaWiki. If you find issues with this holler at me on the twitters.

Let’s Encrypt Renewal

Yeah yeah, SSL certificates expired like what, a month and a half ago? Anyway, I needed to renew it but it attached a bunch of domains to the certificate on this host and I needed to start over. There are a bunch of options but the most straighforward for me was to just delete everything and start over. So, in the shell:

# cp /etc/letsencrypt /etc/letsencrypt.backup -r
# rm -rf /etc/letsencrypt/live
# rm -rf /etc/letsencrypt/archive
# rm -rf /etc/letsencrypt/renewal/YOURCERTIFICATEDOMAIN.conf
# certbot certonly

At this point it’ll ask you for webroot an’ stuff so just answer the questions and nobody gets hurt. These instructions work for Apache but I’m using NginX, so then you have to edit the configuration for the site(s) in /etc/nginx/sites-available/. There will be a couple lines going like:

ssl_certificate /etc/letsencrypt/live/OLDCERTIFICATEDOMAIN/fullchain.pem; # managed by Certbot
ssl_certificate /etc/letsencrypt/archive/OLDCERTIFICATEDOMAIN/fullchain.pem; # managed by Certbot

Just update them to point to the most current certification location, then restart NginX with service nginx restart

nginx + hhvm, continued, part three

It’s starting to sound like a bad Hollywood movie. You think you’re done with it and then BOOM it comes out of nowhere straight to DVD. Back when movies did that.

So after I played around with everything, bloody HHVM kept crashing — I’m starting to think it’s a fucking piece of shit just like facebook, which begat it — and all I could get out of the various things in /var/log were that it kept running out of memory. htop didn’t really say anything, nor ps aux nor netstat

Using systemctl status hhvm.service or journalctl -xn reported only that something was wrong, but didn’t say what. Which is annoying since fucking piece of shit systemd wants everyone to use its own tools for managing the system.

Gods, I dislike systemd. But carrying on…

Checking /var/log/messages manually led me to believe that hhvm was running out of memory and not failing gracefully. So that led me to looking around. Then I got this snippet from this site, which I added to my /etc/hhvm/server.ini:

hhvm.jit = false

# service nginx restart && service hhvm restart

Since I did that I haven’t had to restart hhvm. Sure, I lose performance, but since this is my personal blog and I’ve had issues with php-fpm and hhvm I would much prefer to have reliability.

PS: Yes, I work on root. I’m a real sysadmin, not like y’all wankers who work with sudo.

nginx + hhvm, continued

So after my last post on this, HHVM stopped working again, making nginx give a 502 as usual.

This time, a comment on a forum (who knows which, I looked at a lot of pages) gave me a little snippet that helped me pinpoint the issue:
# netstat -plunt | grep hhvm

This should have told me that HHVM was listening on a TCP port (since that’s what the default for HHVM is). But netstat reported nothing. After checking my configuration, I had forgotten that I had iptables drop all traffic to 127/8 that doesn’t use loopback. Here’s the snippet I use in my iptables configuration file

Commented the line out, imported the new ruleset into iptables and iptables didn’t block traffic sent from nginx to HHVM anymore, letting everything work as is supposed to.

Now I’m hoping it doesn’t die again due to some other reason. This was really annoying.

nginx + hhvm

I’m not sure what I did, but I finally got it to work. My previous configuration had nginx watch out for a 502 BAD GATEWAY error to switch from HHVM to php-fpm.

Too bad that every single PHP request invoked a 502 error. But my blog worked and I didn’t check the logs cos I happily thought everything was working.

Until it didn’t.

MariaDB bitched. Then php-fpm started using 100% CPU until it killed MariaDB. nginx didn’t care cos it doesn’t use that much memory to begin with. But the end result of all this was that my blog didn’t work.

Ended up reinstalling everything from scratch. Then it somehow works. I should write up about it later so I don’t forget.

But right now I need to shower so I can be at work in a half hour. Yes, at 0630. And now WordPress is bitching about something going wrong. I’ll probably just end up switching back to good ol’ Apache with the standard PHP module in.