.DEFAULT_GOAL := help

help:
	@grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}'

command1: command2 ## Alias of command2
	echo 'Done!'

command2: ## run command2
	echo 'Running command2'

command3: # Unlisted command
	@echo "I'm private"

Explanation

1. .DEFAULT_GOAL will run provided make command if none is specified, so by doing make it will execute make help
2. @grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST):
   • @ will suppress output because by default make will print out the command it’s executing
   • grep ... will look for lines that have commands with comments separated by ##
   • '^[a-zA-Z0-9_-]+:.*?## .*$$' means search from start of line, any alphanumeric, _ and - command names followed by : and any character until ## to end of line (since $ has special meaning in make, we have to escape it so end of line becomes $$)
   • MAKEFILE_LIST is make variable that refers to the makefile name in this case
3. sort to get alphabetically sorted output
4. awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}':
   • pipe the lines through awk and set the FS field separator to be whatever between the command and help text (modify ## if you want different separator)
   • for each match print with color: \033[36m%-15s\033[0m %s\n so we will get command, padded with 15 spaces, followed by the help text
   • again since we’re inside make escape $1 & $2 for column number with $$1 & $$2
5. For commands you don’t want to be listed as help output, simply don’t comment or use single # like command3 example

Output

$ make
command1        Alias of command2
command2        run command2

I have tried everything suggested from disabling Gatekeeper to many many iterations of uninstall/reinstall. Almost every solution suggested clicking Allow on the Security & Privacy tab but it never appeared.

What worked for me finally was: macos - VirtualBox 5.2 Won’t Install on Mac OS 10.13 - Ask Different which boils down to:

Go into recovery mode (reboot with Cmd + R during loading screen) and open the terminal:

spctl kext-consent disable
spctl kext-consent add VB5E2TV963
spctl kext-consent enable 
reboot

That is basically the Allow part which will finally make the install successful. VB5E2TV963 is the Oracle id that we are allowing.

If you are in similar situation, I hope this will help.

Adopting Alpine

At work, we started adopting Alpine pretty early on for development, CI, and even production. All is well, except when it’s not. Turns out it’s a lot of work to get packages that are not readily available in Alpine repository. You see, Alpine uses musl libc instead of glibc and most popular distros use the latter. So things compiled in Alpine won’t be usable on Ubuntu, for example, and vice versa. It’s great when all packages you need are there in the main & community alpine repositories. When that is not the case, you have to build it on your own and hope that the dependencies are available or at least easy to build as well (against musl).

Missing package

One day our CI started failing during docker image build phase. mysql package (which is just a compatibility package pointing to mariadb) suddenly went missing. We issued the bug report: Bug #8030: Missing x86_64 architecture for mysql and mysql-client packages in Alpine v3.3 - Alpine Linux - Alpine Linux Development. To Natanael’s credit, the issue was resolved within the day, but this issue got us to start questioning things.

Losing version

I covered this in a previous post, it’s basically about the difficulty in pinning package versions in Alpine. I would continue but I think this guy got through the same situation and has the same thoughts. Recommended read.

When a package is not available

Developers at work needed to use PHP V8js for our experimental branch so I had to get the extension for our alpine-based images. Someone got through most of the trouble for me as he detailed the findings in this Github gist. Basically we had to compile GN, download v8 source, and then build it against musl. Even with the steps laid out, it wasn’t a smooth experience. This method is not sustainable for us, considering future updates and patches.

Syslog limit

Developers rely heavily on app logs via syslog (mounted /dev/log) and Alpine uses busybox syslog by default. The problem is, messages are truncated at 1024-character limit, which is very small. The root issue is musl has hardcoded limit of 1024 syslog buffer, which is a generous increase from the initial 256(!) limit but still not enough. This doesn’t make sense as a default, and I could not find a way to configure this easily.

Final straw

Ubuntu officially launched minimal ubuntu images for cloud / container use around July last year.

These images are less than 50% the size of the standard Ubuntu server image, and boot up to 40% faster.

Not only the base image just 29MB in size, but you also get to use all apt packages! All our previous problems with Alpine made it very easy to switch to ubuntu as our base image and we have been satisfied with the switch so far. This in not to say Alpine is not good. It’s just not a fit for us.

Edit: I need to clarify that this post is not about what’s the better alternative. Rather, it’s about ditching Alpine. There are definitely few good alternatives to ubuntu-minimal. :)

This is the minimal test that could reproduce the problem:

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.google.com");
curl_exec($ch);
curl_close($ch);

Which would get you some variation of this coredump:

Thread 1 "php" received signal SIGSEGV, Segmentation fault.
0x00007ffff540dfc3 in ?? () from /lib/libssl.so.43
(gdb) bt
#0  0x00007ffff540dfc3 in ?? () from /lib/libssl.so.43
#1  0x00007ffff6545eac in ssl_create_cipher_list () from /lib/libssl.so.1.0.0
#2  0x00007ffff653faf4 in SSL_CTX_new () from /lib/libssl.so.1.0.0
#3  0x00007ffff58a495a in ?? () from /usr/lib/libcurl.so.4
#4  0x00007ffff58a59b9 in ?? () from /usr/lib/libcurl.so.4
#5  0x00007ffff58a6553 in ?? () from /usr/lib/libcurl.so.4
#6  0x00007ffff5867b3c in ?? () from /usr/lib/libcurl.so.4
#7  0x00007ffff5868e7f in ?? () from /usr/lib/libcurl.so.4
#8  0x00007ffff5872030 in ?? () from /usr/lib/libcurl.so.4
#9  0x00007ffff5882810 in ?? () from /usr/lib/libcurl.so.4
#10 0x00007ffff5883470 in curl_multi_perform () from /usr/lib/libcurl.so.4
#11 0x00007ffff5acd1ba in ?? () from /usr/lib/php7/modules/curl.so

Ok what the heck, let’s just downgrade, right? That’s what we did via:

RUN echo "http://dl-cdn.alpinelinux.org/alpine/v3.4/main" >> /etc/apk/repositories \
    && apk update \
    && apk add --no-cache \
                ...
                curl=7.59.0-r0 \
                libcurl=7.59.0-r0 \

That solved it… until a month later when build failed again because the pinned version has gone missing from alpine 3.4 repository. I’ve experienced this a few times, and I’m not alone.

Our solution then is to just upgrade to alpine 3.8 with fixed version of curl and unpinned them. As of this moment of writing, we have migrated away from Alpine, which I will cover in future post.

The default naming scheme for containers created by Compose in this version has changed from <project>_<service>_<index> to <project>_<service>_<index>_<slug> where <slug> is a randomly-generated hexadecimal string.

The “warning” came only about 1 month prior, hidden in 1.23.0-rc1 (2018-09-26) release note. So previously your project_app_1 will now be project_app_1_abc123! Unsurprisingly this caused a lot of breakage and rage.

It was promptly reverted a month later in 1.23.2 (2018-11-28).

There’s a lot of lesson to learn here from the perspectives of both maintainer and user of project, as evident by going through the comments in the issue.

I was heavily inspired by this dockerized vim project. Check it out if you think that’s enough for you. Or you can use the images I have prepared:

• asyazwan/vim:tiny - 24MB compressed / 61MB uncompressed
• asyazwan/vim:small - 24MB compressed / 61MB uncompressed
• asyazwan/vim:normal - 25MB compressed / 62MB uncompressed
• asyazwan/vim:big - 25MB compressed / 62MB uncompressed
• asyazwan/vim:huge - 25MB compressed / 62MB uncompressed

Check out this table of comparison for the differences between features

But if you want to make your own image, here’s how I did it.

I have pushed the Dockerfiles used into this repository. Check out the main Dockerfile:

FROM alpine:3.8 as builder

WORKDIR /tmp

RUN apk add --no-cache \
  build-base \
  ctags \
  git \
  libx11-dev \
  libxpm-dev \
  libxt-dev \
  make \
  ncurses-dev \
  python3 \
  python3-dev \
  perl-dev \
  ruby-dev

RUN git clone https://github.com/vim/vim && cd vim \
  && ./configure \
  --disable-gui \
  --disable-netbeans \
  --enable-multibyte \
  --enable-perlinterp \
  --enable-rubyinterp \
  --enable-python3interp \
  --with-features=huge \
  --with-python3-config-dir=/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/ \
  --with-compiledby=asyazwan@gmail.com \
  && make install


FROM alpine:3.8

COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /usr/local/share/vim  /usr/local/share/vim
RUN apk add --update --no-cache \
  libice \
  libsm \
  libx11 \
  libxt \
  ncurses \
  python3 \
  ruby \
  perl \
  php7

RUN apk add --update --no-cache \
  git \
  bash \
  fish \
  ctags \
  fzf \
  the_silver_searcher

RUN git clone --depth 1 https://github.com/junegunn/fzf.git ~/.fzf && ~/.fzf/install


ENTRYPOINT ["vim"]

I am utilizing multi-stage build for this. It’s to reduce the size of final image since it won’t have all the packages required for building vim like gcc and various libraries and their headers. Also, if you take out the last 2 RUN statements and also remove ruby, perl and php7 packages, your vim image (uncompressed) will only be around 30MB!

Remove --enable-<language>interp from the build stage and the language package from the last stage if you don’t want them. Chances are you won’t need them if you don’t already know what they are for.

The second-last RUN will install additional stuff that I need. the_silver_searcher is for using :Ag in vim and fzf is for lots of useful commands that I use many times a day. The last RUN will clone fzf vim plugin and run the install script.

Use docker build -t vim-base . to build and you will have a pretty capable vim. Run docker run --rm vim-base "--version" to see what features have been compiled with vim.

Now that I have base vim, it’s time to bake in some plugins. I’ve been using the same plugins for some time now and I don’t think they’re going to change anytime soon, so I decided to put the plugins into my final vim image. Check out Dockerfile.plugins:

FROM asyazwan/vim

ARG USERNAME=syazwan
ARG GROUPNAME=syazwan
ARG WORKSPACE=/home/syazwan
ARG UID=1000
ARG GID=1000
ARG SHELL=/bin/sh

RUN apk add --no-cache sudo \
  && mkdir -p "${WORKSPACE}" \
  && echo "${USERNAME}:x:${UID}:${GID}:${USERNAME},,,:${WORKSPACE}:${SHELL}" \
  >> /etc/passwd \
  && echo "${USERNAME}::17032:0:99999:7:::" \
  >> /etc/shadow \
  && echo "${USERNAME} ALL=(ALL) NOPASSWD: ALL" \
  > "/etc/sudoers.d/${USERNAME}" \
  && chmod 0440 "/etc/sudoers.d/${USERNAME}" \
  && echo "${GROUPNAME}:x:${GID}:${USERNAME}" >> /etc/group \
  && chown "${UID}":"${GID}" "${WORKSPACE}"

WORKDIR $WORKSPACE

USER $USERNAME

COPY plug.vim ${WORKSPACE}/.vim/autoload/plug.vim
COPY plugged ${WORKSPACE}/.vim/plugged/
COPY vimrc ${WORKSPACE}/.vimrc
COPY fzf-default.sh ${WORKSPACE}/.vim/fzf-default
RUN sudo mv /root/.fzf ${WORKSPACE}/.vim/

RUN sudo chown -R "${UID}":"${GID}" .vimrc .vim/
RUN ~/.vim/.fzf/install
RUN mkdir -p .vim/backups .vim/undo
ENV FZF_DEFAULT_COMMAND ~/.vim/fzf-default

ENTRYPOINT ["/usr/bin/fish"]

Let’s break down what is going on here. From the base vim image, I create user account the same as I have on the current machine, which can be overriden via --build-arg. This is to avoid possible file permission/ownership issues. Then I copy my plugin manager Plug, the plugins, vimrc file, and fzf files. Entrypoint is set to my current favorite terminal.

To build:

docker build -f Dockerfile.plugins -t vim \
  --build-arg USERNAME=$(id -un) \
  --build-arg GROUPNAME=$(id -gn) \
  --build-arg UID=$(id -u) \
  --build-arg GID=$(id -g) \
  --build-arg WORKSPACE=$$HOME \
  --build-arg SHELL=/usr/bin/fish .

Then to run:

docker run --rm -it --hostname vim -v "$PWD":$HOME/dev/ vim

Which will bring you into fish terminal and $HOME/dev/ workspace. Invoke vim with vim ..

That’s the idea, basically. From any machine with docker I can just clone the repo and build Dockerfile.plugins to get my familiar development environment.

Please share if you have suggestions on how to improve it.

Earlier versions of Postgres only allows read-only data. As of Postgres 9.3, writing is available so if you are doing write operations take note of that.

My use case was this: I wanted to transfer gigabytes of data from provider A in Singapore region to another provider B in EU region. These providers provide DB access only, no OS-level access. So it wasn’t possible to dump as usual and there’s the issue of size too. To make my workstation the intermediate node (import from A - export to B) would also be a huge waste of bandwidth and space (about 400 GB total). I also don’t have the best internet connection. Enter FDW.

First create the extension:

CREATE EXTENSION postgres_fdw;

Then create the connection:

CREATE SERVER myremotedb
FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (host 'myremotedb.providera.com', port '5432', dbname 'mydb', sslmode 'require', fetch_size '250000');

Important gotcha: find the right fetch_size for you. The default is 100 rows and extremely small for tables with millions of rows thus can cause big overhead depending on latency between the two servers.

You may exclude or include any server options as you require.

Now that we have the server, next we declare the user mapping:

CREATE USER MAPPING FOR newuser1
SERVER myremotedb
OPTIONS (user 'olduser1', password 'password1');

Lastly, the schema:

CREATE SCHEMA oldschema;

IMPORT FOREIGN SCHEMA public FROM SERVER myremotedb INTO oldschema;

Now you can query the remote database from the new one as if it’s a local db:

SELECT * FROM oldschema.table1;

-- Migrate: (obviously their structure must be the same)
INSERT INTO public.table1 SELECT * FROM oldschema.table1;

Results

FDW is usually not the most performant way to do things so it’s rarely the first thing I look at. But here it proved to be useful since my migration wasn’t so time-critical. With this setup I was able to move over 500k rows, 250 MB data in 30s (as opposed to 20m with default fetch_size!). Moving 2.7m rows, 8 GB data took 80 minutes, which is not so bad.

However, if intermediate node is a non-issue for you, it’s better to use it. In my initial tests using \copy to dump CSV did over 1k rows/s on average, which is much faster than FDW. But of course that’s exluding importing the dump to the new db.

Recommended reads:

• List of FDW (More on PGXN)
• Intro to Postgres FDW

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
linuxkit-025000000001:~# uname -a
Linux linuxkit-025000000001 4.9.75-linuxkit-aufs #1 SMP Tue Jan 9 10:58:17 UTC 2018 x86_64 Linux

But eventually you’ll discover /dev/log is missing. Apparently syslog is not running too.

So one way to get the log working again is to run syslog:

linuxkit-025000000001:~# syslogd

Which will create /dev/log socket and write to /var/log/messages by default and you can continue tailing that log file like before.

Happy logging!

There could be a lot of reasons. Use docker enough and eventually you’ll bump into situation where docker exec isn’t sufficient. This is just a cool trick to add to your docker swiss army knife collection.

The tool we will be using is called nsenter (NameSpaceENTER). The original docker image uses debian:jessie which is over 300 MB in size. If you don’t mind using someone’s binary, there is a standalone binary with statically compiled musl as the dependency and the whole thing is only 500 KB.

Get into container

Suppose you need to use a container’s network (eg. to access non-exposed services) or debug the process itself (PID: 1):

# enter by container name
$ docker run --rm -it --privileged --net=container:mycontainer --pid=container:mycontainer  jpetazzo/nsenter -t 1 -m -u -i -n sh
# enter by container id
$ docker run --rm -it --privileged --net=container:21ee7ab8ddd7 --pid=container:21ee7ab8ddd7  jpetazzo/nsenter -t 1 -m -u -i -n sh

The equivalent* with official ways docker exec / docker run:

$ docker exec -it --privileged --net=container:mycontainer --pid=container:mycontainer bash
# use minimal privelege instead of --privileged because strace only need these 2
$ docker run --rm -it --cap-add sys_admin --cap-add sys_ptrace --net=container:mycontainer --pid=container:mycontainer myimage/strace strace

Get into docker host / Docker for Mac

Suppose you want to see docker’s syslog or debug docker host itself:

$ docker run --rm -it --privileged --pid=host --net=host jpetazzo/nsenter -t 1 -m -u -i -n sh

When docker ditched Alpine for xhyve, I didn’t know how to get into docker itself to eg. see logs.

The xhyve hypervisor is a port of bhyve to OS X. It is built on top of Hypervisor.framework in OS X 10.10 Yosemite and higher, runs entirely in userspace, and has no other dependencies.

So it’s a lightweight hypervisor that runs Linux with Docker Engine inside. If you can’t find your logs you need via docker logs (what’s sent to STDOUT), chances are it’s in the docker daemon logs (sent for example via syslog facilities).

This is the magic:

$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

I will spare the explanation of what screen is but it comes bundled with OSX. The trick is just knowing where the tty to attach to is at.

Sometimes you might have been disconnected or you exit the terminal using ^d. This actually detach from the terminal. What this means is that the next time you run the command again, you will see weird artifacts like crazy text indentation and gibberish characters. Restarting docker will of course fix this as the Linux effectively reboots.

Or you can be civilized and just reattach (assuming you only have one session running):

$ screen -R

If you have multiple sessions, find your docker session:

$ screen -ls
There is a screen on:
	48617.ttys019.syazwan	(Detached)
1 Socket in /var/folders/_4/gl_zvn7x4y7g8m66vwwvz4nr0000gn/T/.screen.

Then reattach with the correct <pid>.<tty>.<host>:

$ screen -r 48617.ttys019.syazwan

Happy logging!