Tuesday, November 16, 2021

macOS: Docker volumes and Minikube

This morning I decided to replace Docker Desktop on my mac with the Hyperkit + Minikube combo like explained in Run Docker without Docker Desktop on macOS.

Volumes

However, some file mounted through a volume in docker-compose invariably resulted in a directory being created in the container instead of a file.
docker-compose.yaml

version: "3.8"
services:
  postgres:
    container_name: oss-postgres
    image: postgres:14-alpine
    stdin_open: true
    tty: true
    restart: always
    ports:
      - 5432:5432
    volumes:
      - ./postgres/postgres.sh:/docker-entrypoint-initdb.d/postgres.sh
    environment:
      POSTGRES_PASSWORD: "xxx"
logs
oss-postgres  | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/postgres.sh
oss-postgres  | /usr/local/bin/docker-entrypoint.sh: line 169: /docker-entrypoint-initdb.d/postgres.sh: Is a directory
oss-postgres exited with code 126
It appears that the volumes need being created in the Minikube VM beforehand (source: Replace Docker Desktop with Minikube and Hyperkit on macOS), like in:
> minikube stop
> minikube start --mount --mount-string="/Users/jerome/src/boss/:/Users/jerome/src/boss/" --memory 4096 --cpus 4

Ports

Any service previously reachable through localhost by port forwarding is now available by using docker.dev, mapped to minikube ip in /etc/hosts. 

NOTE

Never use .local domain on macOS, or DNS resolving will become awfully slow: use for ex. docker.dev instead of docker.local for the hostname of the minikube ip, contrary to what the 1st article proposes.

Wednesday, September 15, 2021

Elixir: convert map to keywords list

iex(20)> Enum.reduce(%{"code" => 33, "kind" => 2, "servid" => 13}, [], fn {k, v}, acc -> acc ++ [{String.to_atom(k), v}] end)
[code: 33, kind: 2, servid: 13]
or
iex(23)> for {k, v} <- %{"code" => 33, "kind" => 2, "servid" => 13}, do: {String.to_atom(k), v}
[code: 33, kind: 2, servid: 13]

Tuesday, September 7, 2021

elixir-ls: /usr/local/bin/elixir: line 231: exec: erl: not found

Source: Elixir on Emacs: exec erl not found
If after installing elixir-ls in emacs, you get this error reported in *elixir-ls::stderr* buffer, then install exec-path-from-shell by using M-x package-list-packages.

Thursday, March 4, 2021

PostgreSQL: Get a column value corresponding to the min of another column

Purpose: Get a column value corresponding to the min of another column (i.e. both values must belong to the same row)
E.g. Let's say we have columns a,b,c,b. We want a,b matching min(c) while partitioning by d.
 

crmmbqt=# create table demo (a int, b int, c int, d int);

 

crmmbqt=# insert into demo (a, b, c, d) values (1,2,1,1), (2,1,3,1), (3,4,5,2), (4,3,4,2);

INSERT 0 4

Time: 6.106 ms

crmmbqt=# select * from demo;

┌───┬───┬───┬───┐

│ a │ b │ c │ d │

├───┼───┼───┼───┤

│ 1 │ 2 │ 1 │ 1 │

│ 2 │ 1 │ 3 │ 1 │

│ 3 │ 4 │ 5 │ 2 │

│ 4 │ 3 │ 4 │ 2 │

└───┴───┴───┴───┘

(4 rows)


crmmbqt=# select first_value(a) over (partition by d order by c) as a, first_value(b) over (partition by d order by c) as b, c, d from demo;

┌───┬───┬───┬───┐

│ a │ b │ c │ d │

├───┼───┼───┼───┤

│ 1 │ 2 │ 1 │ 1 │

│ 1 │ 2 │ 3 │ 1 │

│ 4 │ 3 │ 4 │ 2 │

│ 4 │ 3 │ 5 │ 2 │

└───┴───┴───┴───┘

(4 rows)


NB: The use of first_value() is really important here.
For example, if you want to get max(c) instead of min(c), do not use last_value(a) but instead:
first_value(a) over (partition by d order by c desc)

crmmbqt=# select min(a) as a, min(b) as b, min(c), d from (select first_value(a) over (partition by d order by c) as a, first_value(b) over (partition by d order by c) as b, c, d from demo) as foo group by d;

┌───┬───┬─────┬───┐

│ a │ b │ min │ d │

├───┼───┼─────┼───┤

│ 1 │ 2 │ 1   │ 1 │

│ 4 │ 3 │ 4   │ 2 │

└───┴───┴─────┴───┘

(2 rows)


Monday, January 18, 2021

Git: Moving files to a new repository while preserving history

> git --version
git version 2.18.0
This method did not work for me: Move files from one repository to another, preserving git history
But this one did the trick: Moving Files and Directories to a New Repository in Git

Here is the list of commands used in order to extract the contents of the perl/ directory from the connectmv repo. into the ss-sms-vas repo.:
> mkdir cloneA
> cd cloneA/
> git clone git@github.com:<MY_TEAM>/connectmv.git
> cd connectmv/
> git subtree split --prefix perl/ --branch my-subtree
> git checkout my-subtree
> git remote add repoA git@github.com::<MY_TEAM>/ss-sms-vas.git
> git pull repoA main --allow-unrelated-histories

> git push -u repoA my-subtree:main
> cd ../../
> rm -rf cloneA/

> cd ss-sms-vas/
> git pull
> git lol lib/MBQT/mod_perl/AbstractVas.pm 
* 72be224f (Thu Jan 7 18:32:47 2021 Jerome G) PLTBUGS-8700: exploded USSD CDR per process
* 680a9ff2 (Thu Jan 7 13:51:45 2021 Jerome G) PLTBUGS-8700: removed unlock
...

Friday, December 11, 2020

postgreSQL: see trigger dependencies of an extension

Source: How to select functions that belong in a given extension in PostgreSQL?
The StackExchange question shows this is the request to use:
SELECT e.extname, ne.nspname AS extschema, p.proname, np.nspname AS proschema
FROM pg_catalog.pg_extension AS e
    INNER JOIN pg_catalog.pg_depend AS d ON (d.refobjid = e.oid)
    INNER JOIN pg_catalog.pg_proc AS p ON (p.oid = d.objid)
    INNER JOIN pg_catalog.pg_namespace AS ne ON (ne.oid = e.extnamespace)
    INNER JOIN pg_catalog.pg_namespace AS np ON (np.oid = p.pronamespace)
WHERE d.deptype = 'e'
ORDER BY 1, 3
which led me to use
SELECT e.extname, t.tgname
FROM pg_catalog.pg_extension AS e
JOIN pg_catalog.pg_depend AS d ON (d.refobjid = e.oid)
JOIN pg_catalog.pg_trigger AS t ON (t.tgfoid = d.objid)
where e.extname = 'mbqt_shared_api'

┌─────────────────┬─────────────────────────────────────┐
│     extname     │               tgname                │
├─────────────────┼─────────────────────────────────────┤
│ mbqt_shared_api │ dt_dialdigit_after_ins_trg          │
│ mbqt_shared_api │ dt_imsi_range_before_ins_or_upd_trg │
│ mbqt_shared_api │ dt_rzgthlr_after_ins_trg            │
│ mbqt_shared_api │ dt_rzgthlr_before_del_trg           │
│ mbqt_shared_api │ dt_rzgthlr_after_upd_trg            │
│ mbqt_shared_api │ dt_dialdigit_after_upd_trg          │
│ mbqt_shared_api │ dt_tadig_after_ins_trg              │
│ mbqt_shared_api │ dt_mccmnc_after_ins_trg             │
└─────────────────┴─────────────────────────────────────┘

Monday, November 23, 2020

macOS: Brew emacs 27.1 keeps crashing on Big Sur 11.0.1

[Edit] Updated way to compile emacs on macOS 12: https://github.com/jimeh/build-emacs-for-macos

Version: macOS v11.0.1
Constant crashes of Emacs v27.1 installed from Brew for no reason.
As a workaround, let's compile the latest version locally (bleeding edge).
source: Transitioning to GUI'd Emacs on macOS
$ git clone https://git.savannah.gnu.org/git/emacs.git
$ cd emacs
Check out the native compilation branch.
$ git checkout feature/native-comp
Let's make it.
$ export LIBXML2_CFLAGS="-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/libxml2"
$ ./autogen.sh
$ ./configure --without-makeinfo
$ make
$ make install
Then, replace the Brew version.
$ brew cask remove emacs
$ mv nextstep/Emacs.app/ /Applications/

Tuesday, November 10, 2020

jq: keep array values

 [jerome@jeroboam] > echo '["foo", "bar", "Baz"]' | jq '.[] | select(test("b"; "i"))'

"bar"

"Baz"


Wednesday, November 4, 2020

Clojure: destructuring map without naming key

Source: Destructuring map of map with unknown keys
user> (let [[[k {:keys [pim pam]}]] (seq {:basic {:pim 2 :pam 3}})] [k pim pam])
[:basic 2 3]
But no need of seq if we iterate with for.
user> (into {} (for [[k {:keys [pim pam]}] {:basic {:pim 2 :pam 3} :simple {:pim 20 :pam 30}}] [k {:pam pim :pim pam}]))
{:basic {:pam 2, :pim 3}, :simple {:pam 20, :pim 30}}

Friday, September 4, 2020

MacOS: unzip failure: need PK compat. v5.1 (can do v4.5)

 [jerome@jeroboam] > unzip CONNECTSA-565.zip 

Archive:  CONNECTSA-565.zip

   skipping: MBQT01186.OUT           need PK compat. v5.1 (can do v4.5)

   skipping: MBQT01191.OUT           need PK compat. v5.1 (can do v4.5)

   skipping: MBQT01197.OUT           need PK compat. v5.1 (can do v4.5)


[jerome@jeroboam] > brew install p7zip

[jerome@jeroboam] > 7za x CONNECTSA-565.zip 


Friday, August 14, 2020

MacOS: create a bootable Windows 10 install USB drive

Sources: 

Identify USB device

jerome@centurion ~ % diskutil list

This shows the USB drive is using /dev/disk4 in my case.

Format USB drive

jerome@centurion ~ % diskutil eraseDisk MS-DOS "WIN10" MBR /dev/disk4

NB: It is crucial to use MBR and not GPT in order to avoid the infamous error "Windows could not prepare the computer to boot into the next phase of installation. To install Windows, restart the installation." during setup on my laptop after that.

Mount Windows 10 ISO

jerome@centurion ~ % hdiutil mount ~/Desktop/Windows10.iso

This will create /Volumes/CCCOMA_X64FRE_EN-US_DV9

Copy Windows files to USB drive

One file of the installation, install.esd, is more than 4 GB, which is not supported on FAT32.

So, copy all files but this one.

jerome@centurion ~ % rsync -vha --exclude=sources/install.esd /Volumes/CCCOMA_X64FRE_EN-US_DV9/ /Volumes/WIN10

In order to copy install.esd, we will have to split it with wimlib, so install wimlib first.

jerome@centurion ~ % brew install wimlib                                                              

Then, split install.esd into 2 files of less than 4 GB, and copy them to the USB volume.

jerome@centurion ~ % wimlib-imagex split /Volumes/CCCOMA_X64FRE_EN-US_DV9/sources/install.esd /Volumes/WIN10/sources/install.swm 4000
[ERROR] Splitting of WIM containing solid resources is not supported.
        Export it in non-solid format first.
ERROR: Exiting with error code 68:
       The requested operation is unsupported.

install.esd is actually a solid archive compressed with LZMS, and apparently it cannot be split. So, export it to non-solid WIM as advised. For this, follow WIMEXPORT

jerome@centurion ~ % wimexport 
/Volumes/CCCOMA_X64FRE_EN-US_DV9/sources/install.esd all 
~/Desktop/install.wim --compress=LZX            
Using LZX compression with 12 threads
Archiving file data: 13 GiB of 13 GiB (100%) done

 Then try splitting again.

jerome@centurion ~ % wimlib-imagex split ~/Desktop/install.wim /Volumes/WIN10/sources/install.swm 4000
Writing "/Volumes/WIN10/sources/install.swm" (part 1 of 2): 0 MiB of 6037 MiB (0%) written
Writing "/Volumes/WIN10/sources/install2.swm" (part 2 of 2): 3997 MiB of 6037 MiB (66%) written
Finished writing split WIM part 2 of 2
 

Then eject WIN10, and you're good to go to install Windows 10 by booting from the USB drive.


Friday, July 17, 2020

Git: view unpushed commits

Source: Viewing unpushed Git commits
[jerome@jeroboam] > git lol @{u}..
* 00e9d99cf (HEAD -> PLTFRS-12002_Move_USSD_VAS_to_AWS) (Mon Jun 1 12:00:14 2020 GitHub) Merge pull request #726 from AirVantage/PLTFRS-12002_Move_USSD_VAS_to_AWS

Monday, June 22, 2020

MacOS: How to test netcat

Source: How To Use Netcat to Establish and Test TCP and UDP Connections on a VPS
Terminal #1:
[jerome@jeroboam] > nc -l 4444
Terminal #2:
[jerome@jeroboam] > nc localhost 4444
toto

Then terminal #1 will display toto as well.

Monday, June 15, 2020

PgBouncer: WARNING DNS lookup failed: host.docker.internal: result=0

If you're using PgBouncer in a Docker container and try to connect to the host via host.docker.internal, you may experience this error:
2020-06-15 17:23:46.211 UTC [1] WARNING DNS lookup failed: host.docker.internal: result=0
This fix resolves it: Fix DNS for Docker #8

Monday, June 1, 2020

PostgreSQL: plpgsql_check

Source: plpgsql_check
Cannot get it to compile from Github sources on macOS 10.15.5.
So, retrieve it from pgxn.
Then compile and install it.
[jerome@jeroboam] > make
clang -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2  -undefined dynamic_lookup -I/usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../src/pl/plpgsql/src -I. -I./ -I/usr/local/Cellar/postgresql@9.4/9.4.26/include/server -I/usr/local/Cellar/postgresql@9.4/9.4.26/include/internal -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk   -I/usr/local/opt/gettext/include -I/usr/local/opt/openldap/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/tcl-tk/include  -c -o plpgsql_check.o plpgsql_check.c
clang -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2  -undefined dynamic_lookup -I/usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../src/pl/plpgsql/src -bundle -multiply_defined suppress -o plpgsql_check.so plpgsql_check.o -L/usr/local/Cellar/postgresql@9.4/9.4.26/lib    -L/usr/local/opt/gettext/lib -L/usr/local/opt/openldap/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/tcl-tk/lib -Wl,-dead_strip_dylibs   -bundle_loader /usr/local/Cellar/postgresql@9.4/9.4.26/bin/postgres
[jerome@jeroboam] > make install
/bin/sh /usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql@9.4/9.4.26/lib'
/bin/sh /usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql@9.4/9.4.26/share/extension'
/bin/sh /usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql@9.4/9.4.26/share/extension'
/usr/bin/install -c -m 755  plpgsql_check.so '/usr/local/Cellar/postgresql@9.4/9.4.26/lib/plpgsql_check.so'
/usr/bin/install -c -m 644 plpgsql_check.control '/usr/local/Cellar/postgresql@9.4/9.4.26/share/extension/'
/usr/bin/install -c -m 644 plpgsql_check--1.0.sql '/usr/local/Cellar/postgresql@9.4/9.4.26/share/extension/'
[jerome@jeroboam] > 
Then from psql:
crmmbqt=# CREATE EXTENSION plpgsql_check;
CREATE EXTENSION
Time: 317.716 ms

PostgreSQL: ERROR: could not access file "$libdir/plpython2": No such file or directory

After upgrading postgres from 9.4.18 to 9.4.26 on MacOS 10.15.5 with
[jerome@jeroboam] > brew upgrade 'postgresql@9.4'
I then discovered that plpython was gone.
[jerome@jeroboam] > make
touch sql/mbqt_post_api.out
../bin/replace_extension.pl sql/mbqt_post_api.sql
DBG> schema = mbqt_db, extension = mbqt_post_api
DBG> installed version = 1.21, new version = 1.21
CMD> cat sql/mbqt_post_api.sql > sql/mbqt_post_api--1.21.sql
CMD> /usr/bin/install -m 644 sql/mbqt_post_api--1.21.sql sql/mbqt_post_api.control /usr/local/Cellar/postgresql@9.4/9.4.26/share/postgresql@9.4/extension
CMD> psql -Upostgres -h127.0.0.1 -p5432 -dcrmmbqt -q<<END_OF_SQL
\set ON_ERROR_STOP ON
create schema if not exists mbqt_db authorization postgres;
drop extension if exists mbqt_post_api cascade;
create extension mbqt_post_api with schema mbqt_db;
END_OF_SQL
2020-06-01 14:12:47
NOTICE:  schema "mbqt_db" already exists, skipping
Time: 11.778 ms
NOTICE:  extension "mbqt_post_api" does not exist, skipping
Time: 12.900 ms
ERROR:  could not access file "$libdir/plpython2": No such file or directory
And indeed
[jerome@jeroboam] > ll -rt /usr/local/opt/postgresql\@9.4/lib/pl*
-r--r--r--  1 jerome  staff    42K Jun  1 12:41 /usr/local/opt/postgresql@9.4/lib/pltcl.so
-r--r--r--  1 jerome  staff   164K Jun  1 12:41 /usr/local/opt/postgresql@9.4/lib/plpgsql.so
-r--r--r--  1 jerome  staff    84K Jun  1 12:41 /usr/local/opt/postgresql@9.4/lib/plperl.so
I tried
[jerome@jeroboam] > brew reinstall postgresql@9.4 --with-python
But it failed with
Error: invalid option: --with-python
And indeed, as per Remove all options from Homebrew/homebrew-core formulae, they estimated that options in brew are bad from now.
Then I found out: petere / homebrew-postgresql.
So, I used
[jerome@jeroboam] > brew tap petere/postgresql
And
[jerome@jeroboam] > brew reinstall petere/postgresql/postgresql@9.4
which ended up with
configure: error: header file <perl.h> is required for Perl

If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
  https://github.com/petere/homebrew-postgresql/issues


Error: A newer Command Line Tools release is available.
Update them from Software Update in System Preferences or run:
  softwareupdate --all --install --force

If that doesn't show you an update run:
  sudo rm -rf /Library/Developer/CommandLineTools
  sudo xcode-select --install
Then let's go
[jerome@jeroboam] > sudo rm -rf /Library/Developer/CommandLineTools
[jerome@jeroboam] > sudo xcode-select --install
And finally
[jerome@jeroboam] > brew reinstall petere/postgresql/postgresql@9.4
which ended OK this time

==> Summary
🍺  /usr/local/Cellar/postgresql@9.4/9.4.26: 2,989 files, 41.0MB, built in 2 minutes 8 seconds
And guess what? plpython is back.
[jerome@jeroboam] > ll -rt /usr/local/opt/postgresql\@9.4/lib/pl*
-r--r--r--  1 jerome  staff    42K Jun  1 15:53 /usr/local/opt/postgresql@9.4/lib/pltcl.so
-r--r--r--  1 jerome  staff   108K Jun  1 15:53 /usr/local/opt/postgresql@9.4/lib/plpython2.so
-r--r--r--  1 jerome  staff   164K Jun  1 15:53 /usr/local/opt/postgresql@9.4/lib/plpgsql.so
-r--r--r--  1 jerome  staff    84K Jun  1 15:53 /usr/local/opt/postgresql@9.4/lib/plperl.so

Thursday, May 28, 2020

Linux: list SSL certificates

Source: List all available ssl ca certificates
locate .pem | grep "\.pem$"

Perl SSL: get debug details when it's not working

When trying to understand why LWP::UserAgent cannot post to HTTPS, you can try:
use IO::Socket::SSL qw(debug4);
so as to get more details.

Monday, May 25, 2020

postgreSQL: streaming replication delay

From the standby DB:
crmmbqt=# select now() - pg_last_xact_replay_timestamp() as replication_delay;
┌───────────────────────┐
│   replication_delay   │
├───────────────────────┤
│ 1 day 05:45:26.277667 │
└───────────────────────┘
or get more details with
crmmbqt=# pg_last_xlog_receive_location(), pg_last_xlog_replay_location(), pg_last_xact_replay_timestamp(), case when pg_last_xlog_receive_location() = pg_last_xlog_replay_location() then 0 else extract(epoch from now() - pg_last_xact_replay_timestamp()) end as replication_delay;