Elixir: rename_keys

Elixir equivalent of Clojure rename_keys()

  @doc """
  Returns the map with the keys in kmap renamed to the values in kmap.
  """
  def rename_keys(map, kmap) when is_map(map) and is_map(kmap) do
    for {key, value} <- map,
        into: %{},
        do: if(new_key = Map.get(kmap, key), do: {new_key, value}, else: {key, value})
  end

Elixir: if key exists: update, otherwise: assoc

Remember Clojure: if key exists: update, otherwise: assoc? Let's do that with Elixir.

  @doc """
  If `key` exists in `map`, updates its value (assumed to be a list) by concatenating `value`.
  If `key` does not exist in `map`, creates `{key => [value]}` in `map`.
  ## Examples

    iex(35)> Tap.Worker.update_conj!(%{a: [1]}, :a, 2)
    %{a: [1, 2]}

    iex(36)> Tap.Worker.update_conj!(%{a: [1]}, :b, 2)
    %{a: [1], b: [2]}

  """
  @spec update_conj!(map, any, any) :: map
  def update_conj!(map, key, value) when is_map(map) do
    Map.update(map, key, [value], fn
      existing_values when is_list(existing_values) ->
        existing_values ++ [value]

      existing_value ->
        raise(ArgumentError, "Value is not a list: #{inspect(existing_value)}")
    end)
  end

Elixir: map to struct

Courtesy of Jose Valim: (https://groups.google.com/forum/#!msg/elixir-lang-talk/6geXOLUeIpI/L9einu4EEAAJ).

  def to_struct(kind, attrs) do
    struct = struct(kind)

    Logger.debug("attrs = #{inspect(attrs)}")

    Enum.reduce(Map.to_list(struct), struct, fn {k, _}, acc ->
      case Map.fetch(attrs, k) do
        {:ok, v} -> %{acc | k => v}
        :error -> acc
      end
    end)
  end

macOS: Docker volumes and Minikube

This morning I decided to replace Docker Desktop on my mac with the Hyperkit + Minikube combo like explained in Run Docker without Docker Desktop on macOS.

Volumes

However, some file mounted through a volume in docker-compose invariably resulted in a directory being created in the container instead of a file.
docker-compose.yaml

version: "3.8"
services:
  postgres:
    container_name: oss-postgres
    image: postgres:14-alpine
    stdin_open: true
    tty: true
    restart: always
    ports:
      - 5432:5432
    volumes:
      - ./postgres/postgres.sh:/docker-entrypoint-initdb.d/postgres.sh
    environment:
      POSTGRES_PASSWORD: "xxx"

logs

oss-postgres  | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/postgres.sh
oss-postgres  | /usr/local/bin/docker-entrypoint.sh: line 169: /docker-entrypoint-initdb.d/postgres.sh: Is a directory
oss-postgres exited with code 126

It appears that the volumes need being created in the Minikube VM beforehand (source: Replace Docker Desktop with Minikube and Hyperkit on macOS), like in:

> minikube stop
> minikube start --mount --mount-string="/Users/jerome/src/boss/:/Users/jerome/src/boss/" --memory 4096 --cpus 4

Ports

Any service previously reachable through localhost by port forwarding is now available by using docker.dev, mapped to minikube ip in /etc/hosts. 

NOTE

Never use .local domain on macOS, or DNS resolving will become awfully slow: use for ex. docker.dev instead of docker.local for the hostname of the minikube ip, contrary to what the 1st article proposes.

Elixir: convert map to keywords list

iex(20)> Enum.reduce(%{"code" => 33, "kind" => 2, "servid" => 13}, [], fn {k, v}, acc -> acc ++ [{String.to_atom(k), v}] end)
[code: 33, kind: 2, servid: 13]

or

iex(23)> for {k, v} <- %{"code" => 33, "kind" => 2, "servid" => 13}, do: {String.to_atom(k), v}
[code: 33, kind: 2, servid: 13]

elixir-ls: /usr/local/bin/elixir: line 231: exec: erl: not found

Source: Elixir on Emacs: exec erl not found
If after installing elixir-ls in emacs, you get this error reported in *elixir-ls::stderr* buffer, then install exec-path-from-shell by using M-x package-list-packages.

PostgreSQL: Get a column value corresponding to the min of another column

Source: Understanding Window Functions

Purpose: Get a column value corresponding to the min of another column (i.e. both values must belong to the same row)
E.g. Let's say we have columns a,b,c,b. We want a,b matching min(c) while partitioning by d.

 

crmmbqt=# create table demo (a int, b int, c int, d int);

 

crmmbqt=# insert into demo (a, b, c, d) values (1,2,1,1), (2,1,3,1), (3,4,5,2), (4,3,4,2);

INSERT 0 4

Time: 6.106 ms

crmmbqt=# select * from demo;

┌───┬───┬───┬───┐

│ a │ b │ c │ d │

├───┼───┼───┼───┤

│ 1 │ 2 │ 1 │ 1 │

│ 2 │ 1 │ 3 │ 1 │

│ 3 │ 4 │ 5 │ 2 │

│ 4 │ 3 │ 4 │ 2 │

└───┴───┴───┴───┘

(4 rows)

crmmbqt=# select first_value(a) over (partition by d order by c) as a, first_value(b) over (partition by d order by c) as b, c, d from demo;

┌───┬───┬───┬───┐

│ a │ b │ c │ d │

├───┼───┼───┼───┤

│ 1 │ 2 │ 1 │ 1 │

│ 1 │ 2 │ 3 │ 1 │

│ 4 │ 3 │ 4 │ 2 │

│ 4 │ 3 │ 5 │ 2 │

└───┴───┴───┴───┘

(4 rows)

NB: The use of first_value() is really important here.
For example, if you want to get max(c) instead of min(c), do not use last_value(a) but instead:

first_value(a) over (partition by d order by c desc)

crmmbqt=# select min(a) as a, min(b) as b, min(c), d from (select first_value(a) over (partition by d order by c) as a, first_value(b) over (partition by d order by c) as b, c, d from demo) as foo group by d;

┌───┬───┬─────┬───┐

│ a │ b │ min │ d │

├───┼───┼─────┼───┤

│ 1 │ 2 │ 1   │ 1 │

│ 4 │ 3 │ 4   │ 2 │

└───┴───┴─────┴───┘

(2 rows)

Git: Moving files to a new repository while preserving history

> git --version
git version 2.18.0

This method did not work for me: Move files from one repository to another, preserving git history
But this one did the trick: Moving Files and Directories to a New Repository in Git

Here is the list of commands used in order to extract the contents of the perl/ directory from the connectmv repo. into the ss-sms-vas repo.:

> mkdir cloneA
> cd cloneA/
> git clone git@github.com:<MY_TEAM>/connectmv.git
> cd connectmv/
> git subtree split --prefix perl/ --branch my-subtree
> git checkout my-subtree
> git remote add repoA git@github.com::<MY_TEAM>/ss-sms-vas.git
> git pull repoA main --allow-unrelated-histories

> git push -u repoA my-subtree:main
> cd ../../
> rm -rf cloneA/

> cd ss-sms-vas/
> git pull
> git lol lib/MBQT/mod_perl/AbstractVas.pm 
* 72be224f (Thu Jan 7 18:32:47 2021 Jerome G) PLTBUGS-8700: exploded USSD CDR per process
* 680a9ff2 (Thu Jan 7 13:51:45 2021 Jerome G) PLTBUGS-8700: removed unlock
...

postgreSQL: see trigger dependencies of an extension

Source: How to select functions that belong in a given extension in PostgreSQL?
The StackExchange question shows this is the request to use:

SELECT e.extname, ne.nspname AS extschema, p.proname, np.nspname AS proschema
FROM pg_catalog.pg_extension AS e
    INNER JOIN pg_catalog.pg_depend AS d ON (d.refobjid = e.oid)
    INNER JOIN pg_catalog.pg_proc AS p ON (p.oid = d.objid)
    INNER JOIN pg_catalog.pg_namespace AS ne ON (ne.oid = e.extnamespace)
    INNER JOIN pg_catalog.pg_namespace AS np ON (np.oid = p.pronamespace)
WHERE d.deptype = 'e'
ORDER BY 1, 3

which led me to use

SELECT e.extname, t.tgname
FROM pg_catalog.pg_extension AS e
JOIN pg_catalog.pg_depend AS d ON (d.refobjid = e.oid)
JOIN pg_catalog.pg_trigger AS t ON (t.tgfoid = d.objid)
where e.extname = 'mbqt_shared_api'

┌─────────────────┬─────────────────────────────────────┐
│     extname     │               tgname                │
├─────────────────┼─────────────────────────────────────┤
│ mbqt_shared_api │ dt_dialdigit_after_ins_trg          │
│ mbqt_shared_api │ dt_imsi_range_before_ins_or_upd_trg │
│ mbqt_shared_api │ dt_rzgthlr_after_ins_trg            │
│ mbqt_shared_api │ dt_rzgthlr_before_del_trg           │
│ mbqt_shared_api │ dt_rzgthlr_after_upd_trg            │
│ mbqt_shared_api │ dt_dialdigit_after_upd_trg          │
│ mbqt_shared_api │ dt_tadig_after_ins_trg              │
│ mbqt_shared_api │ dt_mccmnc_after_ins_trg             │
└─────────────────┴─────────────────────────────────────┘

postgreSQL: gist

Source: What does exclusion constraint `EXCLUDE USING gist (c WITH &&)` mean?
See PG Geometric Functions and Operators.

macOS: Brew emacs 27.1 keeps crashing on Big Sur 11.0.1

[Edit] Updated way to compile emacs on macOS 12: https://github.com/jimeh/build-emacs-for-macos

Version: macOS v11.0.1
Constant crashes of Emacs v27.1 installed from Brew for no reason.
As a workaround, let's compile the latest version locally (bleeding edge).
source: Transitioning to GUI'd Emacs on macOS

$ git clone https://git.savannah.gnu.org/git/emacs.git
$ cd emacs

Check out the native compilation branch.

$ git checkout feature/native-comp

Let's make it.

$ export LIBXML2_CFLAGS="-I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/libxml2"
$ ./autogen.sh
$ ./configure --without-makeinfo
$ make
$ make install

Then, replace the Brew version.

$ brew cask remove emacs
$ mv nextstep/Emacs.app/ /Applications/

jq: keep array values

 [jerome@jeroboam] > echo '["foo", "bar", "Baz"]' | jq '.[] | select(test("b"; "i"))'

"bar"

"Baz"

Clojure: destructuring map without naming key

Source: Destructuring map of map with unknown keys

user> (let [[[k {:keys [pim pam]}]] (seq {:basic {:pim 2 :pam 3}})] [k pim pam])
[:basic 2 3]

But no need of seq if we iterate with for.

user> (into {} (for [[k {:keys [pim pam]}] {:basic {:pim 2 :pam 3} :simple {:pim 20 :pam 30}}] [k {:pam pim :pim pam}]))
{:basic {:pam 2, :pim 3}, :simple {:pam 20, :pim 30}}

MacOS: unzip failure: need PK compat. v5.1 (can do v4.5)

 [jerome@jeroboam] > unzip CONNECTSA-565.zip 

Archive:  CONNECTSA-565.zip

   skipping: MBQT01186.OUT           need PK compat. v5.1 (can do v4.5)

   skipping: MBQT01191.OUT           need PK compat. v5.1 (can do v4.5)

   skipping: MBQT01197.OUT           need PK compat. v5.1 (can do v4.5)

[jerome@jeroboam] > brew install p7zip

[jerome@jeroboam] > 7za x CONNECTSA-565.zip 

MacOS: create a bootable Windows 10 install USB drive

Sources: 

• How to Make a Windows 10 USB Using Your Mac
• Make a bootable Windows 10 USB drive from a Mac  
• Windows 10 ISO May 2020, Version 2004 Build 19041.264
• USB drive 64 GB
  

Identify USB device

jerome@centurion ~ % diskutil list

This shows the USB drive is using /dev/disk4 in my case.

Format USB drive

jerome@centurion ~ % diskutil eraseDisk MS-DOS "WIN10" MBR /dev/disk4

NB: It is crucial to use MBR and not GPT in order to avoid the infamous error "Windows could not prepare the computer to boot into the next phase of installation. To install Windows, restart the installation." during setup on my laptop after that.

Mount Windows 10 ISO

jerome@centurion ~ % hdiutil mount ~/Desktop/Windows10.iso

This will create /Volumes/CCCOMA_X64FRE_EN-US_DV9

Copy Windows files to USB drive

One file of the installation, install.esd, is more than 4 GB, which is not supported on FAT32.

So, copy all files but this one.

jerome@centurion ~ % rsync -vha --exclude=sources/install.esd /Volumes/CCCOMA_X64FRE_EN-US_DV9/ /Volumes/WIN10

In order to copy install.esd, we will have to split it with wimlib, so install wimlib first.

jerome@centurion ~ % brew install wimlib                                                              

Then, split install.esd into 2 files of less than 4 GB, and copy them to the USB volume.

jerome@centurion ~ % wimlib-imagex split /Volumes/CCCOMA_X64FRE_EN-US_DV9/sources/install.esd /Volumes/WIN10/sources/install.swm 4000
[ERROR] Splitting of WIM containing solid resources is not supported.
        Export it in non-solid format first.
ERROR: Exiting with error code 68:
       The requested operation is unsupported.

install.esd is actually a solid archive compressed with LZMS, and apparently it cannot be split. So, export it to non-solid WIM as advised. For this, follow WIMEXPORT

jerome@centurion ~ % wimexport 
/Volumes/CCCOMA_X64FRE_EN-US_DV9/sources/install.esd all 
~/Desktop/install.wim --compress=LZX            
Using LZX compression with 12 threads
Archiving file data: 13 GiB of 13 GiB (100%) done

 Then try splitting again.

jerome@centurion ~ % wimlib-imagex split ~/Desktop/install.wim /Volumes/WIN10/sources/install.swm 4000
Writing "/Volumes/WIN10/sources/install.swm" (part 1 of 2): 0 MiB of 6037 MiB (0%) written
Writing "/Volumes/WIN10/sources/install2.swm" (part 2 of 2): 3997 MiB of 6037 MiB (66%) written
Finished writing split WIM part 2 of 2
 

Then eject WIN10, and you're good to go to install Windows 10 by booting from the USB drive.

Git: view unpushed commits

Source: Viewing unpushed Git commits

[jerome@jeroboam] > git lol @{u}..
* 00e9d99cf (HEAD -> PLTFRS-12002_Move_USSD_VAS_to_AWS) (Mon Jun 1 12:00:14 2020 GitHub) Merge pull request #726 from AirVantage/PLTFRS-12002_Move_USSD_VAS_to_AWS

MacOS: How to test netcat

Source: How To Use Netcat to Establish and Test TCP and UDP Connections on a VPS
Terminal #1:

[jerome@jeroboam] > nc -l 4444

Terminal #2:

[jerome@jeroboam] > nc localhost 4444
toto

Then terminal #1 will display toto as well.

PgBouncer: WARNING DNS lookup failed: host.docker.internal: result=0

If you're using PgBouncer in a Docker container and try to connect to the host via host.docker.internal, you may experience this error:

2020-06-15 17:23:46.211 UTC [1] WARNING DNS lookup failed: host.docker.internal: result=0

This fix resolves it: Fix DNS for Docker #8

PostgreSQL: plpgsql_check

Source: plpgsql_check
Cannot get it to compile from Github sources on macOS 10.15.5.
So, retrieve it from pgxn.
Then compile and install it.

[jerome@jeroboam] > make
clang -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2  -undefined dynamic_lookup -I/usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../src/pl/plpgsql/src -I. -I./ -I/usr/local/Cellar/postgresql@9.4/9.4.26/include/server -I/usr/local/Cellar/postgresql@9.4/9.4.26/include/internal -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk   -I/usr/local/opt/gettext/include -I/usr/local/opt/openldap/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/tcl-tk/include  -c -o plpgsql_check.o plpgsql_check.c
clang -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -O2  -undefined dynamic_lookup -I/usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../src/pl/plpgsql/src -bundle -multiply_defined suppress -o plpgsql_check.so plpgsql_check.o -L/usr/local/Cellar/postgresql@9.4/9.4.26/lib    -L/usr/local/opt/gettext/lib -L/usr/local/opt/openldap/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/tcl-tk/lib -Wl,-dead_strip_dylibs   -bundle_loader /usr/local/Cellar/postgresql@9.4/9.4.26/bin/postgres
[jerome@jeroboam] > make install
/bin/sh /usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql@9.4/9.4.26/lib'
/bin/sh /usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql@9.4/9.4.26/share/extension'
/bin/sh /usr/local/Cellar/postgresql@9.4/9.4.26/lib/pgxs/src/makefiles/../../config/install-sh -c -d '/usr/local/Cellar/postgresql@9.4/9.4.26/share/extension'
/usr/bin/install -c -m 755  plpgsql_check.so '/usr/local/Cellar/postgresql@9.4/9.4.26/lib/plpgsql_check.so'
/usr/bin/install -c -m 644 plpgsql_check.control '/usr/local/Cellar/postgresql@9.4/9.4.26/share/extension/'
/usr/bin/install -c -m 644 plpgsql_check--1.0.sql '/usr/local/Cellar/postgresql@9.4/9.4.26/share/extension/'
[jerome@jeroboam] > 

Then from psql:

crmmbqt=# CREATE EXTENSION plpgsql_check;
CREATE EXTENSION
Time: 317.716 ms

PostgreSQL: ERROR: could not access file "$libdir/plpython2": No such file or directory

After upgrading postgres from 9.4.18 to 9.4.26 on MacOS 10.15.5 with

[jerome@jeroboam] > brew upgrade 'postgresql@9.4'

I then discovered that plpython was gone.

[jerome@jeroboam] > make
touch sql/mbqt_post_api.out
../bin/replace_extension.pl sql/mbqt_post_api.sql
DBG> schema = mbqt_db, extension = mbqt_post_api
DBG> installed version = 1.21, new version = 1.21
CMD> cat sql/mbqt_post_api.sql > sql/mbqt_post_api--1.21.sql
CMD> /usr/bin/install -m 644 sql/mbqt_post_api--1.21.sql sql/mbqt_post_api.control /usr/local/Cellar/postgresql@9.4/9.4.26/share/postgresql@9.4/extension
CMD> psql -Upostgres -h127.0.0.1 -p5432 -dcrmmbqt -q<<END_OF_SQL
\set ON_ERROR_STOP ON
create schema if not exists mbqt_db authorization postgres;
drop extension if exists mbqt_post_api cascade;
create extension mbqt_post_api with schema mbqt_db;
END_OF_SQL
2020-06-01 14:12:47
NOTICE:  schema "mbqt_db" already exists, skipping
Time: 11.778 ms
NOTICE:  extension "mbqt_post_api" does not exist, skipping
Time: 12.900 ms
ERROR:  could not access file "$libdir/plpython2": No such file or directory

And indeed

[jerome@jeroboam] > ll -rt /usr/local/opt/postgresql\@9.4/lib/pl*
-r--r--r--  1 jerome  staff    42K Jun  1 12:41 /usr/local/opt/postgresql@9.4/lib/pltcl.so
-r--r--r--  1 jerome  staff   164K Jun  1 12:41 /usr/local/opt/postgresql@9.4/lib/plpgsql.so
-r--r--r--  1 jerome  staff    84K Jun  1 12:41 /usr/local/opt/postgresql@9.4/lib/plperl.so

I tried

[jerome@jeroboam] > brew reinstall postgresql@9.4 --with-python

But it failed with

Error: invalid option: --with-python

And indeed, as per Remove all options from Homebrew/homebrew-core formulae, they estimated that options in brew are bad from now.
Then I found out: petere / homebrew-postgresql.
So, I used

[jerome@jeroboam] > brew tap petere/postgresql

And

[jerome@jeroboam] > brew reinstall petere/postgresql/postgresql@9.4

which ended up with

configure: error: header file <perl.h> is required for Perl

If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
  https://github.com/petere/homebrew-postgresql/issues


Error: A newer Command Line Tools release is available.
Update them from Software Update in System Preferences or run:
  softwareupdate --all --install --force

If that doesn't show you an update run:
  sudo rm -rf /Library/Developer/CommandLineTools
  sudo xcode-select --install

Then let's go

[jerome@jeroboam] > sudo rm -rf /Library/Developer/CommandLineTools
[jerome@jeroboam] > sudo xcode-select --install

And finally

[jerome@jeroboam] > brew reinstall petere/postgresql/postgresql@9.4

which ended OK this time

==> Summary
🍺  /usr/local/Cellar/postgresql@9.4/9.4.26: 2,989 files, 41.0MB, built in 2 minutes 8 seconds

And guess what? plpython is back.

[jerome@jeroboam] > ll -rt /usr/local/opt/postgresql\@9.4/lib/pl*
-r--r--r--  1 jerome  staff    42K Jun  1 15:53 /usr/local/opt/postgresql@9.4/lib/pltcl.so
-r--r--r--  1 jerome  staff   108K Jun  1 15:53 /usr/local/opt/postgresql@9.4/lib/plpython2.so
-r--r--r--  1 jerome  staff   164K Jun  1 15:53 /usr/local/opt/postgresql@9.4/lib/plpgsql.so
-r--r--r--  1 jerome  staff    84K Jun  1 15:53 /usr/local/opt/postgresql@9.4/lib/plperl.so