Menu principal

Autres

AideEnLigne
CahierDeBrouillon
Présentation
Administration

MesLectures
[Articles publiés]
[Blog Affordance]
[LibreOffice en ligne]
[Journal d'Ophelia]
[Kim Khal]

Informations
[Rue 89]

DNS etc
[Robtex]

Logins

FreeBSD Jails

Différence (depuis la version majeure précédente) (modification mineure)

Ajouté: 8a9

Modifié: 11,13c12,14

root# zfs create zroot/jails
root# zfs create zroot/jails/www
root# zfs create zroot/jails/dbase

root# zfs create -o mountpoint=/usr/local/jails zroot/jails
root# zfs create -o mountpoint=/usr/locat/jails/www zroot/jails/www
root# zfs create -o mountpoint=/usr/local/jails/dbase zroot/jails/dbase

Modifié: 57c58

2- Then, on the host, I must provide a way to mount /usr/local/jails/usocket every time a jail starts. I create two file named /etc/fstab.www and /etc/fstab.dbase.

2- Then, on the host, I must provide a way to mount /usr/local/jails/usocket every time a jail starts. I create two file named /etc/fstab.www and /etc/fstab.dbase. Note these mounts are of type nullfs.

Modifié: 77c78,80

And now for the network. I would like to config every jail with a valid IP address, so I could see/ping them on the LAN. To do that, I need a bridge and ethernet epairs to configure IP adresses to these two jails. FIXME vnet ?

And now for the network. I would like to config every jail with a valid IP address, so I could see/ping them on the LAN. To do that, I need a bridge and ethernet epair to configure IP adresses to these two jails.

epair is like a virtual ethernet cable. The usual way to do this is ifconfig epair create which yields two virtuals interfaces named e0a and e0b. The epair e0a stays on the host, and the configuration of the jail makes e0b to be attached to a jail, thus disapearing from the host. Of course you can assign TCP/IP address on each pair.

Modifié: 79c82

The usual way to this is ifconfig epair create which yields the ethernet pseudo interfaces named e0a and e0b.

And starting whit FreeBSD?-12, we have a vnet interface which I will use in the jails. "VNET jails give each jail its own isolated copy of the network stack. They get everything from the IP layer up, creating a network stack that is entirely its own, almost anything you could do with a distinct host you can do with a jail and VNET." You can read this here: https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/

Modifié: 81c84

This where the commands jib and jng are useful. jib is used in /etc/jail.conf (will see this later) to manage vnet interfaces and link them to a bridge. But to use them, I first copy these commands so they could be found in the PATH (provided /usr/local/bin is in your PATH!) :

It seems complicated, but there are the commands jib and jng to help me. jib is used in /etc/jail.conf (will see this later) to manage vnet interfaces and link them to a bridge. But to use them, I first copy these commands so they could be found in the PATH (provided /usr/local/bin is in your PATH!) :

Modifié: 83c86

root# cp/usr/share/examples/jails/{jib,jng} /usr/local/sbin

root# cp /usr/share/examples/jails/{jib,jng} /usr/local/sbin

Modifié: 180c183

Here is a rc.conf.sample that I copy in /etc in each jail and modify if necessary:

Here is a rc.conf.sample that I copy in each jail /etc and modify if necessary:

Modifié: 217c220

To verify, I ping a jail. I can ping a jail from the host or from anywhere in the 192.168.1.0/24 in the LAN.

To verify, I ping a jail. I can ping a jail from the host or from anywhere in the 192.168.1.0/24 LAN.

Modifié: 222c225

And now, for info, here the result of the command ifconfig on the host, where I can see: the bge0bridge with attached interfaces: bge0 (native eth interface), e0a_www and e0a_dbase. All this was done by the jib command in jail.conf

And now, for info, here the result of the command ifconfig on the host, where I can see: the bge0bridge with attached interfaces: bge0 (native eth interface), e0a_www and e0a_dbase. All this has been done by the jib command in jail.conf

Modifié: 312c315

root@www:/ #pkg search nginx

root@www:/ # pkg search nginx

Modifié: 316c319

root# pkg install nginx-1.18.0_45,2

root@www:/ # pkg install nginx-1.18.0_45,2

Modifié: 323c326

root# sysrc nginx_enable="YES"

root@www:/ # sysrc nginx_enable="YES"

Modifié: 325c328

root# service nginx start

root@www:/ # service nginx start

Modifié: 334c337

I install PHP to this jail and the module fpm needed for nginx, but it is a bit out of topic. There are many tutos for this on internet. Here are 2 links:

I now install PHP to this jail and the module fpm needed for nginx, but it is a bit out of topic. There are many tutos for this on internet. Here are 2 links:

Modifié: 339c342,344

I install Perl (pkg install perl5-5.32.0_1), instead, with p5-DBD-mysql-4.050 and p5-DBI-1.643

I install Perl (pkg install perl5-5.32.0_1), instead, with p5-DBD-mysql-4.050 and p5-DBI-1.643 which are two Perl module to access to MySQL?.

Due to security problems on my LAN, I configure nginx to listen on port 8080 (instead of port 80). I also modify the configuration file of nginx according to this guide https://kifarunix.com/install-nginx-mysql-php-femp-stack-on-freebsd-12/ :

Effacé: 341,342d345

Due to security problems on my LAN, I configure nginx to listen on port 8080 (instead of port 80). I also modify the configuration file of nginx according to this guide https://kifarunix.com/install-nginx-mysql-php-femp-stack-on-freebsd-12/
<pre>

Modifié: 345,351c348,355

modify lines like this:
user www;
worker-processes 2;
error_log /var/log/nginx/error.log info;
access_log /var/log/nginx/access.log;
server_namme: www;

Modify lines like this:

user www;
worker-processes 2;
error_log /var/log/nginx/error.log info;
access_log /var/log/nginx/access.log;
server_namme: www;

Modifié: 353,363c357,367

in the section server {
modify the line:
listen 80;
to read:
listen 8080;

in the section server {'''
modify the line:
index index.html index.htm;
to read:
index index.php index.html index.htm;

In the section server, modify the line:

listen 80;
to read:
listen 8080;

In the section server, modify the line:

index index.html index.htm;
to read:
index index.php index.html index.htm;

Modifié: 366,367c370,372

</pre>
Back to the host to install MySQL? on the dbase jail.

Back to the host, and jexec dbase to install MySQL? on the dbase jail.

Modifié: 369c374

root@www:/ #exit

root@www:/ # exit

Modifié: 387c392

Notice that MySQL? client is and Perl are also installed!

Notice that MySQL? client and Perl are also installed!

Modifié: 389c394

I used info from this link to install MySQL? correctly: https://kifarunix.com/install-nginx-mysql-php-femp-stack-on-freebsd-12/. In my case, I also want to use a unix domain socket on a decicated directory nullfs mounted (remember ?).

Ajouté: 395a401

Ajouté: 396a403

Ajouté: 397a405

Ajouté: 399a408,409

Enable mysql to start in rc.conf, and start mysql-server.

Modifié: 409c419

Huh? no unix socket here ? I must verify somthing:

Huh? no unix socket here ? I must verify this:

Modifié: 412c422

*incorrect permissions on the /usocket directory?

*incorrect permissions on the /usocket directory? See tip below.

Effacé: 417,418d426

root# service jail start
root# jexec dbase

Modifié: 421c429

root# service jail restart

root@dbase:/ # service jail restart

Modifié: 423c431,436

root#jexec www

root@dbase:/ # exit
root #

I verify again for the mysql.sock:

root # jexec www

Modifié: 428c441,442

root@www:/ #

root@www:/ #exit
root #

Modifié: 430c444

I can also verify the presence of the unix socket with the command netstat:

Another way to verify the presence of the unix socket is with the command netstat:

Ajouté: 431a446

root # jexec dbase

Modifié: 444c459

root@dbase:/ #mysql

root@dbase:/ # mysql

Modifié: 456c471,472

+------------------+------------------------------------------------------------------------+-----------------------+-----------+</nowiki>

+------------------+------------------------------------------------------------------------+-----------------------+-----------+
</nowiki>

Ajouté: 457a474,476

Notice the column plugin with caching_sha2_password. This thing will bite me later.

Modifié: 472c491

root@localhost [mydb]>insert into mytable (username, email) values ("cmic", "cmic@bigfoot.com");

root@localhost [mydb]>insert into mytable (username, email) values ("polo", "polo@bigfoot.com");

Modifié: 483c502

cmic

cmic@bigfoot.com

polo

polo@bigfoot.com

Modifié: 488c507

And last but not least, grant access to smic to mydb database:

And last but not least, grant access to user cmic1 to mydb database:

Modifié: 490c509

root@localhost [mydb]>GRANT SELECT, INSERT, UPDATE ON databaseName.* TO 'cmic1'@'%';

root@localhost [mydb]>GRANT SELECT, INSERT, UPDATE ON mydb.* TO 'cmic1'@'%';

Modifié: 498,502c517

Warning: I am not a Mysql guru, so there might be errors in this database installment. Forgive me. I only want to make a simple example that works.

root@dbase:/ # exit;
Bye
root@dbase:/ #

Warning: I am not a Mysql guru, so there might be errors in this database creation. Forgive me. I only want to make a simple example that works.

Modifié: 512c527

We go in jail www and prepare a Perl cript to access the database mydb thru the unix socket on /usocket.

We go in jail www and prepare a Perl script to access the database mydb thru the unix socket on /usocket.

Ajouté: 528a544

Modifié: 558c574

And I execute the script :

And I execute the script (located on / of the jail):

Modifié: 566c582

1 cmic cmic@bigfoot.com

1 polo polo@bigfoot.com

Modifié: 572c588,589

Now I have a php script /usr/local/www/nginx/index.php on jail www:

Now I have a php script /usr/local/www/nginx/index.php on jail www. To connect php to a unix socket, I found out this link which explains how to do this
https://stackoverflow.com/questions/45080641/specifying-socket-option-on-mysqli-connect. So my index.php follows:

Ajouté: 574a592,594

<pre>
<?php
error_reporting(0);

Modifié: 576,604c596,623

<?php
error_reporting(0);
//
$dbname = 'mydb';
$dbuser = 'cmic1';
$dbpass = 'MyPass?$2';
$dbhost = 'localhost';
//
$link = new mysqli('localhost', $dbuser, $dbpass, "", 3306, '/usocket/mysql.sock');
if ($link->connect_errno) {
echo "Echec lors de la connexion MySQL? : (" . $link->connect_errno . ") " . $link->connect_error;
echo "\n";
}
// some info
echo $link->host_info . " (via host_info) \n";
mysqli_select_db($link, $dbname) or die("Could not open the db '$dbname'");
$sql="select id, username, email from mytable";
$result=$link->query($sql);
// we get the data !!
if ($result->num_rows > 0) {
// output data of each row
while($row = $result->fetch_assoc()) {
echo "Id: " . $row["id"]. " > Name: " . $row["username"]. " > Email: " . $row["email"] . "\n";
}
} else {
echo "0 results";
}
$link->close();
?>

$dbname = 'mydb';
$dbuser = 'cmic1';
$dbpass = 'MyPass?$2';
$dbhost = 'localhost';

$link = new mysqli('localhost', $dbuser, $dbpass, "", 3306, '/usocket/mysql.sock');
if ($link->connect_errno) {
echo "Echec lors de la connexion MySQL? : (" . $link->connect_errno . ") " . $link->connect_error;
echo "\n";
}
// some info
echo $link->host_info . " (via host_info) \n";
mysqli_select_db($link, $dbname) or die("Could not open the db '$dbname'");
$sql="select id, username, email from mytable";
$result=$link->query($sql);
// we get the data !!

if ($result->num_rows > 0) {
// output data of each row
while($row = $result->fetch_assoc()) {
echo "Id: " . $row["id"]. " > Name: " . $row["username"]. " > Email: " . $row["email"] . "\n";
}
} else {
echo "0 results";
}
$link->close();
?>
</pre>

Modifié: 607c626,631

root@www:/ # php index.php

OK. Lets execute he index.php script.

+Testing the php thing

=

root@www:/ # php /usr/local/www/nginx/index.php

Modifié: 609c633,634

root@www:/ #

root@www:/ #exit
root #

Modifié: 611c636,639

Argh! Access denied. This one is stupid. I found out that the culprit is the default encryption method for password with MySQL? version 8. Now it is caching_sha2_password, but php doesn't understand this method for now. There is a fix to this: we go back to old encryptin methos in MySQL?:

Argh! Access denied. This one is stupid. I found out that the culprit is the default encryption method for password with MySQL? version 8. Now it is caching_sha2_password, but php doesn't understand this method for now.
I find the info here: https://programmerlib.com/fix-mysql8-connection-error/

There is a fix for this: go back to old encryption method in MySQL?:

Modifié: 614,615c642,643

root@dbase:/ #cd /usr/local/etc/mysql
root@web:/usr/local/etc/mysql #vi my.cnf

root@dbase:/ #
root@dbase:/ # vi /usr/local/etc/mysql/my.cnf

Modifié: 617c645

In section [mysqld], add the followin line:

In section [mysqld], I add the following line:

Modifié: 620c648,651

root@web:/usr/local/etc/mysql #cd /

Then I modify the encryption method for the user cmic1 in the database mysql, this time specifying mysql_native_password:

root@dbase:/usr/local/etc/mysql #cd /

Modifié: 626,627c657,658

root@localhost [(none)]>alter user 'cmic1'@'%' identified with mysql_native_password by 'MyPass?$2';
root@localhost [(none)]>select user, authentication_string, plugin from mysql.user;

root@localhost [(none)]>ALTER USER 'cmic1'@'%' IDENTIFIED WITH mysql_native_password BY 'MyPass?$2';
root@localhost [(none)]>SELECT user, authentication_string, plugin FROM mysql.user;

Modifié: 632,633c663

mysql.infoschema

$A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED

caching_sha2_password
</nowiki>

mysql.infoschema

$A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED

caching_sha2_password </nowiki>

Ajouté: 634a665,667

Back to jail www to test the php script again:

Modifié: 640c673

Id: 1 > Name: cmic > Email: cmic@bigfoot.com

Id: 1 > Name: polo > Email: polo@bigfoot.com

Modifié: 643c676

=====

This time, it works. You can adapt the index.php script for the web, and then point a browser at 192.168.1.202:8080 (IP adress of jail www) to see the result.

Modifié: 645,647c678

+Testing the whole stuf

lll

OK. The whole thing is not satisfying because of this /usocket shared directory which could be a security flaw. This is why we are trying another method to communicate between jails. Lets see how to do that

Ajouté: 648a680

=====

How to communicate between FreeBSD? Jails

TL;DR This article is about contructing two jails : a jail, named dbase, with MySQL?, the other, named www, with nginx. Jail www is visible on internet and displays table contents from a dabase on jail dbase.

The purpose is to install every thing the hard way to undersstand hos to communicate between the two jails.

+Create the jails

The install is made on a server named zoulou, IP 192.168.1.101, ethernet bge0, with FreeBSD?-12.2-Release. The jails are configured with ZFS on a local HD previously zfs'd zroot:

root# zfs create -o mountpoint=/usr/local/jails zroot/jails
root# zfs create -o mountpoint=/usr/locat/jails/www zroot/jails/www
root# zfs create -o mountpoint=/usr/local/jails/dbase zroot/jails/dbase
root# mkdir /usr/local/jails/.dist-files

Now I populate the ./dist-files directory with the base.txz file which I will use for the creation of the jails. For this install, I only need the base.txz package.

root# fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/amd64/12.2-RELEASE/base.txz -o /usr/local/jails/.dist-files
root# tar xvf /usr/local/jails/.dist-files/base.txz -C /usr/local/jails/www
root# tar xvf /usr/local/jails/.dist-files/base.txz -C /usr/local/jails/dbase

Of course, there are other ways to do this; you can use zfs snaphosts/zfs clone. YMMV

To update the FreeBSd? version of the jails and verify the checksums (IDS line), I must do this:

root# env UNAME_r=12.2-RELEASE freebsd-update -b /usr/local/jails/www fetch install
root# env UNAME_r=12.2-RELEASE freebsd-update -b /usr/local/jails/www IDS
root# env UNAME_r=12.2-RELEASE freebsd-update -b /usr/local/jails/dbase fetch install
root# env UNAME_r=12.2-RELEASE freebsd-update -b /usr/local/jails/dbase IDS

As the files in the jails has not yet been modified, there should be no errors with the checksum thing.

+Configuring the jails with a shared directory

There are three main ways to allow jails to communicate:

using TCP/IP socket: this is out of topic here.
using of unix sockets: (we will see later FIXME)
using a share directory: this is what I am doing here.

The share directory is a way to share a typical info: the famous mysql.sock file, which is the unix socket to access the MySQL? (or MariaDB?) databases.

It's rather simple: a directory is created on the host and on every jail; when the jails starts, the host directory is nullfs mounted on every jail. This happens in /etc/jail.conf

1- First, I create the directories: (feel free to choose any name or place for these directories.)

root# mkdir /usr/local/jails/usocket
root# mkdir /usr/local/jails/www/usocket
root# mkdir /usr/local/jails/dbase/usocket

+code for /etc/fstab.*

# <2021-01-26 18:22:43 cmic> to mount the socket file of mysql.
# fstab.www
# Device        Mountpoint      FStype  Options Dump    Pass#
/usr/local/jails/usocket /usr/local/jails/www/usocket nullfs rw,late 0 0
#
#END

# <2021-01-26 18:22:43 cmic> to mount the socket file of mysql.
# fstab.dbase
# Device        Mountpoint      FStype  Options Dump    Pass#
/usr/local/jails/usocket /usr/local/jails/dbase/usocket nullfs rw,late 0 0
#
#END

epair is like a virtual ethernet cable. The usual way to do this is ifconfig epair create which yields two virtuals interfaces named e0a and e0b. The epair e0a stays on the host, and the configuration of the jail makes e0b to be attached to a jail, thus disapearing from the host. Of course you can assign TCP/IP address on each pair.

 root# cp /usr/share/examples/jails/{jib,jng} /usr/local/sbin

So here is the /etc/jail.conf of the host zoulou:

+/etc/jail.conf

# <2021-01-21 14:40:39 cmic>
# adapted from the following script:
# https://www.cyberciti.biz/faq/configuring-freebsd-12-vnet-jail-using-bridgeepair-zfs/
#
# Todo first:
# cp /usr/share/examples/jails/jib /usr/local/sbin
# cp /usr/share/examples/jails/jng /usr//local/sbin
# chmod +x /usr/sbin/jib /usr/local/sbin/jng
#
# -hard-coded global variables
# -ifconfig the jail and defaultroute
#
$default_route="192.168.1.254";
$host_eth="bge0";
$ifconfig="/sbin/ifconfig";
$route="/sbin/route";
$jib="/usr/local/sbin/jib";
$jng="/usr/local/sbin/jng";
# To share a dir for mysql.sock
mount.fstab = "/etc/fstab.${name}";

#
# The www jail. With nginx (and PHP or Perl?)
#
www {
       $jail_ip="192.168.1.202";
       host.hostname = ${name};   # hostname
       path = "/usr/local/jails/${name}";     # root directory
       exec.clean;
       exec.system_user = "root";
       exec.jail_user = "root";
       #####
       vnet;
       vnet.interface = "e0b_${name}";        # vnet interface(s)
       exec.prestart += "${jib} addm ${name} ${host_eth}";
       exec.poststop += "${jib} destroy ${name}";

        # Standard stuff
       exec.start += "/bin/sh /etc/rc";
       exec.start += "${ifconfig} e0b_${name} inet ${jail_ip} up";
       exec.start += "${route} add default ${default_route}";
       exec.stop  = "/bin/sh /etc/rc.shutdown";
       exec.consolelog = "/var/log/jail_${name}_console.log";
       mount.devfs;          #mount devfs
       allow.raw_sockets;    #allow ping-pong
       devfs_ruleset="5";    #devfs ruleset for this jail
       allow.set_hostname = 1;
}
#
# The MySQL jail
#
dbase {
       $jail_ip="192.168.1.203";
       host.hostname = ${name};   # hostname
       path = "/usr/local/jails/${name}";     # root directory
       exec.clean;
       exec.system_user = "root";
       exec.jail_user = "root";
       #####
       vnet;
       vnet.interface = "e0b_${name}";               # vnet interface(s)
       exec.prestart += "jib addm ${name} ${host_eth}";
       exec.poststop += "jib destroy ${name}";

        # Standard stuff
       exec.start += "/bin/sh /etc/rc";
       exec.start += "${ifconfig} e0b_${name} inet ${jail_ip} up";
       exec.start += "${route} add default ${default_route}";
       exec.stop = "/bin/sh /etc/rc.shutdown";
       exec.consolelog = "/var/log/jail_${name}_console.log";
       mount.devfs;          #mount devfs
       allow.raw_sockets;    #allow ping-pong
       devfs_ruleset="5";    #devfs ruleset for this jail
       allow.set_hostname = 1;
}
#
#END

Some explanations of jail.conf:

when the service jail starts, $name variable is the name of the jail,
consider the lines from $default_route to mount_fstab{name} to be like global variables for service jail,
note the default_route which will be configured in every jail,
the mounted operation happens here: mount.fstab = "/etc/fstab.${name}";,
for each jail, a TPIP/address is given,
the line vnet; indicates that the jail will use a vnet interface; next its name is given and parametrized,
exec.prestart += "${jib} addm ${name} ${host_eth}"; is where the bridge is created and interfaces are added,
the exec.start lines are here to configure the vnet interface and default route.

I have to configure /etc/rc.conf, localtime, /etc/resolv.conf, /etc/hosts in each jail. Here is a rc.conf.sample that I copy in each jail /etc and modify if necessary:

+rc.conf.sample

#
# Example of rc.conf for a jail.
# Customize this file !
#
sendmail_enable="NONE"
keymap="fr.iso.acc"
#keymap="fr"
defaultrouter="192.168.1.254"
sshd_enable="YES"
zfs_enable="YES"
######

And I cp this files too:

root# cp /etc/hosts /usr/local/jails/www/etc
root# cp /etc/localtime /usr/local/jails/www/etc
root# cp /etc/resolv.conf /usr/local/jails/www/etc

I enable the jails on the host, start the jails, and list them with jls:

Notice that the IP adress field is empty, which is not a problem.

root# sysrc jail_enable="YES"
root# service jail start
Starting jails: www dbase.
root# jls
   JID  IP Address      Hostname                      Path
     7                  www                           /usr/local/jails/www
     8                  dbase                         /usr/local/jails/dbase
root#

To verify, I ping a jail. I can ping a jail from the host or from anywhere in the 192.168.1.0/24 LAN.

 root# ping -c 1 192.168.1.203

+ifconfig on the host

bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c0099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether 2c:41:38:87:3e:d4
        inet 192.168.1.101 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bge0bridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:11:c2:f2:6e:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_dbase flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: e0a_www flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: bge0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
        groups: bridge
        nd6 options=1<PERFORMNUD>
e0a_www: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:40:d0:87:3e:d4
        hwaddr 02:71:dc:ce:e1:0a
        inet6 fe80::71:dcff:fece:e10a%e0a_www prefixlen 64 scopeid 0x5
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0a_dbase: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:40:c9:87:3e:d4
        hwaddr 02:64:69:56:1c:0a
        inet6 fe80::64:69ff:fe56:1c0a%e0a_dbase prefixlen 64 scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Now that the jails are running, I add hostname variable on each jail:

root# sysrc -j www hostname="www"
hostname: -> www
root# sysrc -j dbase hostname="dbase"
hostname: -> dbase

I "log" on a jail with the command jexec and, for info, I look at the result of the ifconfig commend:

root# jexec www
root@www:/ #ifconfig 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
e0b_www: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 0e:40:d0:87:3e:d4
        hwaddr 02:71:dc:ce:e1:0b
        inet 192.168.1.202 netmask 0xffffff00 broadcast 192.168.1.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
root#

I can see the other end of epair, e0b_www and its IP address. So I verify that I can ping any host on the LAN. Everything OK. Now I have to install more software on the jail.

+Installing software in the jails

Always logged in the jail ? If not, I do a jexec www. Then I install nginx, enable it, and launch it.

 root@www:/ # pkg search nginx
 ...
 nginx-1.18.0_45,2
 ...
 root@www:/ # pkg install nginx-1.18.0_45,2
 ...
 The following 2 package(s) will be affected (of 0 checked):
 New packages to be INSTALLED:
 nginx: 1.18.0_25,2
 pcre: 8.44
 ...
 root@www:/ # sysrc nginx_enable="YES"
 nginx_enable -> YES
 root@www:/ # service nginx start 
 Performing sanity check on nginx configuration:
 nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
 nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
 Starting nginx.
 root@www:/ #

I verifiy that nginx works OK by starting a browser on a host in the LAN, address pointing to 192.168.1.202 (www jail IP address). I should see a "Welcome to nginx" message..

I now install PHP to this jail and the module fpm needed for nginx, but it is a bit out of topic. There are many tutos for this on internet. Here are 2 links:

nginx & PHP https://www.osradar.com/install-nginx-php-freebsd/ . I fumble thru the docs, because I am not a PHP man...
nginx & PHP & MySQL? https://kifarunix.com/install-nginx-mysql-php-femp-stack-on-freebsd-12/

I install Perl (pkg install perl5-5.32.0_1), instead, with p5-DBD-mysql-4.050 and p5-DBI-1.643 which are two Perl module to access to MySQL?.

 root@www:/ # vi /usr/local/etc/nginx/nginx.conf 
 ...

Modify lines like this:

 user www;
 worker-processes 2;
 error_log  /var/log/nginx/error.log info;
 access_log  /var/log/nginx/access.log;
 server_namme: www;

In the section server, modify the line:

 listen   80;
 to read:
 listen 8080;

In the section server, modify the line:

 index  index.html index.htm;
 to read:
 index  index.php index.html index.htm;

 root@www:/ # service nginx restart

Back to the host, and jexec dbase to install MySQL? on the dbase jail.

 root@www:/ # exit
 root# jexec dbase 
 root@dbase:/ # pkg search mysql
 ..
 mysql80-client-8.0.22_1        Multithreaded SQL database (client)
 mysql80-server-8.0.22_1        Multithreaded SQL database (server)
 ..
 root@dbase:/ # pkg install mysql80-server-8.0.22_1
 ...
 Number of packages to be installed: 21

 The process will require 457 MiB more space.
 57 MiB to be downloaded.

 Proceed with this action? [y/N]:y
 ...
 root@dbase:/ #

Notice that MySQL? client and Perl are also installed!

So I have to modify the file my.cnf to point to the right directory:

 root@dbase:/ # vi /usr/local/etc/mysql/my.cnf

In the [client] section, modify the socket line to:

 socket     = /usocket/mysql.sock

In the [server] section, modify the socket line to:

 socket     = /usocket/mysql.sock

Enable mysql to start in rc.conf, and start mysql-server.

 root@dbase:/ # sysrc mysql_enable="YES"
 mysql_enable:  -> YES
 root@dbase:/ # service mysql-server start
 Starting mysql.

I verify that the famous unix socket is in the right place:

 root@dbase:/ #ls -l /usocket

Huh? no unix socket here ? I must verify this:

look at /var/db/mysql/dbase.err to have a hint!
incorrect permissions on the /usocket directory? See tip below.
incorrect bind-address in my.cnf?

Classic tip for permissions (88 is the userid and groupid of mysql):

 root@dbase:/ # chown mysql /usocket && chgrp mysql /usocket
 root@dbase:/ # exit
 root@dbase:/ # service jail restart
 ..
 root@dbase:/ # exit
 root #

I verify again for the mysql.sock:

 root # jexec www
 root@www:/ # ls -l /usocket/
 total 1
 srwxrwxrwx  1 88  88  0 Feb  5 22:26 mysql.sock
 -rw-------  1 88  88  6 Feb  5 22:26 mysql.sock.lock
 root@www:/ #exit
 root #

Another way to verify the presence of the unix socket is with the command netstat:

 root # jexec dbase
 root@dbase:/ #netstat -an
 ...
 Address          Type   Recv-Q Send-Q            Inode             Conn             Refs          Nextref Addr
 fffff80010225b00 stream      0      0 fffff8004a0d5b40                0                0                0 /usocket/mysql.sock
 fffff80010225c00 stream      0      0 fffff8003d6e45a0                0                0                0 /tmp/mysqlx.sock
 ...
 root@dbase:/ #

MySQL? server is running OK. Now I add a user cmic1 who can connect from anywhere on the lan, whith a password MyPass?$2. Then I create a database with one table and some dummy data. The purpose is to have a useful set of data to work with. The full list of instructions is here:

+Creating data in MySQL?

 root@dbase:/ # mysql
 root@localhost [(none)]>CREATE USER 'cmic1'@'%' IDENTIFIED BY 'MyPass?$2';
 Query OK, 0 rows affected (0.03 sec)
 root@localhost [(none)]>SELECT user,authentication_string,plugin,host FROM mysql.user;
 +------------------+------------------------------------------------------------------------+-----------------------+-----------+
 | user             | authentication_string                                                  | plugin                | host      |
 +------------------+------------------------------------------------------------------------+-----------------------+-----------+
 | cmic1            | $A$005$7|@y(m(E:>0^e@rv,%tQv3zIxEMizVb3gvyAJunrBv9Ocf9/DRKL4QtwNOyz4 | caching_sha2_password | %         |
 | mysql.infoschema | $A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED | caching_sha2_password | localhost |
 | mysql.session    | $A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED | caching_sha2_password | localhost |
 | mysql.sys        | $A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED | caching_sha2_password | localhost |
 | root             |                                                                        | caching_sha2_password | localhost |
 +------------------+------------------------------------------------------------------------+-----------------------+-----------+

 5 rows in set (0.00 sec)

Notice the column plugin with caching_sha2_password. This thing will bite me later.

 root@localhost [(none)]>create database mydb;
 Query OK, 1 row affected (0.03 sec)
 root@localhost [(none)]>use mydb;
 root@localhost [mydb]> CREATE TABLE mytable
    ->  (
    ->    id              int unsigned NOT NULL auto_increment,
    ->    username        varchar(100) NOT NULL,
    ->    email           varchar(100) NOT NULL,
    ->    PRIMARY KEY     (id)
    ->  );
 Query OK, 0 rows affected (0.15 sec)

OK. A database with one table is created. Now I must insert some data in it, then look at the table content:

 root@localhost [mydb]>insert into mytable (username, email) values ("polo", "polo@bigfoot.com");
 Query OK, 1 row affected (0.03 sec)
 root@localhost [mydb]>insert into mytable (username, email) values ("gaston", "king-bee@hive.net");
 Query OK, 1 row affected (0.03 sec)
 root@localhost [mydb]>
 root@localhost [mydb]>
 root@localhost [mydb]>
 root@localhost [mydb]>select id, username, email form mytable;
 +----+----------+------------------------+
 | id | username | email                  |
 +----+----------+------------------------+
 |  1 | polo     | polo@bigfoot.com       |
 |  2 | gaston   | king-bee@hive.net      |
 +----+----------+------------------------+
 2 rows in set (0.00 sec)

And last but not least, grant access to user cmic1 to mydb database:

 root@localhost [mydb]>GRANT SELECT, INSERT, UPDATE ON mydb.* TO 'cmic1'@'%';
 Query OK, 0 rows affected (0.02 sec) 
 root@localhost [mydb]>flush privileges;
 Query OK, 0 rows affected (0.02 sec)
 root@localhost [mydb]>exit;
 Bye
 root@dbase:/ #

Warning: I am not a Mysql guru, so there might be errors in this database creation. Forgive me. I only want to make a simple example that works.

I am now ready to get the info of mydb from the www jail.

+MySQL? is in another jail !

We go in jail www and prepare a Perl script to access the database mydb thru the unix socket on /usocket.

The Perl script on www jail:

+/testdb.pl

#!/usr/local/bin/perl
#
# Testing a databse on jail dbase, Ip 192.168.1.203
#
use strict;
use warnings;
use v5.10; # for say() function    
#
use DBI;
say "Perl MySQL Connect Demo";
say "-----------------------";

#
# database name is mydb
# How to connect to database on another host/jail??

my $database="mydb";
my $portnum="3306";
my $dsn = "DBI:mysql:database=$database;mysql_socket=/usocket/mysql.sock;port=$portnum";
my $username = "cmic1";
my $password = 'MyPass$2';
my @res;

my $dbh  = DBI->connect($dsn,$username,$password) or
           die("Error connecting to the database: $DBI::errstr\n");
say "Connected to MySQL on jail dbase, mydb database \n";
#
# extract data
#
my $prepare=$dbh->prepare('select id, username, email from mytable') or die $dbh->errstr;
$prepare->execute() or die "Echec requete\n";
#
my ($id, $name, $mail);
while (($id, $name, $mail)=$prepare->fetchrow()) {
        print $id . " " . $name  . " " . $mail . "\n";
}
#
#END

And I execute the script (located on / of the jail):

 root@www:/ # chmod +x testdb.pl
 root@www:/ # ./testdb.pl
 Perl MySQL? Connect Demo
 --------------------------
 Connected to MySQL? on jail dbase, mydb database

 1 polo polo@bigfoot.com
 2 gaston king-bee@hive.net
 root@www:/ #

OK. It works as expected.

Now I have a php script /usr/local/www/nginx/index.php on jail www. To connect php to a unix socket, I found out this link which explains how to do this https://stackoverflow.com/questions/45080641/specifying-socket-option-on-mysqli-connect. So my index.php follows:

+index.php on jail www

<?php
error_reporting(0);

$dbname = 'mydb';
$dbuser = 'cmic1';
$dbpass = 'MyPass$2';
$dbhost = 'localhost';

$link = new mysqli('localhost', $dbuser, $dbpass, "", 3306, '/usocket/mysql.sock');
if ($link->connect_errno) {
    echo "Echec lors de la connexion  MySQL : (" . $link->connect_errno . ") " . $link->connect_error;
    echo "\n";
}
// some info
echo $link->host_info . " (via host_info) \n";
mysqli_select_db($link, $dbname) or die("Could not open the db '$dbname'");
$sql="select id, username, email from mytable";
$result=$link->query($sql);
// we get the data !!

if ($result->num_rows > 0) {
  // output data of each row
  while($row = $result->fetch_assoc()) {
    echo "Id:  " . $row["id"]. " > Name: " . $row["username"]. " > Email: " . $row["email"] . "\n";
  }
} else {
  echo "0 results";
}
$link->close();
?>

OK. Lets execute he index.php script.

+Testing the php thing

 root@www:/ # php /usr/local/www/nginx/index.php
 Echec lors de la connexion  MySQL? : (2054) The server requested authentication method unknown to the client.
 root@www:/ #exit
 root #

Argh! Access denied. This one is stupid. I found out that the culprit is the default encryption method for password with MySQL? version 8. Now it is caching_sha2_password, but php doesn't understand this method for now. I find the info here: https://programmerlib.com/fix-mysql8-connection-error/

There is a fix for this: go back to old encryption method in MySQL?:

 root # jexec dbase
 root@dbase:/ # 
 root@dbase:/ # vi /usr/local/etc/mysql/my.cnf

In section [mysqld], I add the following line:

 default_authentication_plugin = mysql_native_password

Then I modify the encryption method for the user cmic1 in the database mysql, this time specifying mysql_native_password:

 root@dbase:/usr/local/etc/mysql #cd /
 root@dbase:/ # service mysql-server restart
 Stopping mysql.
 Waiting for PIDS: 41024.
 Starting mysql.
 root@dbase:/ # mysql
 root@localhost [(none)]>ALTER USER 'cmic1'@'%' IDENTIFIED WITH mysql_native_password BY 'MyPass?$2';
 root@localhost [(none)]>SELECT user, authentication_string, plugin FROM mysql.user;
 +------------------+------------------------------------------------------------------------+-----------------------+
 | user             | authentication_string                                                  | plugin                
 +------------------+------------------------------------------------------------------------+-----------------------+
 | cmic1            | *9824B3D032944E5667EB0D380094F9B02CC39A2A                              | mysql_native_password 
 | mysql.infoschema | $A$005$THISISACOMBINATIONOFINVALIDSALTANDPASSWORDTHATMUSTNEVERBRBEUSED | caching_sha2_password 
 ...

Back to jail www to test the php script again:

 root@localhost [(none)]>exit;
 root@dbase:/ # exit
 root # jexec www
 root@www:/ # php /usr/local/www/nginx/index.php
 Localhost via UNIX socket (via host_info)
 Id:  1 > Name:  polo > Email: polo@bigfoot.com
 Id:  2 > Name:  gaston > Email: king_beel@hive.net

This time, it works. You can adapt the index.php script for the web, and then point a browser at 192.168.1.202:8080 (IP adress of jail www) to see the result.