the fact of real shit

Copy large amount of file to remote server using nohup tar and ssh

This command will copy large amount of file to remote server by compressing and decompressing on the fly. It saves time and bandwidth. It will execute in background, so you can detach current login session.

nohup sh -c “tar -c /any/directory/at/source/server/ | gzip -2 | ssh server-alias ‘cat | tar xz -C /target/directory/of/target/server/'” > /dev/null 2>&1 &

here nohup output sent to /dev/null that means i don’t want any nohup output. you can adjust its behavior.

I use the command for millions of file that occupied more than 500 GB.

Posted in linuxTagged , , ,

add new hard disk into ubuntu more than 2TB size

lsblk

will list available device

parted /dev/sdd

considered device “sdd” from lsblk output

(parted) mklabel gpt
(parted) mkpart primary ext3 0 100%
(parted) print
(parted) quit

mkfs.ext3 /dev/sdd1

this command will format this hard disk into ext3 file system as we instruct by parted command

mkdir /home/data

mounting point directory creating

mount -t ext3 /dev/sdd1 /home/data

mounting formatted hard disk into target directory

blkid

try to find ID of new hard disk to write into fstab so that after restart our hard disk will mount automatically

sample output of blkid

/dev/sdd1: UUID="20e4b16b-4d4c-4053-b6f4-a2c103f2db2f" TYPE="ext3" PARTLABEL="primary" PARTUUID="33658d68-726f-4398-992f-2aaafebe17ff"

vi /etc/fstab

give an entry for our new hard disk at the end of this file

example entry

UUID=20e4b16b-4d4c-4053-b6f4-a2c103f2db2f /home/data ext3 nofail 0 0
Posted in linux, ubuntuTagged , , , ,

recursive change group and file mode in linux and detach active login session

change group

nohup sh -c “find /any/path/that/need/to/change/* -group mygroup -exec chgrp www-data {} \;” > /dev/null 2>&1 &

traditional way

nohup sh -c “chgrp -R www-data /any/path/that/need/to/change” > /dev/null 2>&1 &

change mode

nohup sh -c “find /any/path/that/need/to/change/* -perm u=rw,g=r,o=r -execdir chmod g+w {} \;” > /dev/null 2>&1 &

traditional way

nohup sh -c “chmod -R g+w /any/path/that/need/to/change” > /dev/null 2>&1 &

Posted in linux, ubuntuTagged , , , , , ,

Changing ZF2 Core 2.4.13 Zend\View\Helper\HeadLink::createDataStylesheet

class: Zend\View\Helper\HeadLink
function: createDataStylesheet
php version: 7.3

public function createDataStylesheet(array $args)
{
$rel = ‘stylesheet’;
$type = ‘text/css’;
$media = ‘screen’;
$conditionalStylesheet = false;
$href = array_shift($args);
if ($this->isDuplicateStylesheet($href)) { return false; } if (0 < count($args)) { $media = array_shift($args); if (is_array($media)) { $media = implode(',', $media); } else { $media = (string) $media; } } if (0 < count($args)) { $conditionalStylesheet = array_shift($args); if (!empty($conditionalStylesheet) && is_string($conditionalStylesheet)) { $conditionalStylesheet = (string) $conditionalStylesheet; } else { $conditionalStylesheet = null; } } $attributes = compact('rel', 'type', 'href', 'media', 'conditionalStylesheet'); if (0 < count($args) && is_array($args[0])) { $extras = array_shift($args); $attributes['extras'] = (array) $extras; } return $this->createData($attributes); }

new/modified/final/current code

public function createDataStylesheet(array $args)
{
$rel = ‘stylesheet’;
$type = ‘text/css’;
$media = ‘screen’;
$conditionalStylesheet = false;
$href = array_shift($args);
if ($this->isDuplicateStylesheet($href)) { return false; } if (0 < count($args)) { $media = array_shift($args); if (is_array($media)) { $media = implode(',', $media); } else { $media = (string) $media; } } if (0 < count($args)) { $conditionalStylesheet = array_shift($args); if (!empty($conditionalStylesheet) && is_string($conditionalStylesheet)) { $conditionalStylesheet = (string) $conditionalStylesheet; } else { $conditionalStylesheet = null; } } if (0 < count($args) && is_array($args[0])) { $extras = array_shift($args); $extras = (array) $extras; } $attributes = compact('rel', 'type', 'href', 'media', 'conditionalStylesheet', 'extras'); return $this->createData($attributes); }

previous/original/old code
Posted in php, zf2Tagged , , , ,

Install php 7.2 ssh2 in ubuntu 16x

recently i need to install ssh2 connection through my php code to connect remote server by sftp for file transfer. i’m using php 7.2 and face a “sigment fault” issue for normal installation. i solved this issue by install required demon. hope it may help someone who face same issue.

first you need to install/upgrade some basic program

apt-get install gcc make autoconf libc-dev pkg-config

then install base library

apt-get install libssh2-1-dev

now install required php modules

apt-get install php7.2-dev php-pear

now install ssh by pecl (the most important part of installation)

— pecl channel-update pecl.php.net
pear config-show
— pear config-set php_ini /etc/php/7.2/apache2/php.ini
— pear config-set temp_dir /etc/php/temp/pear
pecl install ssh2-1.1.2

ignore commented line OR use if you understand by yourself

now you need to enable ssh2 extension into your php cli installation

echo “extension=ssh2.so” > /etc/php/7.2/mods-available/ssh2.ini
ln -s /etc/php/7.2/mods-available/ssh2.ini /etc/php/7.2/cli/conf.d/30-ssh2.ini

please check you php installation/configuration path. set priority on your own. i set here 30 without proper understanding ūüėÄ

now another important part is your PHP code. when we use ssh2 in fopen wraper, in other version of ssh2 connection we need to open a connection and we can use resource id. but with the above change you must use user, password, port i.e. full access information every time we need to connect to server. here is my sample code –

$fh = @fopen(‘ssh2.sftp://’ . $this->user.’:’.$this->pass.’@’.$this->ip.’:’.(intval($this->port)>0?intval($this->port):22) . $pRemoteLocation, $pMode);

all other code like directory creation or any other command execution may be same as before.

Posted in php, ubuntuTagged , , , , , , ,

changing zf2 core 2.4.13 Zend/Stdlib/ArrayObject.php

class : Zend\Stdlib\ArrayObject
function: unserialize($data)
php version: 7.3

previous (original) code

/**
* Unserialize an ArrayObject
*
* @param string $data
* @return void
*/
public function unserialize($data)
{
$ar = unserialize($data);
$this->protectedProperties = array_keys(get_object_vars($this));

$this->setFlags($ar[‘flag’]);
$this->exchangeArray($ar[‘storage’]);
$this->setIteratorClass($ar[‘iteratorClass’]);

foreach ($ar as $k => $v) {
switch ($k) {
case ‘flag’:
$this->setFlags($v);
break;
case ‘storage’:
$this->exchangeArray($v);
break;
case ‘iteratorClass’:
$this->setIteratorClass($v);
break;
case ‘protectedProperties’:
continue;
default:
$this->__set($k, $v);
}
}
}

new code

public function unserialize($data)
{
$ar = unserialize($data);
$this->protectedProperties = array_keys(get_object_vars($this));

$this->setFlags($ar[‘flag’]);
$this->exchangeArray($ar[‘storage’]);
$this->setIteratorClass($ar[‘iteratorClass’]);

$need2continue=false;
foreach ($ar as $k => $v) {
$need2continue=false;
switch ($k) {
case ‘flag’:
$this->setFlags($v);
break;
case ‘storage’:
$this->exchangeArray($v);
break;
case ‘iteratorClass’:
$this->setIteratorClass($v);
break;
case ‘protectedProperties’:
/**
* in php 7.3 it shows
* Warning: “continue” targeting switch is equivalent to “break”. Did you mean to use “continue 2”? in /…location of zf2 ………../Zend/Stdlib/ArrayObject.php on line 426
*/
#continue;
$need2continue=true;
break;
default:
$this->__set($k, $v);
}
if(true==$need2continue) continue;
}
}

 

Posted in php, zf2Tagged , , , ,

going to change ZF2.4.9 core

I decided to change ZF 2.4.9 core. I know its a wired decision. But I don’t have any choice, cause they didn’t provide any release for PHP 7x. But the change will noted in this site by adding post for future.

Today I’m changing –

class: Zend\Validator\EmailAddress
function: idnToAscii($email) and
function: idnToUtf8($email)

Previous (original) code –

protected function idnToAscii($email)
{
if (extension_loaded(‘intl’)) {
return (idn_to_ascii($email) ?: $email);
}
return $email;
}

protected function idnToUtf8($email)
{
if (extension_loaded(‘intl’)) {
return idn_to_utf8($email);
}
return $email;
}

New Code

protected function idnToAscii($email)
{
if (extension_loaded(‘intl’)) {
return (idn_to_ascii($email, IDNA_DEFAULT, INTL_IDNA_VARIANT_UTS46) ?: $email);
}
return $email;
}

protected function idnToUtf8($email)
{
if (extension_loaded(‘intl’)) {
return idn_to_utf8($email, IDNA_DEFAULT, INTL_IDNA_VARIANT_UTS46);
}
return $email;
}

I took this decision for –

https://bugs.php.net/bug.php?id=75609 and
https://wiki.php.net/rfc/deprecate-and-remove-intl_idna_variant_2003

 

Posted in php, zf2Tagged , , ,

change mysql data directory ubuntu 17.xx

Follow https://www.digitalocean.com/community/tutorials/how-to-move-a-mysql-data-directory-to-a-new-location-on-ubuntu-16-04
NOTE: you can skip empty directory (/var/lib/mysql/mysql) creation script in ubuntu 17.xx version.

Then try to start mysql server. If fail then log messages are telling you that /usr/sbin/mysqld needs read (r) access to open /proc/14767/status, /sys/devices/system/node/ (trailing slash because it wants to read the directory), and /proc/14767/task/14767/mem. The file to edit is /etc/apparmor.d/usr.sbin.mysqld.

In my case I solved the problem by adding these lines somewhere in the middle (with two spaces in front of each):

/proc/*/status r,
/sys/devices/system/node/ r,
/sys/devices/system/node/** r,

Reload apparmor:

sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.mysqld
sudo /etc/init.d/apparmor reload

After doing that, try starting MySQL, and if you get more errors, add those files too and try again.

In some case mysql may fail to start for assoc i/o error. So, you may need to disable this settings by adding following line into /etc/mysql/mysql.conf.d/mysqld.cnf

innodb_use_native_aio=0

Another note about file permission of data directory is, “mysql” user should have permission into every directory of that path. Like if new data directory is “/media/myusb01/mysql-data” then “mysql” user should read access of “/media” folder and “/media/myusb01” folder and read write access at new data directory.

Hope mysql will start now.

Posted in linux, mysql, ubuntu