how-to
linux
bash

general linux

creating partition size larger than 2tb

in this example, operations are done to create /dev/sdb1 using ext4

$ parted /dev/sdb
GNU Parted 2.3
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)
(parted) mklabel gpt
Warning: The existing disk label on /dev/sdb will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? yes
(parted)
(parted) unit TB
# creating a primary with 3tb? use this
(parted) mkpart primary 0 3
 
# creating a partition using all the space available? use this
(parted) mkpart primary 0% 100%
 
# want to set the filesystem too?
(parted) mkpart primary ext4 0% 100%
# the partition will be used in a raid? set this flag
(parted) set 1 raid on
(parted) print
Model: ATA ST33000651AS (scsi)
Disk /dev/sdb: 3.00TB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
 
Number  Start   End     Size    File system  Name     Flags
 1      0.00TB  3.00TB  3.00TB  ext4         primary
(parted) quit
$ mkfs.ext4 /dev/sdb1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
183148544 inodes, 732566272 blocks
36628313 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
22357 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
	102400000, 214990848, 512000000, 550731776, 644972544
 
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 31 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

find the partition id, and add it to /etc/fstab

$ blkid /dev/sdb1
/dev/sdb1: UUID="7b7bdd66-218c-4a64-b760-82824f87724b" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="primary" PARTUUID="dc00594a-b233-46c3-8f79-cdc3a5cf28ad"
 
$ vim /etc/fstab
UUID=7b7bdd66-218c-4a64-b760-82824f87724b /DISK_BACKUP            ext4     defaults,noatime 0 0
 
$ mkdir /DISK_BACKUP
$ mount -a

create software raid 10

create some partitions using the raid flag. after that, use mdadm to manage the raid array.

yum install mdadm -y
mdadm --create /dev/md0 --level raid10 --name <RAID_NAME> --raid-disks <NUMBER_OF_DISKS> <LIST_OF_PARTITIONS, like /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1> 
echo "MAILADDR [email protected]" >> /etc/mdadm.conf
mdadm --detail --scan >> /etc/mdadm.conf

create new file system on the new raid device

mkfs.xfs /dev/disk/by-id/md-name-<RAID_NAME>
mkdir /data
mount /dev/disk/by-id/md-name-<RAID_NAME> /data

check the array status

mdadm --detail /dev/disk/by-id/md-name-<RAID_NAME>
#or
cat /proc/mdstat

simulate disk sdb1 failure

mdadm --manage --set-faulty /dev/disk/by-id/md-name-<RAID_NAME> /dev/sdb1
 
#Check syslog for new failure messages
tail /var/log/messages
 
Oct  3 16:43:42 centos-62-1 kernel: md/raid10:md0: Disk failure on sdb1, disabling device.
Oct  3 16:43:42 centos-62-1 kernel: md/raid10:md0: Operation continuing on 3 devices.
 
# check array status
mdadm --detail /dev/disk/by-id/md-name-<RAID_NAME>
cat /proc/mdstat

simulate disk sdd1 failure

mdadm --manage --set-faulty /dev/disk/by-id/md-name-<RAID_NAME> /dev/sdd1
 
# check syslog for new failure messages
tail /var/log/messages
 
Oct  3 16:45:01 centos-62-1 kernel: md/raid10:md0: Disk failure on sdd1, disabling device.
Oct  3 16:45:01 centos-62-1 kernel: md/raid10:md0: Operation continuing on 2 devices.
 
# check array status
mdadm --detail /dev/disk/by-id/md-name-<RAID_NAME>
cat /proc/mdstat

remove sdb1 from the array and re-add it

mdadm /dev/disk/by-id/md-name-<RAID_NAME> -r /dev/sdb1
mdadm /dev/disk/by-id/md-name-<RAID_NAME> -a /dev/sdb1
 
# check array status
mdadm --detail /dev/disk/by-id/md-name-<RAID_NAME>
cat /proc/mdstat

remove sdd1 from the array and re-add it

mdadm /dev/disk/by-id/md-name-<RAID_NAME> -r /dev/sdd1
mdadm /dev/disk/by-id/md-name-<RAID_NAME> -a /dev/sdd1
 
# check array status
mdadm --detail /dev/disk/by-id/md-name-<RAID_NAME>
cat /proc/mdstat

kill all processes owned by a user

# kill -9 $(pgrep -u username)

show ssl certificate (https or not)

# https website
openssl s_client -showcerts -servername docs.rda.run -connect docs.rda.run:443 < /dev/null
 
# any other tls service still works
openssl s_client -showcerts -servername tlshost.example.org -connect tlshost.example.org:49056 < /dev/null

create iso file from a directory

$ sudo genisoimage -nobak -iso-level 4 -o ./my.iso /my/path

/bin/rm: Argument list too long

$ find . -name '*.gz' | xargs rm

show dell service tag

$ sudo dmidecode -s system-serial-number

using a lockfile to run a script only one at a time

#!/bin/bash
 
lockfile="$0.lockfile"
 
if [[ -e "$lockfile" ]] && ! kill -0 "$(< "$lockfile")" 2>/dev/null; then
	echo "Warning: lockfile exists but the process is dead, continuing."
	rm -f "$lockfile"
elif [[ -e "$lockfile" ]]; then
	echo "My important work is already running. Exiting."
	exit 1
fi
 
printf '%s\n' "$$" > "$lockfile"
 
# add all your important stuff here.
 
rm -f "$lockfile"
 
exit 0

viewing crontab of all users

for user in $(cut -f1 -d: /etc/passwd); do echo $user; sudo crontab -u $user -l; done

listing only one column of a file

a file with columns separated by : can have its data displayed like this

cat file.log | cut -d":" -f4

showing a file content, removing duplicates

cat file.log | awk '!a[$0]++'

list the recursive contents of a directory

ls -R /my_path | awk '
/:$/&&f{s=$0;f=0}
/:$/&&!f{sub(/:$/,"");s=$0;f=1;next}
NF&&f{ print s"/"$0 }'

besides showing the contents recursively, this command also shows the full path of the files, not just the names.

reference: AskUbuntu.com

simulating the java trim() in bash

echo " test test test " | sed -e 's/^ *//' -e 's/ *$//'

reference: StackOverflow

get a file mime-type

file --mime-type -b my_file

using google drive with bash

this is accomplished with gdrive (opens in a new tab).

list files

$ drive list
Id                             Title                     Size     Created
0B3X9GlR6EmbnenBYSFI4MzN0d2M   drive-freebsd-amd64       5 MB     2013-01-01 21:57:01
0B3X9GlR6EmbnOVRQN0t6RkxVQk0   drive-windows-amd64.exe   5 MB     2013-01-01 21:56:41
0B3X9GlR6Embnc1BtVVU1ZHp2UjQ   drive-linux-arm           4 MB     2013-01-01 21:57:23
0B3X9GlR6EmbnU0ZnbGV4dlk1T00   drive-linux-amd64         5 MB     2013-01-01 21:55:06
0B3X9GlR6EmbncTk1TXlMdjd1ODQ   drive-darwin-amd64        5 MB     2013-01-01 21:53:34

file upload

$ drive upload --file drive-linux-amd64
Id: 0B3X9GlR6EmbnU0ZnbGV4dlk1T00
Title: drive-linux-amd64
Size: 5 MB
Created: 2013-01-01 21:55:06
Modified: 2013-01-01 21:55:06
Owner: Petter Rasmussen
Md5sum: 334ad48f6e64646071f302275ce19a94
Shared: False
Uploaded 'drive-linux-amd64' at 510 KB/s, total 5 MB

file download

$ drive download --id 0B3X9GlR6EmbnenBYSFI4MzN0d2M
Downloaded 'drive-freebsd-amd64' at 2 MB/s, total 5 MB

file sharing

$ drive share --id 0B3X9GlR6EmbnOVRQN0t6RkxVQk0
File 'drive-windows-amd64.exe' is now readable by everyone @ https://drive.google.com/uc?id=0B3X9GlR6EmbnOVRQN0t6RkxVQk0

pipe a file content directly to drive

$ echo "Hello World" | drive upload --stdin --title hello.txt
Id: 0B3X9GlR6EmbnVHlHZWZCZVJ4eGs
Title: hello.txt
Size: 12 B
Created: 2013-01-01 22:05:44
Modified: 2013-01-01 22:05:43
Owner: Petter Rasmussen
Md5sum: e59ff97941044f85df5297e1c302d260
Shared: False
Uploaded 'hello.txt' at 6 B/s, total 12 B

show the file content on stdout

$ drive download --stdout --id 0B3X9GlR6EmbnVHlHZWZCZVJ4eGs
Hello World

get file info

$ drive info --id 0B3X9GlR6EmbnVHlHZWZCZVJ4eGs
Title: hello.txt
Size: 12 B
Created: 2013-01-01 22:05:44
Modified: 2013-01-01 22:05:43
Owner: Petter Rasmussen
Md5sum: e59ff97941044f85df5297e1c302d260
Shared: False

get file download url

$ drive url --id 0B3X9GlR6EmbnVHlHZWZCZVJ4eGs
https://drive.google.com/uc?id=0B3X9GlR6EmbnVHlHZWZCZVJ4eGs

find file by content

grep -lr "text to fine" *.txt

reference: Quickly find any text string in any set of files