User Tools

Site Tools


linux:ubuntu:useful-commands

Table of Contents

Useful commands - linux / ubuntu

[file] Show file type

file file.txt

[cat] Add line numbers to a file

cat -n file.txt
nl file.txt

[tree] Show tree of files

tree -F -L 1 /

-F - show file type
-L - level (how nested should be), 1 - show only files under first directory level.

ln -s lab02/.Hulk hulk

This will create file hulk as a shortcut for the (original) file lab02/.Hulk

[tar] Create a tar archive

tar -cvf archive.tar decrypted_file.txt encrypted_file.enc dir2

c - create
v - verbose
f - file name of the archive

[tar] Add a new file to the tar archive

tar -uf documente.tar raport_nou.txt
tar -rf documente.tar raport_nou.txt

-u - update
-r - add (does not check if the file is newer)

[tar] View tar archive content

tar -tf archive.tar

t - table (view archive content as a 'table')
f - file name (archive name)

[tar] Extract tar content

tar -xvf archive.tar

x - extract
v - verbose
f - file

[tar] Create a tar.gz archive

tar -czvf archive.tar.gz decrypted_file.txt encrypted_file.enc dir2

z - compressed gzip
c - create
v - verbose
f - file name of the archive

[gunzip] Add a file to existing tar.gz archive

This cannot be done directly using tar tool, but we will use gunzip to extract the tar archive, add a new file to it and then gzip it again.

gunzip archive.tar.gz && tar -rf archive.tar newfile.txt && gzip archive.tar

[tar] View tar.gz archive content

tar -tzf archive.tar.gz

t - table (view archive content as a 'table')
z - compressed gzip
f - file name (archive name)

[tar] Extract tar.gz content

tar -xzvf archive.tar.gz

x - extract
v - verbose
f - file

[zip] Create a zip archive

zip archive.zip file1.txt file2.txt

[zip] Add a new file to the zip archive

zip -u archive.zip new_file

-u - update (or add if not already in the archive)

[zip] View zip archive contents

zip -sf archive.zip
zip --show-files archive.zip

[zip] Extract a zip archive

unzip archive.zip -d dir

-d - optional destination directory, if missing the current directory will be used.

Find a file / directory

[find] Find all txt files in the users home directory

find ~ -name "*.txt"

[find] Find insensitive

sudo find / -iname "*ssh*conf"

[find] Find all files > 100 MB

find ~ -type f -size +100M

[find] Find and remove all files in /tmp directory

find /tmp -type f -name "*.tmp" -exec rm {} \;

{} - placeholder for each file, it will contain the full path name
\; - marks the end of the command

[find] Find executable files

find . -type f -executable

[find] Find all empty directories (and subdirectories)

find . -type d -empty

[find] Find all files that contain at least one digit (number) from a directory

find /usr/include -name '*[0-9]*'

Locate a file (searching in database instead of actual files on disk)

[locate] Installing the locate tool

sudo apt install mlocate
sudo updatedb

[locate] Locate a file

locate "*ssh*conf"

[locate] Locate a file insensitive

locate -i "Example.txt"

[locate] Locate only the first 10 results

locate example.txt | head -10

[ps] List all active processes

ps -ef

e - 'every' - we want to see every process
f - 'full' - get full details

ps aux

a - all users
u - user oriented format output (display user who owns the process, CPU, memory, etc.)
x - also includes the processes that are not attached to a terminal.

[ps] List all processes of a certain user

ps -u username

[ps] List details of a specific process

Change 2638 with the actual process pid.

ps -p 2638

[ps] List processes based on cpu usage

ps -aux --sort=-%cpu

[ps] List processes based on memory usage

ps -aux --sort=-%mem

[ps] List processes with custom columns

ps -e -ouser,uid,pid,%mem,%cpu,rss,cmd --sort=%mem

[ps] List processes - descending sorting

Just add a - in the –sort.

ps -e -ouser,uid,pid,%mem,%cpu,rss,cmd --sort=-%mem | less

[ps] Search for a specific process

Replace gnome with your keyword.

ps -ef | grep gnome

[pstree] List the process tree

pstree
pstree -u
pstree -p

u - show users p - show pid

Get the pid of a process

Multiple methods:

pgrep nginx
ps aux | grep nginx | grep -v grep
pidof nginx

[apt] APT commands

sudo apt install nginx
sudo apt search nginx
sudo apt remove nginx
sudo apt purge nginx
sudo apt update
sudo apt upgrade

[ip] Check if a network card is active

sudo ip link show

[ip] UP / DOWN states of network interfaces

Display only the names and states (UP / DOWN) of the network interfaces on the system, using the format interface_name:state.

ip link show | awk '/^[0-9]+:/ {print $2 " " $9}' | sed 's/://g' | awk '{print $1 ":" ($2=="UP"?"UP":"DOWN")}'

Output:

lo:DOWN
ens33:UP
ens34:UP
docker0:DOWN
br1:DOWN
br2:DOWN

[ip] Start a network interface

ip link set dev eth1 up

[ip] View IPs on all network interfaces

sudo ip address show

[ip] Remove a network configuration from an interface

sudo ip address flush eth1

[ping] Check if a station is alive

ping -c 4 10.125.50.50

c - number of packets to send (by default is infinite)

[ip] Show routing table / find default gateway

ip route show

Default gateway (192.168.112.2) is listed under the 'default via' section:

To extract just the default gateway ip:

ip route show | awk 'NR==1 {print $3}'

[ip] [nmcli] Find DNS Servers

nmcli dev show | grep DNS

resolvectl status | grep 'Current DNS'

To print only IPs:

nmcli dev show | grep DNS | cut -d ':' -f2 | tr -d ' '

[ip] [host] Get the IP address of a host / Check if DNS service is working

host elf.cs.pub.ro

Update DNS Servers

Change the /etc/resolv.conf: nameserver 8.8.8.8 nameserver 8.8.4.4

[netstat] Display open UDP ports

sudo netstat -ulpn

u - udp ports only
l - listening state ports only
p - program name / PID that has opened the port
n - shows the actual IP address on which the system is listening for connections, instead of resolving the host name.

sudo ss -ulpn

[netstat] Display open TCP ports

sudo netstat -tlpn

t - TCP

sudo ss -tlpn

[netstat] Display both TCP and UPD ports

sudo netstat -tulpn

t - TCP
u - UDP

sudo ss -tulpn

[nc] Establish a TCP communication with an application - Netcat

nc google.com 80

[wget] Download a single file from a HTTP(s) / FTP server

wget elf.cs.pub.ro

Download in background with a custom file name:

wget -b -O google.html http://google.com

b - run in background
O - output file name.

[curl] Show only HTTP Header of a site

curl -I -L http://ocw.cs.pub.ro

I (or –head) - send only HTTP HEAD request instead of HTTP GET
L - follow any redirectrion.

[ssh] Connect to a SSH server

ssh user@192.168.1.8

[ssh] Connect to a SSH server and run a command

ssh root@10.10.10.3 ip address show

[scp] Send / copy a file to a remote machine

scp /home/file.txt student@10.10.10.3:~/

[scp] Receive / download a file from a remote machine

scp student@10.10.10.3:/home/file.txt ~/

[scp] Send / copy a directory to a remote machine

scp -r /home/dir student@10.10.10.3:~/

[scp] Receive / download a file from a remote machine

scp -r student@10.10.10.3:/home/dir ~/

[ssh][public key] SSH Public Key authentication

ssh-keygen -t rsa -b 4096
ssh-copy-id root@10.10.10.3

What was generated:

  • ~/.ssh/id_rsa –> private key
  • ~/.ssh/id_rsa.pub –> public key

[ls] Escaping globbing patterns / special characters

Just enclose the string in “”

ls support-globbing/"[HOME]"*
find . -name "*search*"

Get memory information

cat /proc/meminfo | grep "Mem"

Get CPU information

cat /proc/cpuinfo

[grep] Search for a string in a file

cat workspace/C/searching/binary_search.c | grep search
grep search workspace/C/searching/binary_search.c
grep 'error' log.txt

[grep] Case insensitive search for a string in a file

grep -i 'string_to_search' file.txt

[grep] Count number of lines of searched string in file

grep -c 'cache' /proc/cpuinfo

[grep] Count number of occurrences (not just lines) of searched string in file

grep -o 'searched_string' file.txt | wc -l

o - print each occurrence on a separate line
l (from wc) - count lines

[grep] Recursive search for a string in a directory

grep -r 'cache' /proc 2>/dev/null

2>/dev/null - don't show errors

[grep] Recursive (and insensitive with line number displayed) search for a string in a directory

grep -nri search workspace/C/ | less

n - show line number
r - recursive
i - case insensitive

[grep] Recursive search for a word in a directory

grep -nri -w "search" workspace/C/ | less

[grep] Display lines that does NOT contain the searched string

Show all lines that does not contain error string:

grep -v 'error' log.txt

[grep] Display lines that starts with a certain string

Search for lines that starts with error string:

grep '^error' log.txt

[grep] Display a certain number of lines AFTER matching a string in a file

Displays each line that contains error, followed by 3 lines:

grep -A 3 'error' log.txt

[grep] Display a certain number of lines BEFORE matching a string in a file

Displays two lines before each line that contains error:

grep -B 2 'error' log.txt

[grep] Display a certain number of lines BEFORE and AFTER matching a string in a file

Displays two lines before and after each line that contains error:

grep -C 2 'error' log.txt

[grep] Search in a file for one string OR another using ERE (Extended Regular Expressions)

grep -E -i "error(s)?|warning(s)?" example.txt
E - extended regular expression
i - case insensitive
error(s) - search for strings that contains error or errors
| - or
warning(s) - search for string that contains warning or warnings
? - makes that previous element (s) to be optional.

[grep] Search in a file for an exact match of string OR another using ERE (Extended Regular Expressions)

grep -E -i "\berror(s)?|warning(s)?\b" example.txt

\b - the match will be made at the start and at the end of a word.

[grep] Add an alias - go for xdg-open

Instead of running xdg-open https://google.com, we want to use go https://google.com.
For this we need to define an alias in ~/.bash_aliases (this is read from ~/.bashrc).
So we need to append line: alias go='xdg-open'

grep -qxF 'alias go='\''xdg-open'\''' ~/.bash_aliases || echo 'alias go='\''xdg-open'\''' >> ~/.bash_aliases
|| - the command after this OR operator will be executed if the command before it fails.
q - run in silent mode, without printing anything
x - search on the ENTIRE line
F - disable regular expression, the string is a simple text.
>> - append to the end.

Next, reload the ~/.bashrc file:

source ~/.bashrc

[;] Execute multiple commands in the same line, one by one

mkdir demo; cd demo; touch Hello; ls

[&&] Execute a command on the same line if the PREVIOUS one SUCCEEDED

mkdir operators/demo && cd operators/demo

[||] Execute a command on the same line if the PREVIOUS one FAILED

ls -d operators || mkdir operators

[find] Print info about files in the current directory using FIND with -exec

find . -maxdepth 1 -type f -exec ls -l {} \;

[find] Print info about files in the current directory using FIND and PIPE

find . -maxdepth 1 -type f -print0 | xargs -0 ls -l

-print0 - finish each string found with a null character instead of newline. This helps in manipulating files that contains spaces or newlines.
0 - treat received input as a list of args separated by the null character - used to correctly processed the output of print0.

[find] Find all files from /lib that does not start with the lib prefix

find -L /lib -type f -not -name 'lib*.*'

L - follow symlinks (this is necessary because lib is a symlink for usr/lib)
not - negate the name

[xargs] Move all files (requiring confirmation) with a certain extension from the current folder to parent

ls *.c | xargs -I {} -p mv {} ..

I {} - placeholder / replacer for each character
p - requires user confirmation

Safer version (handles better the files with spaces):

find . -maxdepth 1 -type f -name "*.c" -exec mv {} .. \;

[1> 2>] Redirect STDOUT and STDERR

Redirect both in firefox-ignore file.

firefox &> firefox-ignore

Another way:

firefox > firefox-ignore 2>&1

To redirect input in a file and output to another file:

firefox > firefox-output.txt 2> firefox-error.txt

OR

firefox 1> firefox-output.txt 2> firefox-error.txt
> is equivalent to 1>

[dd] Generate 100 MB file with null values

dd if=/dev/zero of=null-100mb count=100 bs=1M

if=/dev/zero - input file will get data from /dev/zero
of=null-100mb - name of the output file
count=100 - number of blocks that will be copied
bs=1M - size of each block

[dd] Generate 100 MB file with random values

dd if=/dev/urandom of=rand-100mb count=100 bs=1M

[grep] Simple Regular Expressions Examples

Search for lines that starts with Hello:

grep '^Hello' message.txt

Search for lines that ends with ending:

grep 'ending$' message.txt

Search any lines that contains e followed by any character and then r:

grep 'e.r' message.txt

Search lines that contains sc followed by h or y and the o:

grep 'sc[hy]o' wordlist.txt

Search lines that contains sc followed any character except h or y and the o:

grep 'sc[^hy]o' wordlist.txt

Lines with the character o at least once:

grep -E 'o+' wordlist.txt

This means o with zero or more apparitions. So it will match everything:

grep -E 'o*' wordlist.txt

Search for one or another

grep 'succes|error' mesaje.txt

[grep] List lines that starts with a certain letter

Students whose name starts with G:

grep '^G' students.txt

[grep] List students with any secondary name that starts with F

grep '[ -]F[a-z]\+[^\.]' students.txt

[ -]F - space or - followed by F
[a-z]\+ - any a-z characters at least once. In standard grep (not extended) + should be prefixed with \
[^\.] - following character is not .

[ls] List files with .txt extension and a single character before extension

ls ?.txt

? - any unique character

[ls] List all files that starts with a, b or c and have the .txt extension

ls [abc]*.txt

[abc] - characters a, b or c
* - any number of characters (including 0)
.txt - ends with txt

[grep] List students with their father initial followed optionally by . in their name

grep -E ' [A-Z]\.? ' students.txt

\.? - optional dot

[tr] Convert spaces and tabs to comma

tr is used for translating or deleting characters.
cat students.txt | tr " " "," | tr "\t" ","

OR

cat students.txt | tr " \t" ","

OR (much more cleaner)

tr " \t" "," < students.txt

[tr] Convert lowercase string to uppercase

echo "hello world" | tr 'a-z' 'A-Z'

[tr] Delete characters from a to e

echo "hello world" | tr -d 'a-e'

[tr] Squeeze repetition of characters

echo "helloooo world" | tr -s 'o'

[sed] Replace entire text

sed (Stream Editor) is used for filtering and transforming text (not just characters like tr)
echo "hello world" | sed 's/world/universe/'

[sed] Replace first occurrence

echo "apa de apa" | sed 's/apa/apa minerala/'

s - init substitution command

[sed] Replace all occurrences

echo "apa de apa" | sed 's/apa/apa minerala/g'

g - substitution will be applied globally (for all occurrences on that line)

[sed] Delete lines

echo -e "hello\nworld" | sed '/world/d'

[sed] Insert text after a match

If line1 is found, it will insert line1.5 after it.

echo -e "line1\nline2" | sed '/line1/a\line1.5'

[sed] Edit the file in place

Replaces all occurrences of “old” with “new” in file.txt directly, modifying the file in place.
Use with caution and ideally with a backup.

sed -i 's/old/new/g' file.txt

i - in place

[cut] Print the first two columns (separated by space) from a file

cut is used to extract columns from a file delimited with a specific character. Default separator is tab (\t).
cut -d ' ' -f1,2 students.txt

-d ' ' - delimiter is space
-f1,2 - column numbers to extract (starts with index 1)

[cut] List only the first bytes from each line

cut -b 1,2,3 students.txt

cut -b 1-5 students.txt

[cut] List only specific characters from each line

cut -c 2,5,7 state.txt

cut -c 1-7 students.txt

[awk] Extract column using a custom separator

awk -F: '{print $1}' /etc/passwd

F - field separator

[awk] Use multiple delimiters to extract columns from a file

Another way to set delimiter is in the BEGIN block.

awk 'BEGIN{FS="[ \t]+"} {print $1}' students.txt

BEGIN - special block used to adjust the initial settings before executing awk.
FS - field separator.
[ \t]+ - one or more spaces or tabs.
$1 - first column

[awk] Skip first two lines

w | awk 'FNR>2 {print $1}'

FNR - file number record (internal awk variable holding current line number).

[awk] Skip first column - print just other columns not the first one

history | awk '{$1=""; print $0}'
{$1=""} - sets the first column (represented by $1) to an empty string, effectively removing it from the output. \\ 
print $0 - prints the entire line to the output.

[awk] Skip first two columns

history | awk '{$1=""; $2=""; print $0}'

[awk] Add a header to the output and display the entire file content

awk 'BEGIN {print "The File Contents:"} {print $0}' students.txt

awk 'BEGIN {print "The File Contents:"} {print $0} END {print "This is the end my friends."}' students.txt

[awk] Display first word (string followed by a space) from each line of a file

awk is similar with cut but much more advanced.
Default column separator is any whitespace (space, tab) and newline.
awk '{print $1}' students.txt

$1 - first column

[awk] Display last word (string followed by a space) from each line of a file

awk '{print $NF}' students.txt

NF - number of fields (last column number)

[awk] Display entire line and change some column values

echo "Hello Tom" | awk '{$2="Adam"; print $0}'

$0 - entire line

[awk] List students with notes greater than 5

awk '$5 > 5 {print $1, $2, $5}' students.txt

[awk] Compute the average of final grades

awk '{sum += $5} END {print "Average of final grades:", sum/NR}' students.txt

sum - custom variable to count the sum (it is not predefined).
$5 - fifth column.
END - block which will be executed after all lines from the file were processed.
NR - internal awk variable which stores the total number of lines.

[awk] Customize the columns output

Print - between the first and second column:

awk '{print $1, "-", $2}' students.txt

[awk] Multiply column value based on a condition

awk '{ if ($5>=5) { nr=$5*10} else { nr=$5 } print nr}' students.txt

[awk] Print only lines 5 and 7 from /etc/passwd

cat /etc/passwd | awk 'NR==5 || NR == 7 {print $0}'

[seq] Generate a sequence of numbers from 1 to 5

seq 1 5

[seq] Generate a sequence of numbers from 1 to 10, with an increment of 2

seq 1 2 10

[seq] Generate a sequence of numbers from 0.5 to 2.5, with an increment of 0.5

seq 0.5 0.5 2.5

[seq] Generates numbers with two decimal places

seq -f "%.2f" 1 3

f - format

[seq] Generate numbers with equal width (added 0 in front)

seq -w 1 10

w - equal width for all generated numbers

[sort] Sort numbers ascending order

sorts the text lines. By default it will sort as characters.
echo -e "5\n3\n8\n1" | sort -n

e - interpret escape sequences
n - numbers (sort as numbers)

[sort] Sort numbers descending order

echo -e "5\n3\n8\n1" | sort -nr

r - reverse (descending) order

[sort] Sort by a certain column

Sort numerically by second column.

echo -e "a 2\nc 1\nb 3" | sort -n -k2

k2 - sort by second column

Sort lexicographically by first column.
(k1 can be omitted here, sort will automatically sort lexicographically by the first column.)

echo -e "a 2\nc 1\nb 3" | sort -k1

[sort] Sort by multiple columns

cat multiple.txt | sort -k1,1 -k2,2gr

k1,1 - sort entire first column
k2,2 - sort entire second column
g - general numeric sort (sort better different kind of numbers for ex 2, 2.5, 3.55)
r - reverse

[sort] Unique sort - sort and remove duplicates

cat multiple.txt | sort -k1,1 -u

u - unique

[sort] Sorting with a specific separator

Default separator is tab (\t). We want to change it to :.

echo -e "a:2\nb:1\nc:3" | sort -t: -k2

t - change separator

[sort] Check if the column is sorted (without sorting it, just check)

echo -e "a 2\nc 1\nb 3" | sort -c

c - check

[uniq] Remove duplicated lines

sort file.txt | uniq

To match the lines case insensitive:

sort file.txt | uniq -i

[uniq] Remove duplicated lines and count occurences

sort file.txt | uniq -c

c - count

[uniq] Show only lines that appear multiple times

sort file.txt | uniq -d

d - duplicated

[uniq] Show only not duplicated lines

sort file.txt | uniq -u

u - unique

[sort] [uniq] [awk] Most used 10 history commands

history | awk '{$1=""; print $0}' | sort | uniq -c | sort -nr -k1 | head -10 | awk '{$1=""; print $0}'

[for] Start 10 processes in background then kill them

for i in {1..10}; do sleep 3000 & done; pkill sleep

[for] Create 15 files - their names contains numbers

Using a one-liner, create 15 files with names ranging from final-exam-00.txt to final-exam-15.txt.

for i in {0..15}; do touch final-exam-$(printf "%02d" $i).txt; done

[bash] All types of arguments in a bash script

arg.sh
#!/bin/bash
 
echo "All args"
 
echo "Using \$*:"
for arg in "$*"; do
    echo "$arg"
done
 
echo "Using \$@:"
for arg in "$@"; do
    echo "$arg"
done
 
echo "Using \$1, \$2, ..."
echo "Arg 1: $1", "Arg 2: $2"
 
echo "Script name is $0"
 
echo "Number of args received is $#"
 
echo "Printing all args using \$# and a for loop"
 
nr_args=$#
 
for (( i=1; i<=$nr_args; i++ ))
do
	eval arg=\${$i}
	echo "Arg $i is: $arg"
done

[bash] $var vs ${var}

[bash] Get the length (number of characters) of a variable

[bash] Extract a substring from a variable

Extract from position 7 with a length of 5 characters:

[bash] Command substitution - $() structure

The $(…) syntax in Bash is used for command substitution. It allows the output of a command to replace the command itself in a line of script or command input. This is particularly useful when you need to use the output of one command as an argument for another command, or when assigning the output of a command to a variable.

Execute a command based on the condition of another command:

if [[ $(grep -c "searched_word" fisier.txt) -gt 0 ]]; then
    echo "Word was found in the file."
else
    echo "Word was not found, sorry."
fi

Iterate through files:

for file in $(ls *.txt); do
    echo "Processing txt file: $file"
done

Nesting commands (find kernel version):

[bash] [while-read] Read a file and assign a variable for each column

while-read.sh
#!/bin/bash
 
IFS=$'\t'
while read name group final_grade test_grade practical_grade; do
    echo "$name --> $group --> $final_grade --> $test_grade --> $practical_grade"
done < students.txt

IFS - defines the separator, here tab (\t).

[bash] [while-read] Read a file and adjust values based on a condition

while-read-cond.sh
#!/bin/bash
 
IFS=$'\t'
while read name group final_grade test_grade practical_grade; do
	if test "$final_grade" -gt 5; then
        	echo "$name --> $group --> Good job, final grade is $final_grade --> $test_grade --> $practical_grade"
    	fi
	echo "$name --> $group --> $final_grade --> $test_grade --> $practical_grade"
done < students.txt

[bash] [for] Script for manipulating all files from a folder (print information about them, make backups)

manip.sh
#!/bin/bash
 
for file in "$1"/*
do
    if test -f "$file"; then
        stat -c "%a %F %n" "$file"
        cp "$file" "$file.bkp"
    fi
done

%a - file permission in numeric form.
%F - file type.
%n - complete file name.

[bash] [rm] Script that removes itself

nothing.sh
#!/bin/bash
 
rm -- "$0"

– - end of command line options.
“$0” - the name of the script.

[bash] [for] Script to print the sum of the received arguments, error if no argument

dumb-sum.sh
#!/bin/bash
 
nr_args=$#
 
if [ "$nr_args" -eq 0 ]; then
    echo "Error: No arguments provided. Usage: $0 <arg1> <arg2> ..."
    exit 1
fi
 
sum=0
 
for (( i=1; i<=$nr_args; i++ ))
do
	eval arg=\${$i}
	 sum=$((sum + arg))
done
 
echo "Sum is: $sum"
$((sum + arg)) is to evaluate arithmetic expression.

[bash] Advanced dictionary lookup script

Script to download the wbritish package (contains dictionaries) and which should validate the words from in.txt file located at in.txt with the words from the dictionary.
Example:

./lookup.sh in.txt
apple in dictionary
noapple not in dictionary

Lookup script with the in.txt file downloaded locally (using wget http://elf.cs.pub.ro/uso/res/final/146/in.txt)

lookup.sh
#!/bin/bash
 
if [ "$#" -ne 1 ]; then
    echo "Usage: $0 <input_file>"
    exit 1
fi
 
input_file=$1
 
if ! dpkg -l | grep -qw wbritish; then
    echo "Installing wbritish package."
    sudo apt-get update && sudo apt-get install -y wbritish
fi
 
if [ ! -f "$input_file" ]; then
    echo "File does not exist: $input_file"
    exit 2
fi
 
while IFS= read -r word; do
    if look "$word" /usr/share/dict/british-english > /dev/null; then
        echo "$word in dictionary"
    else
        echo "$word not in dictionary"
    fi
done < "$input_file"

Output:

./lookup.sh in.txt
mine in dictionary
yours not in dictionary
apples in dictionary
notSoApples not in dictionary

Lookup script that will download the in.txt file from the remote location:

lookup.sh
#!/bin/bash
 
if [ "$#" -ne 1 ]; then
    echo "usage: $0 <input_file>"
    exit 1
fi
 
input_file_url=$1
input_file="in.txt"
 
wget -O "$input_file" "$input_file_url" || { echo "Eroare downloading input file."; exit 2; }
 
if ! dpkg -l | grep -qw wbritish; then
    echo "wbritissh not installed, installing"
    sudo apt-get update && sudo apt-get install -y wbritish
fi
 
if [ ! -f "$input_file" ]; then
    echo "input file does not exist: $input_file"
    exit 2
fi
 
#read each word from input file and check if it is present in dictionary
 
while IFS= read -r word; do
    if look "$word" /usr/share/dict/british-english > /dev/null; then
        echo "$word in dictionary"
    else
        echo "$word not in dictionary"
    fi
done < "$input_file"

[bash] Advanced disk usage script

Create a script named disk-usage.sh that reads a number from stdin and displays the partition that has a disk usage percentage greater than that number.
If the input received is not a number, use a default threshold value chosen by you.

Usage example:

$ ./disk-usage.sh
Threshold = 12
WARNING: Partition "/dev/nvme0n1p6" used 13%
WARNING: Partition "/dev/nvme0n1p1" used 27%

$ ./disk-usage.sh
Threshold = ceva
WARNING: Partition "/dev/nvme0n1p6" used 30%
WARNING: Partition "/dev/nvme0n1p1" used 31%
disk-usage.sh
#!/bin/bash
 
# User should input the threshold
echo "Enter the threshold (percent):"
read threshold
 
# Check if the threshold is a number
if ! [[ "$threshold" =~ ^[0-9]+$ ]]; then
    echo "Input is not a number, using default threshold: 20."
    threshold=20
else
    echo "Threshold = $threshold"
fi
 
df -h | awk -v threshold="$threshold" 'NR>1 {gsub(/%/,"",$5); if($5 > threshold) print "WARNING: Partition \""$1"\" used "$5"%"}'

[bash] [sec] [hash] Script to hash two arguments - string

Create a script that takes two arguments (strings). The script hash_it.sh concatenates the two arguments and displays their SHA256 hash sum.

hash_it.sh
#!/bin/bash
 
nr_args=$#
 
if [ "$nr_args" -ne 2 ]; then
    echo "Error: two args should be present. Usage: $0 <arg1> <arg2>"
    exit 1
fi
 
result="$1$2"
echo -n $result | openssl dgst -sha256

[bash] [sec] [hash] Script to read all lines from a file and print the SHA256 sum of each

Suppose the input file is called parole.txt.

hash_it_all.sh
#!/bin/bash
 
IFS=$'\n'
while read password; do
    echo -n $password | openssl dgst -sha256 | cut -d '=' -f2 | awk '{print $1}'
done < parole.txt

[bash] [ps] Script to display every second the process with highest CPU usage

top.sh
#!/bin/bash
while true; do
    clear
    #echo "Update on: $(date)"
    echo "Process with highest CPU usage:"
    ps -eo %cpu,pid,command --sort=-%cpu | head -n 2
    sleep 1
done

[git] Configure initial settings

git config --global user.name "Joe Black"
git config --global user.email "joe.black@github.com"
git config --global init.defaultBranch main

[git] Create an empty local repository in the current directory

git init

git remote add origin https://github.com/odefta/array-sorting-algorithms.git

The sync between the local repo and the remote (github) repo is made with pull (get) and push (put) commands.

[git] Check if the remote repo was configured correctly

git remote -v

[git] Get all branches and history from remote repo

git fetch origin
git pull origin main

[git] Get the current branch

git branch --show-current
git branch
git status

[git] Verify log history

git log

[git] Verify log history - show last commits only

Show only the last 4 commits:

git log -n 4

Show only the last 4 commits - more compact form:

git log -n 4 --oneline

[git] Add / commit a file on the current branch to the local repository

Create a file, and run git status:

echo "Hello World" > hello.txt
git status

We have an untracked file we can add it (in staging area - which is a pre-commit step):

git add hello.txt

Use . to add all files (from current dir - which includes sub-dirs).

Now we can commit the hello.txt to the local repo:

git commit -m "Add hello.txt with hello world"

m - add a comment

After modifying a file (hello.txt) we can issue again a commit.

[git] Push the commit to the remote repo

git push origin main

[git] Adding a new branch

git branch add-gitignore

Check if the new branch was added:

git branch

[git] Switch to other branch

git checkout add-gitignore

Check

git branch

[git] Merge two branches

First, let's commit something to our current branch add-gitignore and sync with the remote:

echo "*.o" > .gitignore
git add .gitignore
git commit -m "Add .gitignore file"
git push origin add-gitignore

Now we want to merge our branch main with the add-gitignore branch (to get the .gitignore file).

First, switch to the main branch and get the changes from the remote:

git checkout main
git pull origin main

Now we are ready to merge our current branch main with add-gitignore:

git merge add-gitignore

Push the changes to the remote:

git push origin main

[git] Remove a branch both locally and remote

We want to delete our secondary branch add-gitignore.

First, we need to be on another branch (main).

git checkout main

Remove local branch.

git branch -d add-gitignore

Remove remote branch.

git push origin --delete add-gitignore

[git] Switch to a remote branch (without creating a local one with the same name)

Remote branch is test.

git fetch origin
git branch -r (verify available remote branches)
git checkout -t origin/test

[user] Check the current user

whoami

id

[user] Check another user

sudo id root

[user] Display all logged-in users

who

To display only the user name:

who | cut -d ' ' -f1

To display additional info:

w

To filter only the logged-in users:

w | awk 'FNR>2 {print $1}'

[user] Display the users who have opened a terminal on the host system

who | awk '$2 ~ /^tty/ {print $1}'
~ - equals for regular expressions
/ / - delimitates a regular expression
^tty - starts with tty.

[user] Get only the UID for a certain user

This will print all info.

id syslog

This will print the first column.

id syslog | awk '{print $1}'

This will get the uid value after =.

id syslog | awk '{print $1}' | cut -d= -f2

Finally, this will get the actual uid value.

id syslog | awk '{print $1}' | cut -d= -f2 | cut -d'(' -f1

[user] Switch user

Switch to another user, you should know only your password. User should be present in sudoers file.

sudo su <user_name>

Switch to another user, you should know only your password.

sudo su - <user_name>

- will use the environment variables of that new user.

Switch to another user, you should know the password of the new user.

su <user_name>

Switch to another user, simulate initial login (-i).

sudo -u <user_name> -i

Start a new login session.

sudo login <user_name>

[user] Create a new user

Simplest and interactive way:

sudo adduser joe

We can add the existing user joe to the joegroup:

sudo adduser joe joegroup

Advanced, with more options (specify home directory, default shell, default group).

sudo groupadd joegroup
sudo useradd -m -d /home/joe -s /bin/bash -g joegroup joe

m - create home directory.
d - path to the home directory.
s - set implicit shell.
g - set primary group.

To add also secondary groups:

sudo groupadd joesecond
sudo groupadd joethird
sudo useradd -m -d /home/joe -s /bin/bash -g joegroup -G joesecond,joethird joe

G - secondary groups separated by comma.

Add / change the joe's password:

sudo passwd joe

[user] Modify an existing user

Add secondary groups to the user:

sudo groupadd joefourth
sudo groupadd joefive
sudo usermod -a -G joefourth,joefive joe

a - add (important!), without it, the user will be eliminated from the other secondary groups.

Modify the home directory:

sudo usermod -d /home/newjoe joe

d - dir name.

Change the joe's default shell (First, inspect the /etc/shells file for a list of all available shells)

sudo usermod -s /bin/sh joe

Same effect can be obtained with

sudo chsh -s /bin/sh joe

Modify the joe's primary group:

sudo usermod -g joesecond joe

[user] Remove an user

Remove joe, but keep its files.

sudo userdel joe

Remove joe with all its files (from home dir and other places).

sudo userdel -r joe

r - recursive

[user] Block an user to authenticate

sudo usermod -s /usr/sbin/nologin joe

[user] View user details

finger joe

[user] Change mode

read r - 4 write w - 2 execute x - 1

Owner read and write, group read, other no permission:

chmod 640 file.txt
chmod u=rw,g=r,o= file.txt

Add execute right for others:

chmod 641 file.txt
chmod o+x file.txt

Eliminate read permission for group:

chmod 601 file.txt
chmod g-r file.txt

Eliminate write permission for all:

chmod ugo-w file.txt

[user] Change owner

Change the owner to joe user and joegroup group.

sudo chown joe:joegroup file.txt

Change only the owner (let the group intact).

sudo chown joe file.txt

Change the owner of all files in a directory (recursive) - let the group intact.

sudo chown -R joe dir

Change the owner of all files in a directory (recursive) - change also the group.

sudo chown -R joe:joegroup dir

[user] Just two users should have right to modify files from a folder

Suppose those 2 users are uso and so. Folder path is /uso.

sudo groupadd usogroup
sudo usermod -aG usogroup uso
sudo usermod -aG usogroup so
sudo chgrp -R usogroup /uso
sudo chmod -R 770 /uso

[user] Generate passwords

Generate 5 password of length 10, one per line, each containing at least one uppercase letter.

for i in {1..5}; do
    echo $(openssl rand -base64 9 | tr -dc 'a-zA-Z0-9' | cut -c 1-9)$(tr -dc 'A-Z' < /dev/urandom | head -c 1)
done

Same thing using pwgen tool.

sudo apt install pwgen
pwgen -c -n -B 10 5 | tr ' ' '\n'

c (–capitalize) - at least one capital letter will be included.
n (–numerals) - at least one number will be included.
B (–ambiguous) - eliminate ambiguous characters (like 0 and O).
10 - length of the password.
5 - generate 5 password.

[sec] [base64] Encoding / decoding a string using base64

Encoding.

echo -n "Base64 is not encryption!" | base64

QmFzZTY0IGlzIG5vdCBlbmNyeXB0aW9uIQ==

Decoding.

echo -n "QmFzZTY0IGlzIG5vdCBlbmNyeXB0aW9uIQ==" | base64 -d

d - decoding
Base64 is not encryption!

[sec] [base64] Encoding / decoding an entire file using base64

Encoding.

base64 file.txt > file64.txt

Decoding.

base64 -d file64.txt > file_dec.txt

[sec] [openssl] Encrypting / decrypting a file using OpenSSL

Encrypt the file file.txt to file file.enc

echo -n "plaintext content" > file.txt
openssl aes-256-cbc -in file.txt -out file.enc -pass pass:"my secret" -pbkdf2 -iter 10000

aes-256-cbc - encrypt the file using AES-256 in CBC mode
in - input file (plain text)
out - output file (encrypted)
pass - sets the passphrase (will be used for decryption)
pbkdf2 - selects PBKDF2 for key derivation, enhancing security against brute-force attacks
iter 10000 - sets the number of iterations in the key derivation, increasing the difficulty of cracking the encryption.

Decrypt the file file.enc to decrypted_file.txt.

openssl aes-256-cbc -d -in file.enc -out decrypted_file.txt -pass pass:"my secret" -pbkdf2 -iter 10000

d - decrypt

[sec] [hash] Hash a string

Hashing transforms data into a fixed-size string of characters, typically a hash code. This process is useful for verifying data integrity, securely storing passwords, and indexing data in databases. A hash function takes input data and produces a unique hash value; any modification to the data changes the hash significantly.

Using md5 (less secure) - two ways of achieving the same result.

echo -n "plaintext content" | md5sum

f88d6c9eab9c6a2ef3cfd3c59832b4d6 -

echo -n "plaintext content" | openssl dgst -md5

MD5(stdin)= f88d6c9eab9c6a2ef3cfd3c59832b4d6

Using SHA256 (much more secure - less collision issues).

echo -n "plaintext content" | openssl dgst -sha256

SHA2-256(stdin)= d800efe86c80b6054cff4850f3044d73e979429809f28029e26cd9c4f6d6cea8

[sec] [hash] Hash an entire file

file.txt content: plaintext content

Using md5.

md5sum file.txt

88d6c9eab9c6a2ef3cfd3c59832b4d6 file.txt

Using SHA256.

sha256sum file.txt

d800efe86c80b6054cff4850f3044d73e979429809f28029e26cd9c4f6d6cea8 file.txt

[sec] [password cracking] Crack linux users passwords using John The Ripper

A good dictionary (word list) can be downloaded from here: rockyou.txt

Install John The Ripper tool.

sudo apt install john

Merge the Linux system files /etc/passwd and /etc/shadow to create a single file (mypasswd.txt) containing user accounts and their hashed passwords.

sudo unshadow /etc/passwd /etc/shadow > mypasswd.txt

Create a word list used to guess the password.

echo -ne "school\nuniversity\nstudent" > wordlist.txt

n - avoid printing the newline
e - interpret backslash escapes.

Now we are ready to run John The Ripper.

john --wordlist=wordlist.txt mypasswd.txt

Finally, crack the passwords:

john --format=crypt mypasswd.txt

View the actual passwords (right after username):

john --format=crypt mypasswd.txt --show

[sec] [password cracking] Crack a zip archive password using John The Ripper

hashfile contains extracted password hashes.
This will use the default dictionary (word list) file.

john --format=zip hashfile

To use a custom dictionary file:

john --format=zip hashfile --wordlist=path/to/your/wordlist.txt

[sec] [password cracking] Crack a zip archive password using fcrackzip

Install the fcrackzip tool.

sudo apt install fcrackzip 

Brute-force guessing the password. We know it has 4 characters.

fcrackzip -v -l 1-4 -u secret_brute_force.zip

v - verbose
l 1-4 - pwd length from 1 to 4
u - extract the zip archive when the password is found.

Use a dictionary to crack the password.

fcrackzip -u -D -p wordlist.txt secret.zip

u - extract the zip archive when the password is found.
D - enables dictionary mode.
p - path to the dictionary file.

[sec] [hash cracking] Find a hidden message from a file with hashes (one per line)

Suppose we have my-hashes.txt:

F5D6C6C424929713C297514199F7CA41
AAD24C09BF816BBC6EA29C35DF52F896
F95B70FDC3088560732A5AC135644506
6EF5FA162DA08580BF5D1F88520F7388
CBB184DD8E05C9709E5DCAEDAA0495CF
1BC29B36F623BA82AAF6724FD3B16718

We'll take each line, put it in crackstation.net
The decrypted info from each line will form the final message.

linux/ubuntu/useful-commands.txt · Last modified: 2024/02/08 17:12 by odefta