====== Useful commands - linux / ubuntu ======
===== [file] Show file type =====
file file.txt
===== [cat] Add line numbers to a file =====
cat -n file.txt
nl file.txt
===== [tree] Show tree of files =====
tree -F -L 1 /
-F - show file type \\
-L - level (how nested should be), 1 - show only files under first directory level.
===== [ln] Symbolic link =====
ln -s lab02/.Hulk hulk
This will create file hulk as a shortcut for the (original) file lab02/.Hulk
===== [tar] Create a tar archive =====
tar -cvf archive.tar decrypted_file.txt encrypted_file.enc dir2
c - create \\
v - verbose\\
f - file name of the archive
===== [tar] Add a new file to the tar archive =====
tar -uf documente.tar raport_nou.txt
tar -rf documente.tar raport_nou.txt
-u - update \\
-r - add (does not check if the file is newer)
===== [tar] View tar archive content =====
tar -tf archive.tar
t - table (view archive content as a 'table') \\
f - file name (archive name) \\
===== [tar] Extract tar content =====
tar -xvf archive.tar
x - extract \\
v - verbose \\
f - file
===== [tar] Create a tar.gz archive =====
tar -czvf archive.tar.gz decrypted_file.txt encrypted_file.enc dir2
z - compressed gzip \\
c - create \\
v - verbose \\
f - file name of the archive
===== [gunzip] Add a file to existing tar.gz archive =====
This cannot be done directly using tar tool, but we will use gunzip to extract the tar archive, add a new file to it and then gzip it again.
gunzip archive.tar.gz && tar -rf archive.tar newfile.txt && gzip archive.tar
===== [tar] View tar.gz archive content =====
tar -tzf archive.tar.gz
t - table (view archive content as a 'table') \\
z - compressed gzip \\
f - file name (archive name)
===== [tar] Extract tar.gz content =====
tar -xzvf archive.tar.gz
x - extract \\
v - verbose \\
f - file
===== [zip] Create a zip archive =====
zip archive.zip file1.txt file2.txt
===== [zip] Add a new file to the zip archive =====
zip -u archive.zip new_file
-u - update (or add if not already in the archive)
===== [zip] View zip archive contents =====
zip -sf archive.zip
zip --show-files archive.zip
===== [zip] Extract a zip archive =====
unzip archive.zip -d dir
-d - optional destination directory, if missing the current directory will be used.
===== Find a file / directory =====
==== [find] Find all txt files in the users home directory ====
find ~ -name "*.txt"
==== [find] Find insensitive ====
sudo find / -iname "*ssh*conf"
==== [find] Find all files > 100 MB ====
find ~ -type f -size +100M
==== [find] Find and remove all files in /tmp directory ====
find /tmp -type f -name "*.tmp" -exec rm {} \;
{} - placeholder for each file, it will contain the full path name \\
\; - marks the end of the command
==== [find] Find executable files ====
find . -type f -executable
==== [find] Find all empty directories (and subdirectories) ====
find . -type d -empty
==== [find] Find all files that contain at least one digit (number) from a directory ====
find /usr/include -name '*[0-9]*'
{{:linux:ubuntu:pasted:20240207-200334.png}}
===== Locate a file (searching in database instead of actual files on disk) =====
==== [locate] Installing the locate tool ====
sudo apt install mlocate
sudo updatedb
==== [locate] Locate a file ====
locate "*ssh*conf"
==== [locate] Locate a file insensitive ====
locate -i "Example.txt"
==== [locate] Locate only the first 10 results ====
locate example.txt | head -10
===== [ps] List all active processes =====
ps -ef
e - 'every' - we want to see every process \\
f - 'full' - get full details
ps aux
a - all users \\
u - user oriented format output (display user who owns the process, CPU, memory, etc.) \\
x - also includes the processes that are not attached to a terminal.
{{:linux:ubuntu:pasted:20240206-172056.png?0x300}}
===== [ps] List all processes of a certain user =====
ps -u username
===== [ps] List details of a specific process =====
Change 2638 with the actual process pid.
ps -p 2638
===== [ps] List processes based on cpu usage =====
ps -aux --sort=-%cpu
===== [ps] List processes based on memory usage =====
ps -aux --sort=-%mem
===== [ps] List processes with custom columns =====
ps -e -ouser,uid,pid,%mem,%cpu,rss,cmd --sort=%mem
===== [ps] List processes - descending sorting =====
Just add a - in the --sort.
ps -e -ouser,uid,pid,%mem,%cpu,rss,cmd --sort=-%mem | less
===== [ps] Search for a specific process =====
Replace gnome with your keyword.
ps -ef | grep gnome
===== [pstree] List the process tree =====
pstree
pstree -u
pstree -p
u - show users
p - show pid
===== Get the pid of a process =====
Multiple methods:
pgrep nginx
ps aux | grep nginx | grep -v grep
pidof nginx
===== [apt] APT commands =====
sudo apt install nginx
sudo apt search nginx
sudo apt remove nginx
sudo apt purge nginx
sudo apt update
sudo apt upgrade
===== [ip] Check if a network card is active =====
sudo ip link show
===== [ip] UP / DOWN states of network interfaces =====
Display only the names and states (UP / DOWN) of the network interfaces on the system, using the format interface_name:state.
ip link show | awk '/^[0-9]+:/ {print $2 " " $9}' | sed 's/://g' | awk '{print $1 ":" ($2=="UP"?"UP":"DOWN")}'
Output:
lo:DOWN
ens33:UP
ens34:UP
docker0:DOWN
br1:DOWN
br2:DOWN
===== [ip] Start a network interface =====
ip link set dev eth1 up
===== [ip] View IPs on all network interfaces =====
sudo ip address show
===== [ip] Remove a network configuration from an interface =====
sudo ip address flush eth1
===== [ping] Check if a station is alive =====
ping -c 4 10.125.50.50
c - number of packets to send (by default is infinite)
===== [ip] Show routing table / find default gateway =====
ip route show
Default gateway (192.168.112.2) is listed under the 'default via' section:
{{:linux:ubuntu:pasted:20240206-175930.png}}
To extract just the default gateway ip:
ip route show | awk 'NR==1 {print $3}'
{{:linux:ubuntu:pasted:20240208-024419.png}}
===== [ip] [nmcli] Find DNS Servers =====
nmcli dev show | grep DNS
{{:linux:ubuntu:pasted:20240206-180303.png}}
resolvectl status | grep 'Current DNS'
{{:linux:ubuntu:pasted:20240206-180425.png}}
To print only IPs:
nmcli dev show | grep DNS | cut -d ':' -f2 | tr -d ' '
===== [ip] [host] Get the IP address of a host / Check if DNS service is working =====
host elf.cs.pub.ro
{{:linux:ubuntu:pasted:20240206-180627.png}}
===== Update DNS Servers =====
Change the /etc/resolv.conf:
nameserver 8.8.8.8
nameserver 8.8.4.4
===== [netstat] Display open UDP ports =====
sudo netstat -ulpn
u - udp ports only \\
l - listening state ports only \\
p - program name / PID that has opened the port \\
n - shows the actual IP address on which the system is listening for connections, instead of resolving the host name.
sudo ss -ulpn
===== [netstat] Display open TCP ports =====
sudo netstat -tlpn
t - TCP
sudo ss -tlpn
===== [netstat] Display both TCP and UPD ports =====
sudo netstat -tulpn
t - TCP \\
u - UDP
sudo ss -tulpn
===== [nc] Establish a TCP communication with an application - Netcat =====
nc google.com 80
{{:linux:ubuntu:pasted:20240206-182806.png}}
===== [wget] Download a single file from a HTTP(s) / FTP server =====
wget elf.cs.pub.ro
Download in background with a custom file name:
wget -b -O google.html http://google.com
b - run in background \\
O - output file name.
===== [curl] Show only HTTP Header of a site =====
curl -I -L http://ocw.cs.pub.ro
I (or --head) - send only HTTP HEAD request instead of HTTP GET \\
L - follow any redirectrion.
===== [ssh] Connect to a SSH server =====
ssh user@192.168.1.8
===== [ssh] Connect to a SSH server and run a command =====
ssh root@10.10.10.3 ip address show
===== [scp] Send / copy a file to a remote machine =====
scp /home/file.txt student@10.10.10.3:~/
===== [scp] Receive / download a file from a remote machine =====
scp student@10.10.10.3:/home/file.txt ~/
===== [scp] Send / copy a directory to a remote machine =====
scp -r /home/dir student@10.10.10.3:~/
===== [scp] Receive / download a file from a remote machine =====
scp -r student@10.10.10.3:/home/dir ~/
===== [ssh][public key] SSH Public Key authentication =====
ssh-keygen -t rsa -b 4096
ssh-copy-id root@10.10.10.3
What was generated:
* ~/.ssh/id_rsa --> private key
* ~/.ssh/id_rsa.pub --> public key
===== [ls] Escaping globbing patterns / special characters =====
Just enclose the string in ""
ls support-globbing/"[HOME]"*
find . -name "*search*"
===== Get memory information =====
cat /proc/meminfo | grep "Mem"
{{:linux:ubuntu:pasted:20240206-190522.png}}
===== Get CPU information =====
cat /proc/cpuinfo
===== [grep] Search for a string in a file =====
cat workspace/C/searching/binary_search.c | grep search
grep search workspace/C/searching/binary_search.c
grep 'error' log.txt
===== [grep] Case insensitive search for a string in a file =====
grep -i 'string_to_search' file.txt
===== [grep] Count number of lines of searched string in file =====
grep -c 'cache' /proc/cpuinfo
===== [grep] Count number of occurrences (not just lines) of searched string in file =====
grep -o 'searched_string' file.txt | wc -l
o - print each occurrence on a separate line \\
l (from wc) - count lines
===== [grep] Recursive search for a string in a directory =====
grep -r 'cache' /proc 2>/dev/null
2>/dev/null - don't show errors
===== [grep] Recursive (and insensitive with line number displayed) search for a string in a directory =====
grep -nri search workspace/C/ | less
n - show line number \\
r - recursive \\
i - case insensitive
===== [grep] Recursive search for a word in a directory =====
grep -nri -w "search" workspace/C/ | less
===== [grep] Display lines that does NOT contain the searched string =====
Show all lines that does not contain error string:
grep -v 'error' log.txt
===== [grep] Display lines that starts with a certain string =====
Search for lines that starts with error string:
grep '^error' log.txt
===== [grep] Display a certain number of lines AFTER matching a string in a file =====
Displays each line that contains error, followed by 3 lines:
grep -A 3 'error' log.txt
===== [grep] Display a certain number of lines BEFORE matching a string in a file =====
Displays two lines before each line that contains error:
grep -B 2 'error' log.txt
===== [grep] Display a certain number of lines BEFORE and AFTER matching a string in a file =====
Displays two lines before and after each line that contains error:
grep -C 2 'error' log.txt
===== [grep] Search in a file for one string OR another using ERE (Extended Regular Expressions) =====
grep -E -i "error(s)?|warning(s)?" example.txt
E - extended regular expression
i - case insensitive
error(s) - search for strings that contains error or errors
| - or
warning(s) - search for string that contains warning or warnings
? - makes that previous element (s) to be optional.
{{:linux:ubuntu:pasted:20240206-203038.png}}
===== [grep] Search in a file for an exact match of string OR another using ERE (Extended Regular Expressions) =====
grep -E -i "\berror(s)?|warning(s)?\b" example.txt
\b - the match will be made at the start and at the end of a word.
{{:linux:ubuntu:pasted:20240206-203834.png}}
===== [grep] Add an alias - go for xdg-open =====
Instead of running xdg-open https://google.com, we want to use go https://google.com. \\
For this we need to define an alias in **~/.bash_aliases** (this is read from **~/.bashrc**). \\
So we need to append line: alias go='xdg-open'
grep -qxF 'alias go='\''xdg-open'\''' ~/.bash_aliases || echo 'alias go='\''xdg-open'\''' >> ~/.bash_aliases
|| - the command after this OR operator will be executed if the command before it fails.
q - run in silent mode, without printing anything
x - search on the ENTIRE line
F - disable regular expression, the string is a simple text.
>> - append to the end.
Next, reload the ~/.bashrc file:
source ~/.bashrc
===== [;] Execute multiple commands in the same line, one by one =====
mkdir demo; cd demo; touch Hello; ls
===== [&&] Execute a command on the same line if the PREVIOUS one SUCCEEDED =====
mkdir operators/demo && cd operators/demo
===== [||] Execute a command on the same line if the PREVIOUS one FAILED =====
ls -d operators || mkdir operators
===== [find] Print info about files in the current directory using FIND with -exec =====
find . -maxdepth 1 -type f -exec ls -l {} \;
===== [find] Print info about files in the current directory using FIND and PIPE =====
find . -maxdepth 1 -type f -print0 | xargs -0 ls -l
-print0 - finish each string found with a null character instead of newline. This helps in manipulating files that contains spaces or newlines. \\
0 - treat received input as a list of args separated by the null character - used to correctly processed the output of print0.
===== [find] Find all files from /lib that does not start with the lib prefix =====
find -L /lib -type f -not -name 'lib*.*'
L - follow symlinks (this is necessary because lib is a symlink for usr/lib) \\
not - negate the name
===== [xargs] Move all files (requiring confirmation) with a certain extension from the current folder to parent =====
ls *.c | xargs -I {} -p mv {} ..
I {} - placeholder / replacer for each character \\
p - requires user confirmation
Safer version (handles better the files with spaces):
find . -maxdepth 1 -type f -name "*.c" -exec mv {} .. \;
===== [1> 2>] Redirect STDOUT and STDERR =====
Redirect both in firefox-ignore file.
firefox &> firefox-ignore
Another way:
firefox > firefox-ignore 2>&1
To redirect input in a file and output to another file:
firefox > firefox-output.txt 2> firefox-error.txt
OR
firefox 1> firefox-output.txt 2> firefox-error.txt
> is equivalent to 1>
===== [dd] Generate 100 MB file with null values =====
dd if=/dev/zero of=null-100mb count=100 bs=1M
if=/dev/zero - input file will get data from /dev/zero \\
of=null-100mb - name of the output file \\
count=100 - number of blocks that will be copied \\
bs=1M - size of each block
===== [dd] Generate 100 MB file with random values =====
dd if=/dev/urandom of=rand-100mb count=100 bs=1M
===== [grep] Simple Regular Expressions Examples =====
Search for lines that starts with Hello:
grep '^Hello' message.txt
Search for lines that ends with ending:
grep 'ending$' message.txt
Search any lines that contains e followed by any character and then r:
grep 'e.r' message.txt
Search lines that contains sc followed by h or y and the o:
grep 'sc[hy]o' wordlist.txt
Search lines that contains sc followed any character except h or y and the o:
grep 'sc[^hy]o' wordlist.txt
Lines with the character o at least once:
grep -E 'o+' wordlist.txt
This means o with zero or more apparitions. So it will match everything:
grep -E 'o*' wordlist.txt
Search for one or another
grep 'succes|error' mesaje.txt
===== [grep] List lines that starts with a certain letter =====
Students whose name starts with G:
grep '^G' students.txt
{{:linux:ubuntu:pasted:20240206-223122.png}}
===== [grep] List students with any secondary name that starts with F =====
grep '[ -]F[a-z]\+[^\.]' students.txt
[ -]F - space or - followed by F \\
[a-z]\+ - any a-z characters at least once. In standard grep (not extended) + should be prefixed with \ \\
[^\.] - following character is not . \\
{{:linux:ubuntu:pasted:20240206-224337.png}}
===== [ls] List files with .txt extension and a single character before extension =====
ls ?.txt
? - any unique character
{{:linux:ubuntu:pasted:20240206-225028.png}}
===== [ls] List all files that starts with a, b or c and have the .txt extension =====
ls [abc]*.txt
[abc] - characters a, b or c \\
* - any number of characters (including 0) \\
.txt - ends with txt
===== [grep] List students with their father initial followed optionally by . in their name =====
grep -E ' [A-Z]\.? ' students.txt
\.? - optional dot
===== [tr] Convert spaces and tabs to comma =====
tr is used for translating or deleting characters.
cat students.txt | tr " " "," | tr "\t" ","
OR
cat students.txt | tr " \t" ","
OR (much more cleaner)
tr " \t" "," < students.txt
{{:linux:ubuntu:pasted:20240206-232944.png}}
===== [tr] Convert lowercase string to uppercase =====
echo "hello world" | tr 'a-z' 'A-Z'
{{:linux:ubuntu:pasted:20240206-231910.png}}
===== [tr] Delete characters from a to e =====
echo "hello world" | tr -d 'a-e'
{{:linux:ubuntu:pasted:20240206-232023.png}}
===== [tr] Squeeze repetition of characters =====
echo "helloooo world" | tr -s 'o'
{{:linux:ubuntu:pasted:20240206-232241.png}}
===== [sed] Replace entire text =====
sed (Stream Editor) is used for filtering and transforming text (not just characters like tr)
echo "hello world" | sed 's/world/universe/'
{{:linux:ubuntu:pasted:20240206-233424.png}}
===== [sed] Replace first occurrence =====
echo "apa de apa" | sed 's/apa/apa minerala/'
s - init substitution command
{{:linux:ubuntu:pasted:20240206-234812.png}}
===== [sed] Replace all occurrences =====
echo "apa de apa" | sed 's/apa/apa minerala/g'
g - substitution will be applied globally (for all occurrences on that line)
{{:linux:ubuntu:pasted:20240206-234958.png}}
===== [sed] Delete lines =====
echo -e "hello\nworld" | sed '/world/d'
{{:linux:ubuntu:pasted:20240206-233713.png}}
===== [sed] Insert text after a match =====
If line1 is found, it will insert line1.5 after it.
echo -e "line1\nline2" | sed '/line1/a\line1.5'
{{:linux:ubuntu:pasted:20240206-234037.png}}
===== [sed] Edit the file in place =====
Replaces all occurrences of "old" with "new" in file.txt directly, modifying the file in place. \\ Use with caution and ideally with a backup.
sed -i 's/old/new/g' file.txt
i - in place
===== [cut] Print the first two columns (separated by space) from a file =====
cut is used to extract columns from a file delimited with a specific character.
Default separator is tab (\t).
cut -d ' ' -f1,2 students.txt
-d ' ' - delimiter is space \\
-f1,2 - column numbers to extract (starts with index 1)
{{:linux:ubuntu:pasted:20240207-000923.png}}
===== [cut] List only the first bytes from each line =====
cut -b 1,2,3 students.txt
{{:linux:ubuntu:pasted:20240207-001501.png}}
cut -b 1-5 students.txt
{{:linux:ubuntu:pasted:20240207-001534.png}}
===== [cut] List only specific characters from each line =====
cut -c 2,5,7 state.txt
{{:linux:ubuntu:pasted:20240207-001816.png}}
cut -c 1-7 students.txt
{{:linux:ubuntu:pasted:20240207-001905.png}}
===== [awk] Extract column using a custom separator =====
awk -F: '{print $1}' /etc/passwd
F - field separator
===== [awk] Use multiple delimiters to extract columns from a file =====
Another way to set delimiter is in the BEGIN block.
awk 'BEGIN{FS="[ \t]+"} {print $1}' students.txt
BEGIN - special block used to adjust the initial settings before executing awk. \\
FS - field separator. \\
[ \t]+ - one or more spaces or tabs. \\
$1 - first column
{{:linux:ubuntu:pasted:20240207-004733.png}}
===== [awk] Skip first two lines =====
w | awk 'FNR>2 {print $1}'
FNR - file number record (internal awk variable holding current line number).
===== [awk] Skip first column - print just other columns not the first one =====
history | awk '{$1=""; print $0}'
{$1=""} - sets the first column (represented by $1) to an empty string, effectively removing it from the output. \\
print $0 - prints the entire line to the output.
===== [awk] Skip first two columns =====
history | awk '{$1=""; $2=""; print $0}'
===== [awk] Add a header to the output and display the entire file content =====
awk 'BEGIN {print "The File Contents:"} {print $0}' students.txt
{{:linux:ubuntu:pasted:20240207-005907.png}}
===== [awk] Add a header and a footer to the output and display the entire file content =====
awk 'BEGIN {print "The File Contents:"} {print $0} END {print "This is the end my friends."}' students.txt
{{:linux:ubuntu:pasted:20240207-010258.png}}
===== [awk] Display first word (string followed by a space) from each line of a file =====
awk is similar with cut but much more advanced. \\
Default column separator is any whitespace (space, tab) and newline.
awk '{print $1}' students.txt
$1 - first column
{{:linux:ubuntu:pasted:20240207-002957.png}}
===== [awk] Display last word (string followed by a space) from each line of a file =====
awk '{print $NF}' students.txt
NF - number of fields (last column number)
{{:linux:ubuntu:pasted:20240207-003342.png}}
===== [awk] Display entire line and change some column values =====
echo "Hello Tom" | awk '{$2="Adam"; print $0}'
$0 - entire line
{{:linux:ubuntu:pasted:20240207-005442.png}}
===== [awk] List students with notes greater than 5 =====
awk '$5 > 5 {print $1, $2, $5}' students.txt
{{:linux:ubuntu:pasted:20240207-003553.png}}
===== [awk] Compute the average of final grades =====
awk '{sum += $5} END {print "Average of final grades:", sum/NR}' students.txt
sum - custom variable to count the sum (it is not predefined). \\
$5 - fifth column. \\
END - block which will be executed after all lines from the file were processed. \\
NR - internal awk variable which stores the total number of lines.
{{:linux:ubuntu:pasted:20240207-003913.png}}
===== [awk] Customize the columns output =====
Print - between the first and second column:
awk '{print $1, "-", $2}' students.txt
===== [awk] Multiply column value based on a condition =====
awk '{ if ($5>=5) { nr=$5*10} else { nr=$5 } print nr}' students.txt
{{:linux:ubuntu:pasted:20240207-010904.png}}
===== [awk] Print only lines 5 and 7 from /etc/passwd =====
cat /etc/passwd | awk 'NR==5 || NR == 7 {print $0}'
{{:linux:ubuntu:pasted:20240207-205849.png}}
===== [seq] Generate a sequence of numbers from 1 to 5 =====
seq 1 5
{{:linux:ubuntu:pasted:20240207-101045.png}}
===== [seq] Generate a sequence of numbers from 1 to 10, with an increment of 2 =====
seq 1 2 10
{{:linux:ubuntu:pasted:20240207-101200.png}}
===== [seq] Generate a sequence of numbers from 0.5 to 2.5, with an increment of 0.5 =====
seq 0.5 0.5 2.5
{{:linux:ubuntu:pasted:20240207-101317.png}}
===== [seq] Generates numbers with two decimal places =====
seq -f "%.2f" 1 3
f - format
{{:linux:ubuntu:pasted:20240207-101459.png}}
===== [seq] Generate numbers with equal width (added 0 in front) =====
seq -w 1 10
w - equal width for all generated numbers
{{:linux:ubuntu:pasted:20240207-101808.png}}
===== [sort] Sort numbers ascending order =====
sorts the text lines.
By default it will sort as characters.
echo -e "5\n3\n8\n1" | sort -n
e - interpret escape sequences \\
n - numbers (sort as numbers)
{{:linux:ubuntu:pasted:20240207-103102.png}}
===== [sort] Sort numbers descending order =====
echo -e "5\n3\n8\n1" | sort -nr
r - reverse (descending) order
{{:linux:ubuntu:pasted:20240207-103128.png}}
===== [sort] Sort by a certain column =====
Sort numerically by second column.
echo -e "a 2\nc 1\nb 3" | sort -n -k2
k2 - sort by second column
{{:linux:ubuntu:pasted:20240207-103529.png}}
Sort lexicographically by first column. \\
(k1 can be omitted here, sort will automatically sort lexicographically by the first column.)
echo -e "a 2\nc 1\nb 3" | sort -k1
{{:linux:ubuntu:pasted:20240207-112253.png}}
===== [sort] Sort by multiple columns =====
cat multiple.txt | sort -k1,1 -k2,2gr
k1,1 - sort entire first column \\
k2,2 - sort entire second column \\
g - general numeric sort (sort better different kind of numbers for ex 2, 2.5, 3.55) \\
r - reverse
{{:linux:ubuntu:pasted:20240207-111158.png}}
===== [sort] Unique sort - sort and remove duplicates =====
cat multiple.txt | sort -k1,1 -u
u - unique
{{:linux:ubuntu:pasted:20240207-111939.png}}
===== [sort] Sorting with a specific separator =====
Default separator is tab (\t). We want to change it to :.
echo -e "a:2\nb:1\nc:3" | sort -t: -k2
t - change separator
{{:linux:ubuntu:pasted:20240207-112133.png}}
===== [sort] Check if the column is sorted (without sorting it, just check) =====
echo -e "a 2\nc 1\nb 3" | sort -c
c - check
{{:linux:ubuntu:pasted:20240207-104223.png}}
===== [uniq] Remove duplicated lines =====
sort file.txt | uniq
{{:linux:ubuntu:pasted:20240207-183016.png}}
To match the lines case insensitive:
sort file.txt | uniq -i
===== [uniq] Remove duplicated lines and count occurences =====
sort file.txt | uniq -c
c - count
{{:linux:ubuntu:pasted:20240207-183042.png}}
===== [uniq] Show only lines that appear multiple times =====
sort file.txt | uniq -d
d - duplicated
{{:linux:ubuntu:pasted:20240207-183234.png}}
===== [uniq] Show only not duplicated lines =====
sort file.txt | uniq -u
u - unique
{{:linux:ubuntu:pasted:20240207-183351.png}}
===== [sort] [uniq] [awk] Most used 10 history commands =====
history | awk '{$1=""; print $0}' | sort | uniq -c | sort -nr -k1 | head -10 | awk '{$1=""; print $0}'
===== [for] Start 10 processes in background then kill them =====
for i in {1..10}; do sleep 3000 & done; pkill sleep
===== [for] Create 15 files - their names contains numbers =====
Using a one-liner, create 15 files with names ranging from final-exam-00.txt to final-exam-15.txt.
for i in {0..15}; do touch final-exam-$(printf "%02d" $i).txt; done
===== [bash] All types of arguments in a bash script =====
#!/bin/bash
echo "All args"
echo "Using \$*:"
for arg in "$*"; do
echo "$arg"
done
echo "Using \$@:"
for arg in "$@"; do
echo "$arg"
done
echo "Using \$1, \$2, ..."
echo "Arg 1: $1", "Arg 2: $2"
echo "Script name is $0"
echo "Number of args received is $#"
echo "Printing all args using \$# and a for loop"
nr_args=$#
for (( i=1; i<=$nr_args; i++ ))
do
eval arg=\${$i}
echo "Arg $i is: $arg"
done
{{:linux:ubuntu:pasted:20240207-012912.png}}
===== [bash] $var vs ${var} =====
{{:linux:ubuntu:pasted:20240207-112745.png}}
===== [bash] Get the length (number of characters) of a variable =====
{{:linux:ubuntu:pasted:20240207-112900.png}}
===== [bash] Extract a substring from a variable =====
Extract from position 7 with a length of 5 characters:
{{:linux:ubuntu:pasted:20240207-113033.png}}
===== [bash] Command substitution - $() structure =====
The $(...) syntax in Bash is used for command substitution. It allows the output of a command to replace the command itself in a line of script or command input. This is particularly useful when you need to use the output of one command as an argument for another command, or when assigning the output of a command to a variable.
{{:linux:ubuntu:pasted:20240207-114022.png}}
Execute a command based on the condition of another command:
if [[ $(grep -c "searched_word" fisier.txt) -gt 0 ]]; then
echo "Word was found in the file."
else
echo "Word was not found, sorry."
fi
Iterate through files:
for file in $(ls *.txt); do
echo "Processing txt file: $file"
done
Nesting commands (find kernel version):
{{:linux:ubuntu:pasted:20240207-114534.png}}
===== [bash] [while-read] Read a file and assign a variable for each column =====
#!/bin/bash
IFS=$'\t'
while read name group final_grade test_grade practical_grade; do
echo "$name --> $group --> $final_grade --> $test_grade --> $practical_grade"
done < students.txt
IFS - defines the separator, here tab (\t).
{{:linux:ubuntu:pasted:20240207-014050.png}}
===== [bash] [while-read] Read a file and adjust values based on a condition =====
#!/bin/bash
IFS=$'\t'
while read name group final_grade test_grade practical_grade; do
if test "$final_grade" -gt 5; then
echo "$name --> $group --> Good job, final grade is $final_grade --> $test_grade --> $practical_grade"
fi
echo "$name --> $group --> $final_grade --> $test_grade --> $practical_grade"
done < students.txt
{{:linux:ubuntu:pasted:20240207-014529.png}}
===== [bash] [for] Script for manipulating all files from a folder (print information about them, make backups) =====
#!/bin/bash
for file in "$1"/*
do
if test -f "$file"; then
stat -c "%a %F %n" "$file"
cp "$file" "$file.bkp"
fi
done
%a - file permission in numeric form. \\
%F - file type. \\
%n - complete file name.
===== [bash] [rm] Script that removes itself =====
#!/bin/bash
rm -- "$0"
-- - end of command line options. \\
"$0" - the name of the script.
===== [bash] [for] Script to print the sum of the received arguments, error if no argument =====
#!/bin/bash
nr_args=$#
if [ "$nr_args" -eq 0 ]; then
echo "Error: No arguments provided. Usage: $0 ..."
exit 1
fi
sum=0
for (( i=1; i<=$nr_args; i++ ))
do
eval arg=\${$i}
sum=$((sum + arg))
done
echo "Sum is: $sum"
$((sum + arg)) is to evaluate arithmetic expression.
===== [bash] Advanced dictionary lookup script =====
Script to download the wbritish package (contains dictionaries) and which should validate the words from in.txt file located at [[http://elf.cs.pub.ro/uso/res/final/146/in.txt|in.txt]] with the words from the dictionary. \\
Example:
./lookup.sh in.txt
apple in dictionary
noapple not in dictionary
Lookup script with the in.txt file downloaded locally (using wget http://elf.cs.pub.ro/uso/res/final/146/in.txt)
#!/bin/bash
if [ "$#" -ne 1 ]; then
echo "Usage: $0 "
exit 1
fi
input_file=$1
if ! dpkg -l | grep -qw wbritish; then
echo "Installing wbritish package."
sudo apt-get update && sudo apt-get install -y wbritish
fi
if [ ! -f "$input_file" ]; then
echo "File does not exist: $input_file"
exit 2
fi
while IFS= read -r word; do
if look "$word" /usr/share/dict/british-english > /dev/null; then
echo "$word in dictionary"
else
echo "$word not in dictionary"
fi
done < "$input_file"
Output:
./lookup.sh in.txt
mine in dictionary
yours not in dictionary
apples in dictionary
notSoApples not in dictionary
Lookup script that will download the in.txt file from the remote location:
#!/bin/bash
if [ "$#" -ne 1 ]; then
echo "usage: $0 "
exit 1
fi
input_file_url=$1
input_file="in.txt"
wget -O "$input_file" "$input_file_url" || { echo "Eroare downloading input file."; exit 2; }
if ! dpkg -l | grep -qw wbritish; then
echo "wbritissh not installed, installing"
sudo apt-get update && sudo apt-get install -y wbritish
fi
if [ ! -f "$input_file" ]; then
echo "input file does not exist: $input_file"
exit 2
fi
#read each word from input file and check if it is present in dictionary
while IFS= read -r word; do
if look "$word" /usr/share/dict/british-english > /dev/null; then
echo "$word in dictionary"
else
echo "$word not in dictionary"
fi
done < "$input_file"
===== [bash] Advanced disk usage script =====
Create a script named disk-usage.sh that reads a number from stdin and displays the partition that has a disk usage percentage greater than that number. \\ If the input received is not a number, use a default threshold value chosen by you.
Usage example:
$ ./disk-usage.sh
Threshold = 12
WARNING: Partition "/dev/nvme0n1p6" used 13%
WARNING: Partition "/dev/nvme0n1p1" used 27%
$ ./disk-usage.sh
Threshold = ceva
WARNING: Partition "/dev/nvme0n1p6" used 30%
WARNING: Partition "/dev/nvme0n1p1" used 31%
#!/bin/bash
# User should input the threshold
echo "Enter the threshold (percent):"
read threshold
# Check if the threshold is a number
if ! [[ "$threshold" =~ ^[0-9]+$ ]]; then
echo "Input is not a number, using default threshold: 20."
threshold=20
else
echo "Threshold = $threshold"
fi
df -h | awk -v threshold="$threshold" 'NR>1 {gsub(/%/,"",$5); if($5 > threshold) print "WARNING: Partition \""$1"\" used "$5"%"}'
{{:linux:ubuntu:pasted:20240208-012914.png}}
===== [bash] [sec] [hash] Script to hash two arguments - string =====
Create a script that takes two arguments (strings). The script hash_it.sh concatenates the two arguments and displays their SHA256 hash sum.
#!/bin/bash
nr_args=$#
if [ "$nr_args" -ne 2 ]; then
echo "Error: two args should be present. Usage: $0 "
exit 1
fi
result="$1$2"
echo -n $result | openssl dgst -sha256
{{:linux:ubuntu:pasted:20240208-022830.png}}
===== [bash] [sec] [hash] Script to read all lines from a file and print the SHA256 sum of each =====
Suppose the input file is called parole.txt.
#!/bin/bash
IFS=$'\n'
while read password; do
echo -n $password | openssl dgst -sha256 | cut -d '=' -f2 | awk '{print $1}'
done < parole.txt
{{:linux:ubuntu:pasted:20240208-023728.png}}
===== [bash] [ps] Script to display every second the process with highest CPU usage =====
#!/bin/bash
while true; do
clear
#echo "Update on: $(date)"
echo "Process with highest CPU usage:"
ps -eo %cpu,pid,command --sort=-%cpu | head -n 2
sleep 1
done
{{:linux:ubuntu:pasted:20240208-030524.png}}
===== [git] Configure initial settings =====
git config --global user.name "Joe Black"
git config --global user.email "joe.black@github.com"
git config --global init.defaultBranch main
===== [git] Create an empty local repository in the current directory =====
git init
{{:linux:ubuntu:pasted:20240207-020220.png}}
===== [git] Link a local repository with a github one =====
git remote add origin https://github.com/odefta/array-sorting-algorithms.git
The sync between the local repo and the remote (github) repo is made with pull (get) and push (put) commands.
===== [git] Check if the remote repo was configured correctly =====
git remote -v
{{:linux:ubuntu:pasted:20240207-021555.png}}
===== [git] Get all branches and history from remote repo =====
git fetch origin
git pull origin main
{{:linux:ubuntu:pasted:20240207-021840.png}}
===== [git] Get the current branch =====
git branch --show-current
git branch
git status
{{:linux:ubuntu:pasted:20240207-022231.png}}
===== [git] Verify log history =====
git log
{{:linux:ubuntu:pasted:20240207-022604.png}}
===== [git] Verify log history - show last commits only =====
Show only the last 4 commits:
git log -n 4
Show only the last 4 commits - more compact form:
git log -n 4 --oneline
===== [git] Add / commit a file on the current branch to the local repository =====
Create a file, and run git status:
echo "Hello World" > hello.txt
git status
{{:linux:ubuntu:pasted:20240207-023225.png}}
We have an untracked file we can add it (in staging area - which is a pre-commit step):
git add hello.txt
Use . to add all files (from current dir - which includes sub-dirs).
Now we can commit the hello.txt to the local repo:
git commit -m "Add hello.txt with hello world"
m - add a comment
{{:linux:ubuntu:pasted:20240207-023935.png}}
After modifying a file (hello.txt) we can issue again a commit.
===== [git] Push the commit to the remote repo =====
git push origin main
{{:linux:ubuntu:pasted:20240207-092707.png}}
===== [git] Adding a new branch =====
git branch add-gitignore
Check if the new branch was added:
git branch
{{:linux:ubuntu:pasted:20240207-093141.png}}
===== [git] Switch to other branch =====
git checkout add-gitignore
{{:linux:ubuntu:pasted:20240207-093658.png}}
Check
git branch
{{:linux:ubuntu:pasted:20240207-093737.png}}
===== [git] Merge two branches =====
First, let's commit something to our current branch add-gitignore and sync with the remote:
echo "*.o" > .gitignore
git add .gitignore
git commit -m "Add .gitignore file"
git push origin add-gitignore
{{:linux:ubuntu:pasted:20240207-094609.png}}
Now we want to merge our branch main with the add-gitignore branch (to get the .gitignore file).
First, switch to the main branch and get the changes from the remote:
git checkout main
git pull origin main
{{:linux:ubuntu:pasted:20240207-094916.png}}
Now we are ready to merge our current branch main with add-gitignore:
git merge add-gitignore
{{:linux:ubuntu:pasted:20240207-095041.png}}
Push the changes to the remote:
git push origin main
{{:linux:ubuntu:pasted:20240207-095153.png}}
===== [git] Remove a branch both locally and remote =====
We want to delete our secondary branch add-gitignore. \\
First, we need to be on another branch (main).
git checkout main
Remove local branch.
git branch -d add-gitignore
{{:linux:ubuntu:pasted:20240207-095753.png}}
Remove remote branch.
git push origin --delete add-gitignore
{{:linux:ubuntu:pasted:20240207-095855.png}}
===== [git] Switch to a remote branch (without creating a local one with the same name) =====
Remote branch is test.
git fetch origin
git branch -r (verify available remote branches)
git checkout -t origin/test
===== [user] Check the current user =====
whoami
{{:linux:ubuntu:pasted:20240207-115121.png}}
id
{{:linux:ubuntu:pasted:20240207-115219.png}}
===== [user] Check another user =====
sudo id root
{{:linux:ubuntu:pasted:20240207-115351.png}}
===== [user] Display all logged-in users =====
who
{{:linux:ubuntu:pasted:20240207-181234.png}}
To display only the user name:
who | cut -d ' ' -f1
{{:linux:ubuntu:pasted:20240207-181335.png}}
To display additional info:
w
{{:linux:ubuntu:pasted:20240207-181515.png}}
To filter only the logged-in users:
w | awk 'FNR>2 {print $1}'
===== [user] Display the users who have opened a terminal on the host system =====
who | awk '$2 ~ /^tty/ {print $1}'
~ - equals for regular expressions
/ / - delimitates a regular expression
^tty - starts with tty.
===== [user] Get only the UID for a certain user =====
This will print all info.
id syslog
{{:linux:ubuntu:pasted:20240207-115758.png}}
This will print the first column.
id syslog | awk '{print $1}'
{{:linux:ubuntu:pasted:20240207-115857.png}}
This will get the uid value after =.
id syslog | awk '{print $1}' | cut -d= -f2
{{:linux:ubuntu:pasted:20240207-120000.png}}
Finally, this will get the actual uid value.
id syslog | awk '{print $1}' | cut -d= -f2 | cut -d'(' -f1
{{:linux:ubuntu:pasted:20240207-120101.png}}
===== [user] Switch user =====
Switch to another user, you should know only your password. User should be present in sudoers file.
sudo su
Switch to another user, you should know only your password.
sudo su -
- will use the environment variables of that new user.
Switch to another user, you should know the password of the new user.
su
Switch to another user, simulate initial login (-i).
sudo -u -i
Start a new login session.
sudo login
===== [user] Create a new user =====
Simplest and interactive way:
sudo adduser joe
{{:linux:ubuntu:pasted:20240207-122120.png}}
We can add the existing user joe to the joegroup:
sudo adduser joe joegroup
Advanced, with more options (specify home directory, default shell, default group).
sudo groupadd joegroup
sudo useradd -m -d /home/joe -s /bin/bash -g joegroup joe
m - create home directory. \\
d - path to the home directory. \\
s - set implicit shell. \\
g - set primary group.
To add also secondary groups:
sudo groupadd joesecond
sudo groupadd joethird
sudo useradd -m -d /home/joe -s /bin/bash -g joegroup -G joesecond,joethird joe
G - secondary groups separated by comma.
Add / change the joe's password:
sudo passwd joe
===== [user] Modify an existing user =====
Add secondary groups to the user:
sudo groupadd joefourth
sudo groupadd joefive
sudo usermod -a -G joefourth,joefive joe
a - add (important!), without it, the user will be eliminated from the other secondary groups.
{{:linux:ubuntu:pasted:20240207-123705.png}}
Modify the home directory:
sudo usermod -d /home/newjoe joe
d - dir name.
Change the joe's default shell (First, inspect the /etc/shells file for a list of all available shells)
sudo usermod -s /bin/sh joe
Same effect can be obtained with
sudo chsh -s /bin/sh joe
Modify the joe's primary group:
sudo usermod -g joesecond joe
===== [user] Remove an user =====
Remove joe, but keep its files.
sudo userdel joe
Remove joe with all its files (from home dir and other places).
sudo userdel -r joe
r - recursive
===== [user] Block an user to authenticate =====
sudo usermod -s /usr/sbin/nologin joe
===== [user] View user details =====
finger joe
{{:linux:ubuntu:pasted:20240207-125334.png}}
===== [user] Change mode =====
read r - 4
write w - 2
execute x - 1
Owner read and write, group read, other no permission:
chmod 640 file.txt
chmod u=rw,g=r,o= file.txt
Add execute right for others:
chmod 641 file.txt
chmod o+x file.txt
Eliminate read permission for group:
chmod 601 file.txt
chmod g-r file.txt
Eliminate write permission for all:
chmod ugo-w file.txt
===== [user] Change owner =====
Change the owner to joe user and joegroup group.
sudo chown joe:joegroup file.txt
Change only the owner (let the group intact).
sudo chown joe file.txt
Change the owner of all files in a directory (recursive) - let the group intact.
sudo chown -R joe dir
Change the owner of all files in a directory (recursive) - change also the group.
sudo chown -R joe:joegroup dir
===== [user] Just two users should have right to modify files from a folder =====
Suppose those 2 users are uso and so. Folder path is /uso.
sudo groupadd usogroup
sudo usermod -aG usogroup uso
sudo usermod -aG usogroup so
sudo chgrp -R usogroup /uso
sudo chmod -R 770 /uso
===== [user] Generate passwords =====
Generate 5 password of length 10, one per line, each containing at least one uppercase letter.
for i in {1..5}; do
echo $(openssl rand -base64 9 | tr -dc 'a-zA-Z0-9' | cut -c 1-9)$(tr -dc 'A-Z' < /dev/urandom | head -c 1)
done
{{:linux:ubuntu:pasted:20240207-190423.png}}
Same thing using pwgen tool.
sudo apt install pwgen
pwgen -c -n -B 10 5 | tr ' ' '\n'
c (--capitalize) - at least one capital letter will be included. \\
n (--numerals) - at least one number will be included. \\
B (--ambiguous) - eliminate ambiguous characters (like 0 and O). \\
10 - length of the password. \\
5 - generate 5 password.
{{:linux:ubuntu:pasted:20240207-191156.png}}
===== [sec] [base64] Encoding / decoding a string using base64 =====
Encoding.
echo -n "Base64 is not encryption!" | base64
QmFzZTY0IGlzIG5vdCBlbmNyeXB0aW9uIQ==
Decoding.
echo -n "QmFzZTY0IGlzIG5vdCBlbmNyeXB0aW9uIQ==" | base64 -d
d - decoding \\
Base64 is not encryption!
===== [sec] [base64] Encoding / decoding an entire file using base64 =====
Encoding.
base64 file.txt > file64.txt
Decoding.
base64 -d file64.txt > file_dec.txt
===== [sec] [openssl] Encrypting / decrypting a file using OpenSSL =====
Encrypt the file file.txt to file file.enc
echo -n "plaintext content" > file.txt
openssl aes-256-cbc -in file.txt -out file.enc -pass pass:"my secret" -pbkdf2 -iter 10000
aes-256-cbc - encrypt the file using AES-256 in CBC mode \\
in - input file (plain text) \\
out - output file (encrypted) \\
pass - sets the passphrase (will be used for decryption) \\
pbkdf2 - selects PBKDF2 for key derivation, enhancing security against brute-force attacks \\
iter 10000 - sets the number of iterations in the key derivation, increasing the difficulty of cracking the encryption.
{{:linux:ubuntu:pasted:20240207-132922.png}}
Decrypt the file file.enc to decrypted_file.txt.
openssl aes-256-cbc -d -in file.enc -out decrypted_file.txt -pass pass:"my secret" -pbkdf2 -iter 10000
d - decrypt
===== [sec] [hash] Hash a string =====
Hashing transforms data into a fixed-size string of characters, typically a hash code. This process is useful for verifying data integrity, securely storing passwords, and indexing data in databases. A hash function takes input data and produces a unique hash value; any modification to the data changes the hash significantly.
Using md5 (less secure) - two ways of achieving the same result.
echo -n "plaintext content" | md5sum
f88d6c9eab9c6a2ef3cfd3c59832b4d6 -
echo -n "plaintext content" | openssl dgst -md5
MD5(stdin)= f88d6c9eab9c6a2ef3cfd3c59832b4d6
Using SHA256 (much more secure - less collision issues).
echo -n "plaintext content" | openssl dgst -sha256
SHA2-256(stdin)= d800efe86c80b6054cff4850f3044d73e979429809f28029e26cd9c4f6d6cea8
===== [sec] [hash] Hash an entire file =====
file.txt content: plaintext content
Using md5.
md5sum file.txt
88d6c9eab9c6a2ef3cfd3c59832b4d6 file.txt
Using SHA256.
sha256sum file.txt
d800efe86c80b6054cff4850f3044d73e979429809f28029e26cd9c4f6d6cea8 file.txt
===== [sec] [password cracking] Crack linux users passwords using John The Ripper =====
A good dictionary (word list) can be downloaded from here: [[https://github.com/brannondorsey/naive-hashcat/releases/download/data/rockyou.txt|rockyou.txt]]
Install John The Ripper tool.
sudo apt install john
Merge the Linux system files /etc/passwd and /etc/shadow to create a single file (mypasswd.txt) containing user accounts and their hashed passwords.
sudo unshadow /etc/passwd /etc/shadow > mypasswd.txt
{{:linux:ubuntu:pasted:20240207-140549.png}}
Create a word list used to guess the password.
echo -ne "school\nuniversity\nstudent" > wordlist.txt
n - avoid printing the newline \\
e - interpret backslash escapes.
Now we are ready to run John The Ripper.
john --wordlist=wordlist.txt mypasswd.txt
{{:linux:ubuntu:pasted:20240207-140923.png}}
Finally, crack the passwords:
john --format=crypt mypasswd.txt
{{:linux:ubuntu:pasted:20240207-141340.png}}
View the actual passwords (right after username):
john --format=crypt mypasswd.txt --show
{{:linux:ubuntu:pasted:20240207-141509.png}}
===== [sec] [password cracking] Crack a zip archive password using John The Ripper =====
hashfile contains extracted password hashes. \\
This will use the default dictionary (word list) file.
john --format=zip hashfile
To use a custom dictionary file:
john --format=zip hashfile --wordlist=path/to/your/wordlist.txt
===== [sec] [password cracking] Crack a zip archive password using fcrackzip =====
Install the fcrackzip tool.
sudo apt install fcrackzip
Brute-force guessing the password. We know it has 4 characters.
fcrackzip -v -l 1-4 -u secret_brute_force.zip
v - verbose \\
l 1-4 - pwd length from 1 to 4 \\
u - extract the zip archive when the password is found.
Use a dictionary to crack the password.
fcrackzip -u -D -p wordlist.txt secret.zip
u - extract the zip archive when the password is found. \\
D - enables dictionary mode. \\
p - path to the dictionary file.
{{:linux:ubuntu:pasted:20240207-172357.png}}
===== [sec] [hash cracking] Find a hidden message from a file with hashes (one per line) =====
Suppose we have my-hashes.txt:
F5D6C6C424929713C297514199F7CA41
AAD24C09BF816BBC6EA29C35DF52F896
F95B70FDC3088560732A5AC135644506
6EF5FA162DA08580BF5D1F88520F7388
CBB184DD8E05C9709E5DCAEDAA0495CF
1BC29B36F623BA82AAF6724FD3B16718
We'll take each line, put it in crackstation.net \\
The decrypted info from each line will form the final message.
{{:linux:ubuntu:pasted:20240207-195601.png}}