Active Defense Intelligence for the dark web

bitcoin-gollum

good guys and bad guys use Bitcoins too

gAtO – reading about  tools to enable business to have a proactive intelligence of the dark web for an “ Active Defense”. This new model includes not only traditional but unconventional methods using OSINT to gather the intelligence needed.

Companies are getting sick and tired after years of focusing mainly on the defensive postures like malware used in data breaches and financially motivated hacks, some security experts have begun to turn the spotlight on the attacker himself, attempting to profile the bad actors stealing your blueprints or customer credit card numbers, or leaking your usernames and passwords on Pastebin.

The facts are that corporations are spending millions of dollars in defense and defense-in-depth and best practices, and it’s still not helping. With the new active defense we’re making the adversary earn their medals, but they are still getting in. It may take two days now instead of one but we have a fighting chance.

The corporate world have defense and now adding this new model Active Defense (means = offensive) now you need a little dark web intelligence to bind the two together – gAtO OuT

Advertisements

Mapping Tor Relays

gAtO- been working on Mapping Tor-OR and here is some fun stuff – just got o – https://maps.google.com – google maps – the for location type this inhttp://uscyberlabs.com/tormap.kml  – or the .kml file will load Google Earth to- Google Maps – or Google Earth – your flavor..// tor-map-01

You may need to reload it or hit the return a few times but you should get a big map of the world with Tor OR all over the place – This is a static view 2013-04-01 14:57:00 MET – I have a nice automated code that can produce this anytime with whatever the consensus document in Tor gives me. This is a good little tool to have and monitor all known Tor OR-relays.

tor_2010

Biggest Growth Tor OR Usage Washington-DC 2011-2013 

I found some mapping code from 2011 Tor-OR that shows all 900 OR-(2011)  in Tor at that time (currently – 3798 OR-relays june-2013) . Then I got a hold of some code that get’s me all OR-relays 2013. When I compared them both my biggest shock was the number of OR-relays in Washington, DC area shows the biggest growth (2011- 2013)  of OR-relays  on the To network.

 

So tell me why the US government seems to be the biggest user of Tor??? We did hear that The NSA Shroden guy had 2 stickers on his laptop computer – yeah the one with all the secret he got from the NSA – Sticker 1 – EFF 2 – Tor… does NSA use Tor?? it’s contractors do I guess….ummmmm

tor_2013

Last year we where running about 3,000 Tor-OR this year so far we have another 500 more OR bringing us up to 3,500 OR we have also increased the Authority-Directory servers to 10 from 8 that’s a good thing. Anyway here are some statistics from the last few days – gAtO oUt

 

Tor Network – Total Number of Routers: 3426 100%
Routers in Current Query Result Set: 3416 99.71%
Total Number of ‘Authority’ Routers: 10 0.29%
Total Number of ‘Bad Directory’ Routers: 0 0%
Total Number of ‘Bad Exit’ Routers: 2 0.06%
Total Number of ‘Exit’ Routers: 875 25.54%
Total Number of ‘Fast’ Routers: 3015 88%
Total Number of ‘Guard’ Routers: 1154 33.68%
Total Number of ‘Hibernating’ Routers: 1 0.03%
Total Number of ‘Named’ Routers: 2164 63.16%
Total Number of ‘Stable’ Routers: 2311 67.45%
Total Number of ‘Running’ Routers: 3426 100%
Total Number of ‘Valid’ Routers: 3426 100%
Total Number of ‘V2Dir’ Routers: 2087 60.92%
Total Number of ‘Directory Mirror’ Routers: 2087 60.92%

2013-04-01 14:57:00 MET

Mapping Tor OR – we will be doing more Tor-mapping project that will make things funs with Google-Maps – gAtO oUt

Tor Logs:- jun 16-1213

 

——————————————————————————————

Tor network – Application Server Details
Cache Last Updated (Local Server Time): 2013-06-16 14:57:00 MET
Last Update Cycle Processing Time (Seconds): 646
Current Cache Expire Time (Seconds): 300
Number of Routers In Cache: 3798
Number of Descriptors In Cache: 9172
Approximate Page Generation Time (Seconds): 0.0137
Aggregate Network Statistic Summary | Total Number of Routers:
Total Bandwidth of displayed Routers [KBytes/s]: 2572613
Total Number of Routers: 3798 100%
Routers in Current Query Result Set: 3796 99.95%
Total Number of ‘Authority’ Routers: 10 0.26%
Total Number of ‘Bad Directory’ Routers: 0 0%
Total Number of ‘Bad Exit’ Routers: 1 0.03%
Total Number of ‘Exit’ Routers: 894 23.54%
Total Number of ‘Fast’ Routers: 3303 86.97%
Total Number of ‘Guard’ Routers: 1228 32.33%
Total Number of ‘Hibernating’ Routers: 0 0%
Total Number of ‘Named’ Routers: 2244 59.08%
Total Number of ‘Stable’ Routers: 2363 62.22%
Total Number of ‘Running’ Routers: 3798 100%
Total Number of ‘Valid’ Routers: 3798 100%
Total Number of ‘V2Dir’ Routers: 2342 61.66%
Total Number of ‘Directory Mirror’ Routers: 2342 61.66%

——————————————————————————————

Tor Network – Application Server Details
Cache Last Updated (Local Server Time): 2013-06-07 22:02:39 MET
Last Update Cycle Processing Time (Seconds): 477
Current Cache Expire Time (Seconds): 300
Number of Routers In Cache: 3546
Number of Descriptors In Cache: 6712
Approximate Page Generation Time (Seconds): 0.0099

 

Aggregate Network Statistic Summary |
Total Bandwidth of displayed Routers [KBytes/s]: 2434525
Total Number of Routers: 3546 100%
Routers in Current Query Result Set: 3544 99.94%
Total Number of ‘Authority’ Routers: 10 0.28%
Total Number of ‘Bad Directory’ Routers: 0 0%
Total Number of ‘Bad Exit’ Routers: 2 0.06%
Total Number of ‘Exit’ Routers: 848 23.91%
Total Number of ‘Fast’ Routers: 3076 86.75%
Total Number of ‘Guard’ Routers: 1217 34.32%
Total Number of ‘Hibernating’ Routers: 1 0.03%
Total Number of ‘Named’ Routers: 2228 62.83%
Total Number of ‘Stable’ Routers: 2336 65.88%
Total Number of ‘Running’ Routers: 3546 100%
Total Number of ‘Valid’ Routers: 3546 100%
Total Number of ‘V2Dir’ Routers: 2131 60.1%
Total Number of ‘Directory Mirror’ Routers: 2131 60.1%

——————————————————————————————

Tor Network – Application Server Details
Cache Last Updated (Local Server Time): 2013-06-06 13:29:39 MET
Last Update Cycle Processing Time (Seconds): 539
Current Cache Expire Time (Seconds): 300
Number of Routers In Cache: 3502
Number of Descriptors In Cache: 6383
Approximate Page Generation Time (Seconds): 0.0084

 

Aggregate Network Statistic Summary |
Total Bandwidth of displayed Routers [KBytes/s]: 2474946
Total Number of Routers: 3502 100%
Routers in Current Query Result Set: 3501 99.97%
Total Number of ‘Authority’ Routers: 10 0.29%
Total Number of ‘Bad Directory’ Routers: 0 0%
Total Number of ‘Bad Exit’ Routers: 0 0%
Total Number of ‘Exit’ Routers: 849 24.24%
Total Number of ‘Fast’ Routers: 3049 87.06%
Total Number of ‘Guard’ Routers: 1201 34.29%
Total Number of ‘Hibernating’ Routers: 0 0%
Total Number of ‘Named’ Routers: 2225 63.54%
Total Number of ‘Stable’ Routers: 2355 67.25%
Total Number of ‘Running’ Routers: 3502 100%
Total Number of ‘Valid’ Routers: 3502 100%
Total Number of ‘V2Dir’ Routers: 2112 60.31%
Total Number of ‘Directory Mirror’ Routers: 2112 60.31%

——————————————————————————————

Tor network –Application Server Details
Cache Last Updated (Local Server Time): 2013-06-05 16:06:50 MET
Last Update Cycle Processing Time (Seconds): 582
Current Cache Expire Time (Seconds): 300
Number of Routers In Cache: 3534
Number of Descriptors In Cache: 6168
Approximate Page Generation Time (Seconds): 0.0098

 

Aggregate Network Statistic Summary | 
Total Bandwidth of displayed Routers [KBytes/s]: 2572752
Total Number of Routers: 3534 100%
Routers in Current Query Result Set: 3532 99.94%
Total Number of ‘Authority’ Routers: 10 0.28%
Total Number of ‘Bad Directory’ Routers: 0 0%
Total Number of ‘Bad Exit’ Routers: 0 0%
Total Number of ‘Exit’ Routers: 851 24.08%
Total Number of ‘Fast’ Routers: 3088 87.38%
Total Number of ‘Guard’ Routers: 1210 34.24%
Total Number of ‘Hibernating’ Routers: 0 0%
Total Number of ‘Named’ Routers: 2230 63.1%
Total Number of ‘Stable’ Routers: 2363 66.86%
Total Number of ‘Running’ Routers: 3534 100%
Total Number of ‘Valid’ Routers: 3534 100%
Total Number of ‘V2Dir’ Routers: 2123 60.07%
Total Number of ‘Directory Mirror’ Routers: 2123 60.07%

——————————————————————————————

 

Tor network –Application Server Details
Cache Last Updated (Local Server Time): 2013-06-04 02:11:43 MET
Last Update Cycle Processing Time (Seconds): 553
Current Cache Expire Time (Seconds): 300
Number of Routers In Cache: 3599
Number of Descriptors In Cache: 5817
Approximate Page Generation Time (Seconds): 0.01

——————————————————————————————

 

Tor network –Application Server Details
Cache Last Updated (Local Server Time): 2013-05-29 23:19:07 MET
Last Update Cycle Processing Time (Seconds): 645
Current Cache Expire Time (Seconds): 300
Number of Routers In Cache: 3582
Number of Descriptors In Cache: 16099
Approximate Page Generation Time (Seconds): 0.1987

——————————————————————————————

——————————————————————————————

Weaponize the Tor Network:

weaponizing-the-web1-720x2808

Weaponize the Web

prism-01

if you got nothing to hide – you got nothing to worry about

 gAtO wAs – asked the Tor-Network is slow as heck, does not support sending outgoing email and does not support UDP packets of the TCP/IP protocol, so can it be weaponized? Maybe monitoring the Tor-Network like Prism and Nucleon or the Japan based Daedalus Monitoring program at the very least?

Data collection in Tor:

I guess this all depends on your definition of what a weaponize cyber weapon is-///-IP theft- here we have a vast collection of both /IP-(intellectual Property) and /copyright – /hacking /sql-i in Tor// -.- /hacktivism -how about /personal privacy online-collection of all internet transaction and data sharing with Google, Facebook, Microsoft and others— /government censorship of it’s people /Worldwide Internet monitoring-///  Like a room 641a for Tor only traffic.

prism-03

Daedalus Monitoring program

Mix a little more counter-offensive cyber class weapons like Stuxnet, Flame and DuKu – add a bit of misinformation and propaganda to the mix and we have a better question.

Next we have a more military type cyber weaponized solution. Control Drones planes in Tor -another one is dDos, attacks on the electric grid or sabotage satellites. Cyber attacks like power outage, hacking attacks on cell phones and wall street computers and add traffic lights and traffic in the northeast going wacko. Like they say trains, planes and automobiles are all connected to cyberspace from China to Canada… prism-02

Tor can also be used in all the above scenario- Yes big brother/sister it can. So the answer is Yes, but Tor is not the pony network that can do this work. There are other kinds of anonymized networks that can be used, and with your own relays all over the world you can create your own Tor-private network that only you use so it will be faster and side nobody can see it – well Tor is not the only network one to watch for cyber weaponized products – gAtO oUt

 

 

 

Tor Websites over 1/3 TANGO DOWN

gAtO bEeN- doing some work on his Tor- search engine and finding Tor-websites IP but other are doing the same thing and publishes the news-

I guess the news is getting out and people are bringing their Tor-hidden service-websites are going DOWN. Not by my work – I wish – but by a simple little report -:tor-revealing_guard_nodes

IEEE 2013 just put out a report: Trawling for Tor Hidden Services: Detection, Measurement, Deanonymization

http://www.ieee-security.org/TC/SP2013/papers/4977a080.pdf

yes kiddies wee can find your Tor-Website and find the IP and get the geo-location and track you down. The worst part now others know and Tor-websites are being taken down by their own administrators  so they can do countermeasures and not be caught.

2013-05-29 we had 16,000 Tor websites

2013-06-04 we have 3,517 Tor Websites

Application Server Details
Cache Last Updated (Local Server Time): 2013-05-29 23:19:07 MET
Last Update Cycle Processing Time (Seconds): 645
Current Cache Expire Time (Seconds): 300
Number of Routers In Cache: 3582
Number of Descriptors In Cache: 16099
Approximate Page Generation Time (Seconds): 0.1987
Application Server Details
Cache Last Updated (Local Server Time): 2013-06-04 02:11:43 MET
Last Update Cycle Processing Time (Seconds): 553
Current Cache Expire Time (Seconds): 300
Number of Routers In Cache: 3599
Number of Descriptors In Cache: 5817
Approximate Page Generation Time (Seconds): 0.01

So what happened to all the Tor-hidden serve-websites? All I care about is that my work now backed up by this reports shows we are on the right track and we can do what we say we can do and that is to bring down pedophiles websites down in the Tor-network.

The Tor-network is great but these monsters are making Tor a bad place to work and do legit business. Let’s hope other get the message that we are hunting you down even in Tor cowards- gAtO oUt

Finding Tor Websites –geo-location

Finding Tor Websites –geo-location

gAtO tHiNkInG- to find Tor-hidden service-website geo-location we must understand Tor and how it works better. Figure 1 shows us how a basic Tor connection is made. Let’s take a closer look, to understand the weak points in Tor and to find the location of the Tor-hidden service-website:Tor-connection

1,2 and 3 are how a Tor-hidden service-websites tells the world that it is available to the world. 4-5 and 6  create the map’s to the location of the meeting between the client and the HS. 7,8 and 9 are the key’s to finding the website…

The HS –hidden service needs to advertise that it’s available thru the IP –introduction points to the DS- Tor-DNS –so other Tor-clients can find them. The workload of data exchange goes on between the RP -Rendezvous Point and the client and the Tor-website.

 

All Tor connection have 3 relays they must use to connect to the Tor-network.

Client–|> 1.Entry-node 2.Relay-node 3.Exit-nodeHS-website

1. Tor weakness :-A hidden service uses 3 IP and/or 3 RP as part of the ”descriptor information“ so the TOR-DNS can find the site.tor-passive_attack111

a}. To find the geo-location we need to find the 3RP for a HS-website and direct our crawlers to crawl from 8 different geo-location– the delay signals from all location should be the [same/different] from the RP to the HS. This data with data from the OR should give us enough information to tag a location to these signals.

B}. –this is part of the information that is kept in the ”descriptor information“ that the Tor-DNS (directory service) uses to find and connect to the hidden service-website.

We will now have 8-Tor servers  from different worldwide locations finding these 3 RP for the target hidden service-website. Once we have the geo-location of the RP –using  network delay signals that we collect with our cralws. This data can give us triangulation information using data correlation to find the geo-location of the target- Tor hidden tor-relay_01service-website. At least in threory it works, we have started testing some of these new ideas and will keep you posted. So far we can find the country of the target hidden service-website but we need to come closer and get a pinpoint location without an IP address with our medthod of triangulation and data correlation – gAtO oUt

Tor – Unix Commands

gAtO hAs- been in linux onion-land for too long but I found these unix commands to help me out with my work, so I wanted to share them. – well at least it helps the gAtO.

Command Description
apropos whatis Show commands pertinent to string. See also threadsafe
man -t ascii | ps2pdf – > ascii.pdf make a pdf of a manual page
  which command Show full path name of command
  time command See how long a command takes
time cat Start stopwatch. Ctrl-d to stop. See also sw
dir navigation
cd – Go to previous directory
cd Go to $HOME directory
  (cd dir && command) Go to dir, execute command and return to current dir
pushd . Put current dir on stack so you can popd back to it
file searching
alias l=’ls -l –color=auto’ quick dir listing
ls -lrt List files by date. See also newest and find_mm_yyyy
ls /usr/bin | pr -T9 -W$COLUMNS Print in 9 columns to width of terminal
  find -name ‘*.[ch]’ | xargs grep -E ‘expr’ Search ‘expr’ in this dir and below. See also findrepo
  find -type f -print0 | xargs -r0 grep -F ‘example’ Search all regular files for ‘example’ in this dir and below
  find -maxdepth 1 -type f | xargs grep -F ‘example’ Search all regular files for ‘example’ in this dir
  find -maxdepth 1 -type d | while read dir; do echo $dir; echo cmd2; done Process each item with multiple commands (in while loop)
find -type f ! -perm -444 Find files not readable by all (useful for web site)
find -type d ! -perm -111 Find dirs not accessible by all (useful for web site)
locate -r ‘file[^/]*.txt’ Search cached index for names. This re is like glob *file*.txt
look reference Quickly search (sorted) dictionary for prefix
grep –color reference /usr/share/dict/words Highlight occurances of regular expression in dictionary
archives and compression
  gpg -c file Encrypt file
  gpg file.gpg Decrypt file
  tar -c dir/ | bzip2 > dir.tar.bz2 Make compressed archive of dir/
  bzip2 -dc dir.tar.bz2 | tar -x Extract archive (use gzip instead of bzip2 for tar.gz files)
  tar -c dir/ | gzip | gpg -c | ssh user@remote ‘dd of=dir.tar.gz.gpg’ Make encrypted archive of dir/ on remote machine
  find dir/ -name ‘*.txt’ | tar -c –files-from=- | bzip2 > dir_txt.tar.bz2 Make archive of subset of dir/ and below
  find dir/ -name ‘*.txt’ | xargs cp -a –target-directory=dir_txt/ –parents Make copy of subset of dir/ and below
  ( tar -c /dir/to/copy ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) copy/ dir to /where/to/ dir
  ( cd /dir/to/copy && tar -c . ) | ( cd /where/to/ && tar -x -p ) Copy (with permissions) contents of copy/ dir to /where/to/
  ( tar -c /dir/to/copy ) | ssh -C user@remote ‘cd /where/to/ && tar -x -p’ Copy (with permissions) copy/ dir to remote:/where/to/ dir
  dd bs=1M if=/dev/sda | gzip | ssh user@remote ‘dd of=sda.gz’ Backup harddisk to remote machine
rsync (Network efficient file copier: Use the –dry-run option for testing)
  rsync -P rsync://rsync.server.com/path/to/file file Only get diffs. Do multiple times for troublesome downloads
  rsync –bwlimit=1000 fromfile tofile Locally copy with rate limit. It’s like nice for I/O
  rsync -az -e ssh –delete ~/public_html/ remote.com:’~/public_html’ Mirror web site (using compression and encryption)
  rsync -auz -e ssh remote:/dir/ . && rsync -auz -e ssh . remote:/dir/ Synchronize current directory with remote one
ssh (Secure SHell)
  ssh $USER@$HOST command Run command on $HOST as $USER (default command=shell)
ssh -f -Y $USER@$HOSTNAME xeyes Run GUI command on $HOSTNAME as $USER
  scp -p -r $USER@$HOST: file dir/ Copy with permissions to $USER’s home directory on $HOST
  scp -c arcfour $USER@$LANHOST: bigfile Use faster crypto for local LAN. This might saturate GigE
  ssh -g -L 8080:localhost:80 root@$HOST Forward connections to $HOSTNAME:8080 out to $HOST:80
  ssh -R 1434:imap:143 root@$HOST Forward connections from $HOST:1434 in to imap:143
  ssh-copy-id $USER@$HOST Install public key for $USER@$HOST for password-less log in
wget (multi purpose download tool)
(cd dir/ && wget -nd -pHEKk http://www.pixelbeat.org/cmdline.html) Store local browsable version of a page to the current dir
  wget -c http://www.example.com/large.file Continue downloading a partially downloaded file
  wget -r -nd -np -l1 -A ‘*.jpg’ http://www.example.com/dir/ Download a set of files to the current directory
  wget ftp://remote/file[1-9].iso/ FTP supports globbing directly
wget -q -O- http://www.pixelbeat.org/timeline.html | grep ‘a href’ | head Process output directly
  echo ‘wget url’ | at 01:00 Download url at 1AM to current dir
  wget –limit-rate=20k url Do a low priority download (limit to 20KB/s in this case)
  wget -nv –spider –force-html -i bookmarks.html Check links in a file
  wget –mirror http://www.example.com/ Efficiently update a local copy of a site (handy from cron)
networking (Note ifconfig, route, mii-tool, nslookup commands are obsolete)
  ethtool eth0 Show status of ethernet interface eth0
  ethtool –change eth0 autoneg off speed 100 duplex full Manually set ethernet interface speed
  iwconfig eth1 Show status of wireless interface eth1
  iwconfig eth1 rate 1Mb/s fixed Manually set wireless interface speed
iwlist scan List wireless networks in range
ip link show List network interfaces
  ip link set dev eth0 name wan Rename interface eth0 to wan
  ip link set dev eth0 up Bring interface eth0 up (or down)
ip addr show List addresses for interfaces
  ip addr add 1.2.3.4/24 brd + dev eth0 Add (or del) ip and mask (255.255.255.0)
ip route show List routing table
  ip route add default via 1.2.3.254 Set default gateway to 1.2.3.254
host pixelbeat.org Lookup DNS ip address for name or vice versa
hostname -i Lookup local ip address (equivalent to host `hostname`)
whois pixelbeat.org Lookup whois info for hostname or ip address
netstat -tupl List internet services on a system
netstat -tup List active connections to/from system
windows networking (Note samba is the package that provides all this windows specific networking support)
smbtree Find windows machines. See also findsmb
  nmblookup -A 1.2.3.4 Find the windows (netbios) name associated with ip address
  smbclient -L windows_box List shares on windows machine or samba server
  mount -t smbfs -o fmask=666,guest //windows_box/share /mnt/share Mount a windows share
  echo ‘message’ | smbclient -M windows_box Send popup to windows machine (off by default in XP sp2)
text manipulation (Note sed uses stdin and stdout. Newer versions support inplace editing with the -i option)
  sed ‘s/string1/string2/g’ Replace string1 with string2
  sed ‘s/(.*)1/12/g’ Modify anystring1 to anystring2
  sed ‘/^ *#/d; /^ *$/d’ Remove comments and blank lines
  sed ‘:a; /\$/N; s/\n//; ta’ Concatenate lines with trailing
  sed ‘s/[ t]*$//’ Remove trailing spaces from lines
  sed ‘s/([`”$])/\1/g’ Escape shell metacharacters active within double quotes
seq 10 | sed “s/^/      /; s/ *(.{7,})/1/” Right align numbers
seq 10 | sed p | paste – – Duplicate a column
  sed -n ‘1000{p;q}’ Print 1000th line
  sed -n ‘10,20p;20q’ Print lines 10 to 20
  sed -n ‘s/.*<title>(.*)</title>.*/1/ip;T;q’ Extract title from HTML web page
  sed -i 42d ~/.ssh/known_hosts Delete a particular line
  sort -t. -k1,1n -k2,2n -k3,3n -k4,4n Sort IPV4 ip addresses
echo ‘Test’ | tr ‘[:lower:]’ ‘[:upper:]’ Case conversion
tr -dc ‘[:print:]’ < /dev/urandom Filter non printable characters
tr -s ‘[:blank:]’ ‘t’ </proc/diskstats | cut -f4 cut fields separated by blanks
history | wc -l Count lines
set operations (Note you can export LANG=C for speed. Also these assume no duplicate lines within a file)
  sort file1 file2 | uniq Union of unsorted files
  sort file1 file2 | uniq -d Intersection of unsorted files
  sort file1 file1 file2 | uniq -u Difference of unsorted files
  sort file1 file2 | uniq -u Symmetric Difference of unsorted files
  join -t” -a1 -a2 file1 file2 Union of sorted files
  join -t” file1 file2 Intersection of sorted files
  join -t” -v2 file1 file2 Difference of sorted files
  join -t” -v1 -v2 file1 file2 Symmetric Difference of sorted files
math
echo ‘(1 + sqrt(5))/2’ | bc -l Quick math (Calculate ?). See also bc
seq -f ‘4/%g’ 1 2 99999 | paste -sd-+ | bc -l Calculate ? the unix way
echo ‘pad=20; min=64; (100*10^6)/((pad+min)*8)’ | bc More complex (int) e.g. This shows max FastE packet rate
echo ‘pad=20; min=64; print (100E6)/((pad+min)*8)’ | python Python handles scientific notation
echo ‘pad=20; plot [64:1518] (100*10**6)/((pad+x)*8)’ | gnuplot -persist Plot FastE packet rate vs packet size
echo ‘obase=16; ibase=10; 64206’ | bc Base conversion (decimal to hexadecimal)
echo $((0x2dec)) Base conversion (hex to dec) ((shell arithmetic expansion))
units -t ‘100m/9.58s‘ ‘miles/hour’ Unit conversion (metric to imperial)
units -t ‘500GB’ ‘GiB’ Unit conversion (SI to IEC prefixes)
units -t ‘1 googol’ Definition lookup
seq 100 | (tr ‘n’ +; echo 0) | bc Add a column of numbers. See also add and funcpy
calendar
cal -3 Display a calendar
cal 9 1752 Display a calendar for a particular month year
date -d fri What date is it this friday. See also day
[ $(date -d ’12:00 +1 day’ +%d) = ’01’ ] || exit exit a script unless it’s the last day of the month
date –date=’25 Dec’ +%A What day does xmas fall on, this year
date –date=’@2147483647′ Convert seconds since the epoch (1970-01-01 UTC) to date
TZ=’America/Los_Angeles’ date What time is it on west coast of US (use tzselect to find TZ)
date –date=’TZ=”America/Los_Angeles” 09:00 next Fri’ What’s the local time for 9AM next Friday on west coast US
locales
printf “%’dn” 1234 Print number with thousands grouping appropriate to locale
BLOCK_SIZE=’1 ls -l Use locale thousands grouping in ls. See also l
echo “I live in `locale territory`” Extract info from locale database
LANG=en_IE.utf8 locale int_prefix Lookup locale info for specific country. See also ccodes
locale -kc $(locale | sed -n ‘s/(LC_.{4,})=.*/1/p’) | less List fields available in locale database
recode (Obsoletes iconv, dos2unix, unix2dos)
recode -l | less Show available conversions (aliases on each line)
  recode windows-1252.. file_to_change.txt Windows “ansi” to local charset (auto does CRLF conversion)
  recode utf-8/CRLF.. file_to_change.txt Windows utf8 to local charset
  recode iso-8859-15..utf8 file_to_change.txt Latin9 (western europe) to utf8
  recode ../b64 < file.txt > file.b64 Base64 encode
  recode /qp.. < file.qp > file.txt Quoted printable decode
  recode ..HTML < file.txt > file.html Text to HTML
recode -lf windows-1252 | grep euro Lookup table of characters
echo -n 0x80 | recode latin-9/x1..dump Show what a code represents in latin-9 charmap
echo -n 0x20AC | recode ucs-2/x2..latin-9/x Show latin-9 encoding
echo -n 0x20AC | recode ucs-2/x2..utf-8/x Show utf-8 encoding
CDs
  gzip < /dev/cdrom > cdrom.iso.gz Save copy of data cdrom
  mkisofs -V LABEL -r dir | gzip > cdrom.iso.gz Create cdrom image from contents of dir
  mount -o loop cdrom.iso /mnt/dir Mount the cdrom image at /mnt/dir (read only)
  cdrecord -v dev=/dev/cdrom blank=fast Clear a CDRW
  gzip -dc cdrom.iso.gz | cdrecord -v dev=/dev/cdrom – Burn cdrom image (use dev=ATAPI -scanbus to confirm dev)
  cdparanoia -B Rip audio tracks from CD to wav files in current dir
  cdrecord -v dev=/dev/cdrom -audio -pad *.wav Make audio CD from all wavs in current dir (see also cdrdao)
  oggenc –tracknum=$track track.cdda.wav -o track.ogg Make ogg file from wav file
disk space (See also FSlint)
ls -lSr Show files by size, biggest last
du -s * | sort -k1,1rn | head Show top disk users in current dir. See also dutop
du -hs /home/* | sort -k1,1h Sort paths by easy to interpret disk usage
df -h Show free space on mounted filesystems
df -i Show free inodes on mounted filesystems
fdisk -l Show disks partitions sizes and types (run as root)
rpm -q -a –qf ‘%10{SIZE}t%{NAME}n’ | sort -k1,1n List all packages by installed size (Bytes) on rpm distros
dpkg-query -W -f=’${Installed-Size;10}t${Package}n’ | sort -k1,1n List all packages by installed size (KBytes) on deb distros
dd bs=1 seek=2TB if=/dev/null of=ext3.test Create a large test file (taking no space). See also truncate
> file truncate data of file or create an empty file
monitoring/debugging
tail -f /var/log/messages Monitor messages in a log file
strace -c ls >/dev/null Summarise/profile system calls made by command
strace -f -e open ls >/dev/null List system calls made by command
strace -f -e trace=write -e write=1,2 ls >/dev/null Monitor what’s written to stdout and stderr
ltrace -f -e getenv ls >/dev/null List library calls made by command
lsof -p $$ List paths that process id has open
lsof ~ List processes that have specified path open
tcpdump not port 22 Show network traffic except ssh. See also tcpdump_not_me
ps -e -o pid,args –forest List processes in a hierarchy
ps -e -o pcpu,cpu,nice,state,cputime,args –sort pcpu | sed ‘/^ 0.0 /d’ List processes by % cpu usage
ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS List processes by mem (KB) usage. See also ps_mem.py
ps -C firefox-bin -L -o pid,tid,pcpu,state List all threads for a particular process
ps -p 1,$$ -o etime= List elapsed wall time for particular process IDs
last reboot Show system reboot history
free -m Show amount of (remaining) RAM (-m displays in MB)
watch -n.1 ‘cat /proc/interrupts’ Watch changeable data continuously
udevadm monitor Monitor udev events to help configure rules
system information (see also sysinfo) (‘#’ means root access is required)
uname -a Show kernel version and system architecture
head -n1 /etc/issue Show name and version of distribution
cat /proc/partitions Show all partitions registered on the system
grep MemTotal /proc/meminfo Show RAM total seen by the system
grep “model name” /proc/cpuinfo Show CPU(s) info
lspci -tv Show PCI info
lsusb -tv Show USB info
mount | column -t List mounted filesystems on the system (and align output)
grep -F capacity: /proc/acpi/battery/BAT0/info Show state of cells in laptop battery
# dmidecode -q | less Display SMBIOS/DMI information
# smartctl -A /dev/sda | grep Power_On_Hours How long has this disk (system) been powered on in total
# hdparm -i /dev/sda Show info about disk sda
# hdparm -tT /dev/sda Do a read speed test on disk sda
# badblocks -s /dev/sda Test for unreadable blocks on disk sda
interactive (see also linux keyboard shortcuts)
readline Line editor used by bash, python, bc, gnuplot, …
screen Virtual terminals with detach capability, …
mc Powerful file manager that can browse rpm, tar, ftp, ssh, …
gnuplot Interactive/scriptable graphing
links Web browser
xdg-open . open a file or url with the registered desktop application