Monday, 27 June 2011

Ubuntu 11.04 - Network and Security (6/6)


General Information
This is the result of personal research through community and official documentation about GNU/Linux and Ubuntu. References used are provided also. It will be presented in 6 posts as listed below:
Network
Linux support standard Internet protocols used for Unix-to-Unix communication, but also implements a number of protocols native to other, non-Unix operating systems. “The system currently supports the Internet protocols, the Xerox Network Systems(tm) protocols, and some of the ISO OSI protocols” (http://manpages.ubuntu.com/manpages/natty/en/man4/net.4freebsd.html) TCP/IP is the most important set of protocols in Linux networking system. (Operating System Concepts, p.838-9)
Internally, networking in the Linux kernel is implemented by three layers of software:
  • The socket interface
All user applications are performed through the socket interface.
  • Protocol drivers
Data can reach at this layer through network devices or user applications. Using the already existing tag in a packet, it decides which device or which application will be used. It has the ability to rewrite, create, split etc packets.
  • Network-device drivers
This layer is responsible for the routing. The routing of packets takes place by using a set of pointers in memory (socket buffer), with tags as mentioned before, linking the device with the appropriate protocol from a hash table. Two tables are used for the implementation of routing.
  • The forwarding information base (FIB), which uses a specific destination address or by specifying destinations according to wildcards.
  • A cache of recent routing decisions
(Operating System Concepts, p.838-9)
Supported Protocols
The command
cat /etc/protocols
provides us with a full list of the supported Internet Protocols
Port usage is described in
“/etc/services” and can be listed using cat command
Overview of the available network applications
  • 6.1. Web browsers
  • 6.2. The mail system
    • SMTP
    • Smarthost – SMTP AUTH
  • 6.3. Mail transport agent (MTA)
  • 6.4. Mail user agent (MUA)
  • 6.5. The remote mail retrieval and forward utility
    • POP3-APOP-IMAP
    • PO3-IMAP4-SDPS
    • mail retriever (POP3) with regex filtering capability
    • mail retriever (POP3) and DMA with filtering capability
  • 6.6. Mail delivery agent (MDA) with filter
    • DMA with filter
    • DMA with Perl filter
    • DMA with structured filtering language
  • 6.7. POP3/IMAP4 server
  • 6.8. The print server and utilities
    • Line printer daemon
    • Common UNIX Printing System
  • 6.9. The remote access server and utility (SSH)
  • 6.10. Other network application servers
    • TELNET Server (telnet protocol)
    • TELNET SSL support
    • Unix file sharing (NFS)
    • Windows file sharing (SMB)
    • Appla/Mac file and printer sharing (AppleTalk) - (ATP)
    • General file download (FTP)
    • General web server (HTTP)
    • General proxy server
    • OpenSLP Server as LDAP server (SLP)
    • IP address of client itself (DHCP)
    • IP address for other hosts (DNS)
    • VPN (PPTP-Cisco VPN-OpenVPN-IPSec PN) (https://wiki.ubuntu.com/VPN)
  • 6.11. Diagnosis of the system daemons
(http://www.debian.org/doc/manuals/debian-reference/ch06.en.html)
Available Networking tools
In Ubuntu GNU/Linux, a user in order to use administrative tools, he must log in as root. The tools for normal user and administrators are the same, but if a simple user does not have root password, then certain tools cannot be accessed or some options are disabled.
Graphical tools
Available graphical network tools can be found in the system settings.
There is a whole category named “Internet and Networks” in “System settings”. This category includes the following sub-categories :
  • Bluetooth settings
  • Personal file sharing settings
  • Remote Desktop settings
  • Network proxy settings and
  • Network connections settings
    Bluetooth settings
Using this tool, root can add bluetooth devices and share files. The GUI is simple and straightforward.
Personal file sharing
Using this tool, root can determine which folders can be shared over the network or via bluetooth, setting privileges and passwords.
Remote Desktop
Using this tool, we can determine if we wish to allow others to use the desktop. We can set passwords, for those we wish to allow, and we also determine the way we will be notified in the main panel, if someone from the list above, is currently logged in.
Network Proxy
Using this tool we can setup the settings to connect to proxy servers.
Network Connections
This tool allow root to create connections and to edit existing ones. 

Adding to the tools above, as far networking is concerned, there are two more that must be mentioned.

            gconf-editor”
It is a configuration tool for the system in general (applications, graphic settings. nautilus etc),  there are  also preferences for networks. The important thing in this tool is that it is the only place where “WORKGROUP” can be setup for samba servers, which affects performance with MS Windows PCs.
Network tools”
Under system settings in control centre, in the system category there is the “network tools” option. With this tool, users can benefit from the use of commands, normally found only through terminal, as listed below, through an easy GUI:
  • ping
  • netstat
  • traceroute
  • postscan
  • lookup
  • whois
  • finger
Firewall
Ubuntu Linux has a built-in firewall called netfilter. (Ubuntu pocket Guide and Reference, 2009). There is no default installed GUI for firewall and mainly it is used through terminal. The recommended GUI tool is the “firestarter” (Ubuntu pocket Guide and Reference, 2009) package which can be found through the control software or though terminal by typing
sudo apt-get install firestarter
Otherwise the command to configure the firewall through terminal is “ufw”.
Firestarter allows two policies. The “whitelist” policy and the “blacklist” one.
If white list policy is used, then all traffic is blocked unless otherwise determined by a rule. Using “blacklist” policy, every transmission in allowed unless otherwise determined by a rule. (Ubuntu pocket Guide and Reference, 2009)
TerminalUsing the terminal offers an advance user ful access to the options available for networking configuration in Ubuntu. Listed below are some of the basic commands for network configuration and information listing.
/proc/sys/net/ipv4 folder
In this folder in the virtual fs of /proc, an advance user can redetermine the TCP’s values, such as the use of acknowledgements, in order to improve performance. eg. if /proc/sys/net/ipv4/tcp_sack value is 1, then acknowledgement is enabled. (Pro Ubuntu server administration, 2009)
Security
Ubuntu security model is closely related to typical Unix security mechanisms (Operating system Concepts, p.840)
It can be classified in two groups:
  • Authentication, and
  • Access control
    Authentication
Authentication is making sure that only authorised people can access the system, proving their identity. Ubuntu uses the pluggable authentication modules (PAM) system, which a mechamism on a shared library making it possible to be used by any system component. (Operating system Concepts, p.840-841)
Access control is providing a mechanism to check whether a user has the right to access a certain object and preventing access if required. In Ubuntu there are three categories of users.
  • User/owner
  • Group
  • The rest
A user is identified by the UID. We can also determine privileges for a group of user, who will be identify by the GID. Root is a special type of user. It is the equivalent of administrator and has full system access.
There are 3 types for permissions in file/folders.
  • Execute (x)
  • Write (w)
  • Read only (r)
The implementation of security scheme in Linux is directly linked, on the physical layer, with the file system. The file system must be able to store the privileges for users and groups in each file. The speciality of Ubuntu, in the implementation of security policy of linux, is that it allows normal users to “borrow” root privileges. Ubuntu encourages a simple user to log in with the minimum privileges required. However, when required, eg to access system tools, files or perform administrative actions in general, he does not have to log out and relogin. A pop up window shows up and the root password is required to proceed, otherwise no action is taken. In terminal, the $ symbol determine a simple user and # character determines that superuser is logged in.
Encyption
Ubuntu offers the ability to encypt the /home folder so that only the current user and the root will have access to the personal files. Additionally the ability to encrypt a file or folder is also available. PGP protocol is used to do so and the keys are 2024-bit long. (Ubuntu Pocket Guide and Reference, p126-128)
Keyring
Keyring a repository for passwords. It is there to help us with password control. Since a simple user may have several passwords for several objects (browsers, sites etc), keying is there to gather all of them. When an application needs to access the keyring to get the correct password, then a popup window appears to enter the default keyring password. It is a helpful attribute of the OS.
Security tools
Graphical
Security tools can be found in the Control center, under the “Other” and “System” category. 
User and Groups tool, allow us to create users and groups, assigning them rights.
Passwords and Encryptions keys is where keying preferences are stored and encryption options can be found. 


Terminal
Below are listed some basic commands to manipulate users and permissions through the terminal
- sudo adduser --home /home/john john
command creates a user with the name john and also creates the Home folder
- deluser --remove-all-files, command will delete the user john as well as all the file sand the Home folder
- sudo adduser john accounts, will add john to the accounts group
- sudo delgroup accounts, will remove accounts group
- sudo deluser john accounts, will remove john from the accounts group
- ls -la, will list all files and folder in a folder with permissions and ownerships
  • chmod {options} filename, command to change ownership and permissions in files and folders
Options
Definition
u
owner
g
group
o
other
a
all (same as ugo)
x
execute
w
write
r
read
+
add permission
-
remove permission
=
set permission

References
  1. Operating System Concepts, 8th Edition, Abraham Silberschatz, Peter B. Galvin, Greg Gange, 2010, Jonh Wiley and Sons
  2. https://wiki.ubuntu.com/VPN, as found on 22nd June 2011
  3. Ubuntu Pocket Guide and Reference, Keir Thomas, 2009, MacFreda
  4. Pro Ubuntu Server Administration, Sander van Vugt, Apress, 2009

Ubuntu 11.04 - File Systems (5/6)


General Information
This is the result of personal research through community and official documentation about GNU/Linux and Ubuntu. References used are provided also. It will be presented in 6 posts as listed below:


General overview of the file system (FS) implementation in Linux
Linux uses the UNIX's standard file-system model, where everything appear in the form of files for the user. However a file does not have to be stored as object on a disk, it can be a stream of data. (Operating System Concepts, p.828) The kernel handles all these types of files by hiding the implementation details of any type behind a layer of software, the virtual file system (VFS). (Operating System Concepts, p.828)
The VFS is designed with the object-oriented principles. It has two components. One the specifies what file-system objects are allowed to look like and a layer of software to manipulate the objects. (Operating System Concepts, p.829) . The VFS works with objects each one contains a pointer to a function table. (Operating System Concepts, p.469)
  • The main VFS objects are:
  • the inode object and the file object
    Both are mechanisms to access files. Inode objects contain pointers to the disk blocks and the file object contains pointers to the actual file data.
  • the superblock object
    Its main responsibility is to provide access to inodes. It is a connected set of files that form a self-contained file-system. The VFS keeps in pairs the file-system/inode numbers to identify each node and it used the superblock object to get the inode with that pair.
  • the dentry object
    It represents the directory entries found in a path name.
    (Operating System Concepts, p.469 / p.830)
File systems in Linux are either loaded during boot time, or manually.
File systems and UBUNTU 11.04
Ubuntu 11.04 support the Linux Logical Volume Manager, which allows us to increase the size of a partition online, while the system is running.
In addition, Ubuntu 11.04 allows us to use different devices as mount points for several “heavy/crucial” directories. These are:
  • /boot, the directory containing the basic instructions to load the OS
  • / (root) directory
  • /home, the user(s) home directory, equivalent to “My documents” folder in MS Windows
A full list of the supported file systems in UBUNTU 11.04 can be found using the “cat” command in the virtual file system /proc as shown below.
cat /proc/filesystems
Table 1. Supported file systems in UBUNTU 11.04 according to Ubuntu natty manpages.
minix
is the file system used in the Minix operating system, the first to run under Linux. It has a number of shortcomings: a 64MB partition size limit, short filenames, a single timestamp, etc. It remains useful for floppies and RAM
disks.
ext
an elaborate extension of the minix file system. It has been completely superseded by the second version of the extended file system (ext2) and has been removed from the kernel (in 2.1.21).
ext2
the high performance disk file system used by Linux for fixed disks as well as removable media. The second extended file system was designed as an extension of the extended file system (ext). ext2 offers the best performance (in terms of speed and CPU usage) of the file systems supported under Linux.
ext3
a journaling version of the ext2 file system. It is easy to switch back and forth between ext2 and ext3.
ext4
a set of upgrades to ext3 including substantial performance and reliability enhancements, plus large increases in volume, file, and directory size limits.
Reiserfs
a journaling file system, designed by Hans Reiser, that was integrated into Linux in kernel 2.4.1.
XFS
a journaling file system, developed by SGI, that was integrated into Linux in kernel 2.4.20.
JFS
a journaling file system, developed by IBM, that was integrated into Linux in kernel 2.4.24.
msdos
the file system used by DOS, Windows, and some OS/2 computers. msdos filenames can be no longer than 8 characters, followed by an optional period and 3 character extension.
umsdos
an extended DOS file system used by Linux. It adds capability for long filenames, UID/GID, POSIX permissions, and special files (devices, named pipes, etc.) under the DOS file system, without sacrificing compatibility with DOS.
vfat
an extended DOS file system used by Microsoft Windows95 and Windows NT. VFAT adds the capability to use long filenames under the MSDOS file system.
ntfs
replaces Microsoft Window's FAT file systems (VFAT, FAT32). It has reliability, performance, and space-utilization enhancements plus features like ACLs, journaling, encryption, and so on.
proc
a pseudo file system which is used as an interface to kernel data structures rather than reading and interpreting /dev/kmem. In particular, its files do not take disk space.
iso9660
a CD-ROM file system type conforming to the ISO 9660 standard.
High Sierra
Linux supports High Sierra, the precursor to the ISO 9660 standard for CD-ROM file systems. It is automatically recognized within the iso9660 file-system support under Linux.
Rock Ridge
Linux also supports the System Use Sharing Protocol records specified by the Rock Ridge Interchange Protocol. They are used to further describe the files in the iso9660 file system to a Unix host, and provide information such as long filenames, UID/GID, POSIX permissions, and devices. It is automatically recognized within the iso9660 file-system support under Linux.
hpfs
the High Performance Filesystem, used in OS/2. This file system is read-only under Linux due to the lack of available documentation.
sysv
an implementation of the SystemV/Coherent file system for Linux. It implements all of Xenix FS, SystemV/386 FS, and Coherent FS.
nfs
the network file system used to access disks located on remote computers.
smb
a network file system that supports the SMB protocol, used by Windows for Workgroups, Windows NT, and Lan Manager.
ncpfs
a network file system that supports the NCP protocol, used
by Novell NetWare.
However, support for additional file systems can be added by adding additional modules to kernel.
As it can be seen above, UBUNTU 11.04 supports several file systems. The result when using the cat /proc/filesystems – command, will include several virtual file systems connected to no devices, that are used from Ubuntu kernel for administrative actions, such as memory caching. e.g.
ramfs, a very simple filesystem that exports Linux's disk caching mechanisms (the page cache and dentry cache) as a dynamically resizable RAM-based filesystem.
rootfs, a special instance of ramfs (or tmpfs, if that's enabled), which is
always present in 2.6 systems. You can't unmount rootfs for approximately the
same reason you can't kill the init process; rather than having special code
to check for and handle an empty list, it's smaller and simpler for the kernel
to just make sure certain lists can't become empty.”
Adding to those above, the swap file system must be noted as well, even though it is a raw file system, used for memory caching. Swap is a very important parameter in Linux.
The ones available to use in a physical drive during a clean installation, using the default partitioning tool are:
A very brief comparison of the File systems most commonly found in UBUNTU.
File System
Max File Size
Max Partition Size
Journaling
Notes
Fat16
2 GB
2 GB
No
Legacy
Fat32
4 GB
8 TB
No
Legacy
NTFS
2 TB
256 TB
Yes
For Windows Compatibility) NTFS-3g is installed by default in Ubuntu, allowing Read/Write support)
ext2
2 TB
32 TB
No
Legacy
ext3
2 TB
31 TB
Yes
Standard linux filesystem for many years. Best choice for super-standard installation.
ext4
16 TB
1EB
Yes
Modern iteration of ext3. Best choice for new installations where super-standard isn't necessary.
reiserfs
8 TB
16 TB
Yes
No longer well-maintained.
JFS
4 PB
32 PB
Yes (metadata)
Created by IBM - Not well maintained.
XFS
8 EB
8 EB
Yes (metadata)
Created by SGI. Best choice for a mix of stability and advanced journaling.
GB = Gigabyte (1024 MB) :: TB = Terabyte (1024 GB) :: PB = Petabyte (1024 TB) :: EB = Exabyte (1024 PB)

(https://help.ubuntu.com/community/LinuxFilesystemsExplained)
However, as mentioned before, not all are available during installation to be used in a physical drive, since not all support the read/write and file permissions required for Ubuntu to work properly. NTFS cannot be used to install Linux at all, and FAT16/32 are not recommended to install Linux, since FAT does not have any of the permissions of a true Unix FS. They are used for compatibility reasons with Microsoft Windows.
From the file systems described above, the most commonly used for Ubuntu installation, are:
  • ext2/ext3
  • ext4
    and
  • XFS
Ext2 as shown in table 1, is not a journaling FS. It is still commonly used however, to mount in boot time the /boot directory. Since it is not a journaling system, there are less read/write operations and since the /boot directory is only used in boot time to load the kernel, ext2 can result in faster OS boot.
Ext3 is the successor of ext2. However, it is a journaling system. It has been the default file system for a lot of years and since the LTS versions starting from 6.04 are only now starting to expire, many still use this file system for their / (root) and /home directories. Ubuntu supports the conversion from ext3 to ext4, the new default journaling default file system. The possible limit for a sub directory is 32000. It uses an indirect block mapping scheme to keep track of each block used for the blocks corresponding to the data of a file.
(http://kernelnewbies.org/Ext4)
Ext4 is the evolution of ext3 and it is a journaling file system, but it is an attribute that can be disabled. Ext4 breaks sub directory limit and allows an unlimited number of sub directories. Moreover, it uses “extents” to improve performance for large files. “For example, a 100 MB file can be allocated into a single extent of that size, instead of needing to create the indirect mapping for 25600 blocks (4 KB per block). Huge files are split in several extents. Extents improve the performance and also help to reduce the fragmentation, since an extent encourages continuous layouts on the disk. When Ext3 needs to write new data to the disk, there's a block allocator that decides which free blocks will be used to write the data. But the Ext3 block allocator only allocates one block (4KB) at a time. That means that if the system needs to write the 100 MB data mentioned in the previous point, it will need to call the block allocator 25600 times (and it was just 100 MB!). Not only this is inefficient, it doesn't allow the block allocator to optimize the allocation policy because it doesn't knows how many total data is being allocated, it only knows about a single block. Ext4 uses a "multiblock allocator" (mballoc) which allocates many blocks in a single call, instead of a single block per call, avoiding a lot of overhead. This improves the performance, and it's particularly useful with delayed allocation and extents.” (http://kernelnewbies.org/Ext4)
In addition, delayed allocation procedure is used in ext4 improving performance and reducing fragmentation and it is supported for online de fragmentation.
XFS is a journaled 64-bit file system. It is organized in balanced B+Trees and uses Extends to store data. XFS is fully supported by all Ubuntu-Versions. It is not recommended to be used to mount the /boot directory, due to GRUB support. It I sused to mount the root directory and the /home one.
Advantages
  • Part of official Kernel
  • Online optimizing (defragmenting) filesystem using "xfs_fsr" (part of xfsdump-package)
  • Ability to grow an existing filesystem using "xfs_growfs" (part of xfsprogs-package)
  • Own dump- and restore-utilities to keep fs-specific info
  • Fast seeking through directories
  • Efficient file storage, low fragmentation
  • Relatively low memory usage
  • Journal (internal or external)
  • Balance between speed and safety
  • Delayed allocation and improved sparse file handling
Disadvantages
  • Slower than non-journaled fs (like ext2)
  • Due to balance of speed and safety, there is the possibility to lose data on aprupt power loss (outdated journal metadata), but not filesystem consistence
  • Currently there is no ability to shrink an xfs filesystem. However, there are some people working on this issue
  • GRUB-Support is in an early stage
  • The journal is currently not 32-bit / 64-bit portable. Before mounting a xfs prevously used on 32-bit linux in 64-bit linux (or different arch) the journal has to be emptied using xfs_repair.

Tools for the ext2,ext3,ext4 and XFS file systems in Ubuntu.
The available tools for file system information and modifications, are separated ion this paper in two parts. The one where the only the graphical tools are explained, and the second one where the commands in terminal are briefly explained.
GUI tools
A fast overview of the available file systems can be found in the system monitor, in the file systems tab.

Further actions and information for the available storage devices can be found in the Disk Utility in the Control Center.
Disk Utility provide us with an overview of the storage devices currently connected in the system. It also provides us with an easy graphical interface to:
  • get hardware information
  • mount devices
  • unmount devices
  • check the filesystem in the device
  • edit partitions
  • delete partitions
  • format volumes
Graphical tools for fie systems, as for any other administrative actions, are poorer comparing to the equivalents in terminal. They addressed to normal users and available actions are limited, e.g define block size, cannot be processed thought them.
Terminal Commands
They are addressed to advanced users and the can modify any available option for a file system. This is why they must be used as superuser, or by borrowing su privileges. Additional information, about parameters and syntax, are provided in the links below.
  • blkid, used to find the Universally Unique Identifier (UUID) for a device, which is prefered from Ubuntu over the deveice path, eg /dev/sda, since it can change during boot.

  • mkfs, used to create a file system
  • fsck, used to check and repair file system
  • fdisk
  • df, used to report file system usage

  • badblocks, used to check a device for bad blocks
  • umount and mount, used to unmount and mount devices

References

  1. Operating System Concepts, 8th Edition, Abraham Silberschatz, Peter B. Galvin, Greg Gange, 2010, Jonh Wiley and Sons
  2. http://www.raiden.net/node/576, as found on June 15th 2011
  3. Pro Ubuntu Server Administration, Sander van Vugt, 2009, Apress
  4. http://kernelnewbies.org/Ext4, as found on June 14th 2011
  5. https://wiki.ubuntu.com/XFS , as found on June 15th 2011
  6. http://oss.sgi.com/projects/xfs , as found on June 15th 2011
  7. https://help.ubuntu.com/community/Mount, as found on June 15th 2011


Raspberry pi - Introduction

Hello once again. This is the first of series of Raspberry pi related posts. I believe that the Raspberry pi (rpi from now on) can help u...