General Information
This is the result of personal research through community and official documentation about GNU/Linux and Ubuntu. References used are provided also. It will be presented in 6 posts as listed below:
This is the result of personal research through community and official documentation about GNU/Linux and Ubuntu. References used are provided also. It will be presented in 6 posts as listed below:
Overview – Allocation
UBUNTU as a GNU/Linux distribution is designed for different architectures. Due to this characteristic, UBUNTU separates physical memory into three different zones:
- zone_DMA
- zone_NORMAL
- zone_HIGHMEM
Zone_DMA is the zone that includes the first 16MB of physical memory. It is consisted of the DMA-able pages (Direct Access Memory). This happens because certain hardware, Industry Standard Architecture (ISA) devices, can only access the lower 16MB of physical memory. (Operating System Concepts, p.820)
Zone_NORMAL is the zone that includes the normally addressable pages up to the first 896 MB of physical memory. The remaining physical memory (>896 MB) is allocated from zone_HIGHMEM. It is referred as high memory and includes the dynamically mapped pages.
“The primary physical-memory manager in the Linux kernel is the page allocator” (Operating System Concepts, p.820). There is one allocator for each zone, responsible for allocating, freeing all physical pages for each zone, and capable of allocating ranges of physically contiguous pages on request, but even though some allocations may require a page from a particular zone, a normal allocation can come form zone_DMA or zone_NORMAL. However, it is preferable to use a page from the particular zone, mostly to save the DMA space, but if needed and memory is low, Kernel can use whichever zone is available and suitable. (Robert Love, Linux Kernel Development Second Edition). It is important to note that many kernel operations can only take place using zone_NORMAL so it is the most critical zone about performance. (http://www.kernel.org/doc/gorman/html/understand/understand005.html)
UBUNTU 11.04 uses 4096-byte page size (http://manpages.ubuntu.com/manpages/natty/man1/pagesize.1.html) that utilises available physical memory, reducing the percentage of internal fragmentation. However, as of 2.6.38 kernel, the one UBUNTU 11.04 uses (https://wiki.ubuntu.com/NattyNarwhal/ReleaseNotes), the transparent huge page (THP) feature is added, which takes advantage of the page sizes that contemporary processors can handle simultaneously. For example, the Intel® Pentium® Processor E5700 has a 2M Cache. (http://ark.intel.com/Product.aspx?id=42801) This means that a 2M page requires one entry in the processor's translation lookaside buffer. Instead, with page size 4096, it needs 512 entries. “THP simply substitutes huge pages for the regular kind whenever they are available and it appear to make sense. The result is that Linux systems take advantage of this hardware feature and run faster on almost all workloads” (http://www.linux.com/news/featured-blogs/171-jonathan-corbet/421304-2638-making-things-just-work).
There are two ways that memory is allocated in UBUNTU. Statically, during boot time for drivers that deserve contiguous areas in memory, and dynamically by the page allocator.
There are two mechanisms for allocating Kernel memory:
- The buddy service technique, and
- The slab allocation
kmalloc is the normal method of allocating memory in the kernel. (http://manpages.ubuntu.com/manpages/dapper/en/man9/__kmalloc.9.html)
Using a “buddy service” type technique of allocation, the kmalloc() service is used, when the size of the request is not known in advance and it can be a few bytes, to allocate entire pages on demand and then splitting them into smaller pieces. The allocation by kmalloc() service is permanent until they are freed explicitly. The kernel maintains a list with the pages used by the kmalloc() service. (Operating System Concepts, p.820-821)
The slab technique is used “for allocating memory for Kernel data structures and is made up for one or more physically contiguous pages”. (Operating System Concepts, p.821) The slab allocation algorithm uses caches to store kernel objects. A cache consists of one or more slabs. Each kernel structure has its own cache. “A slab may be in one of three possible states:
- Full. All objects in the slab are marked as used.
- Empty. All objects in the slab are marked as free.
- Partial. The slab consists of both used and free objects.”
(Operating System Concepts, p.823) Priority is first given to partial slabs. If one does not exist, then a free object is assigned from a free one.
Except of these two main memory managers, there are two more subsystems that do their own management of physical-memory: the page cache and the virtual memory system.
“The page cache is the kernel's main cache for block devices and memory-mapped files and is the main mechanism though which I/O to these devices is performed.” (Operating System Concepts, p.823). The page cache can also cahce networked data. Block devices provide the main interface for disks in a system. UBUNTU 11.04 use a deadline I/O scheduler which provide deadline for read requests of 0.5 second and 5 seconds deadline for write requests. The deadline scheduler maintains a sorted queue, a read and a write queue. (Operating System Concepts, p.836)
UBUNTU do very high use of virtual memory. mmap() service is responsible for mapping files or devices into memory. “Linux reserves for its own internal use a constant, architecture-dependent region of the virtual memory address space of every process. The page-table entries that map to these kernel pages are marked as protected” (Operating System Concepts, p.826) Two regions are used, one where the core of kernel, along with all pages allocated by the normal page allocator reside, and another which is reserved for no specific purpose. The vmalloc() service is used to allocate an arbitrary number of physical pages of memory that may not be contuguous into a single region of virtually contiguous kernel memory. (Operating System Concepts, p.826)
The mmap() service has flag arguments that transform the mapping to shared with other processes, or private creating a private copy-on0write mapping. (http://manpages.ubuntu.com/manpages/lucid/man2/mmap.2.html) The virtual mapping procedure is directly connected to the way the process was created. If it is created through the exec() servicem then a new mapping address is provided. If it is created through fork() then the parent's pages are directly copied to the child's, resulting that the parent and child share physical pages in memory. UBUNTU 11.04 use the least frequently used (LFU) as page out policy (Operating System Concepts, p.825). It is through the age variable of a page that it is chosen which page to swap.
UBUNTU extends the physical memory of the system using a so called “swap” partition in the HDD. It is a raw type file-system where pages are swapped when faulting. Due to the type of the file system the I/O are faster than normal file systems.
It also important to note that to the available architecture of the OS, the physical limitation of a 32-bit version of UBUNTU would normally be limited to a ~4GB of usable RAM. However, the Physical Address Extension kernel option is available for UBUNTU 9.10 and above. This technology allows 32-bit OS to use up to 64 GB of RAM (https://help.ubuntu.com/community/EnablingPAE). Even though the additional RAM is slower, it is still faster than a HDD. In general in these cases a 64-bit OS is prefered, however, so far drivers and applications are still better supported in a 32-bit OS.
Memory Management OS Utilities
The only graphical tool about memory management that comes with UBUNTU installation is the System Monitor. One can reach it from System Settings → System Monitor. As it can be seen, the only available information one can get is the amount and percentage of memory and swap currently being used. A graphical representation over time is created helping us to visualize the constant chagnes in the values. However, these information are poor for an advanced user and even less for the system root user. More detailed utilities are available through the command-line, most of which are only available for the root group.
At this point it must be noted, as it has been mentioned in assignment 1, there is a /proc folder in the file system. It is a pseudofile system which is used as an interface to kernel data structures. It is commonly mounted at /proc.Most of it is read-only, but some files allow kernel variables to be chagned.” (http://manpages.ubuntu.com/manpages/natty/man5/proc.5.html)
It can seen that the “folder” includes several sub folders with numerical names and some files.
The folders with numerical names, correspond to PIDs. Each folder contains several files that carry information about that process. It must be made clear that when we open one such file, root privileges are required, so a $sudo should be used. In addition, what we see is an instance of the constant memory changes that take place. Since we cannot recognise a task by the PID, $top should be used.
- PID : Process ID
- USER: User the process belongs to
- PR: Priority
- NI: Nice value
- VIRT: Virtual Image
- RES: Resident size
- SHR:Shared memory size
- %CPU: CPU usage
- %MEM: Memory usage
- TIME+:CPU Time
- COMMAND: command name/line
(Sander van Vugt, 2009)
$top command can be customized about the available shown options. In addition, it provides updates about swap, cache and memory usage.
Combining the $top command with the proc virtual file system, we can get all the available information about a process. First we get an overview with top of the current situation, we identify the terget process, get the PID and then we move to the proc file at /proc for further investigation. http://manpages.ubuntu.com/manpages/natty/man5/proc.5.html provides a full overview of the information we can obtain through the use of proc file system and an text editor, such as vi.
Additional commands and files containing memory information are shown below:
free -m | Limited memory information |
/proc/meminfo | Full memory information |
/proc/interrupts | System interrups served per CPU |
uptime | System uptime |
vmstat -s | Virtual memory information |
sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches" | Drops caches |
pagesize | Page size currently used |
uname -r | Kernel version (PAE included) |
sudo swapoff -a | Turning swap off |
sudo swapon -a | Turning swap on |
References
- Operating System Concepts, 8th Edition, Abraham Silberschatz, Peter B. Galvin, Greg Gange, 2010, Jonh Wiley and Sons
- Linux Kernel Development Second Edition, Robert Love, Sams Publishing, 2005 as found at http://book.chinaunix.net/special/ebook/Linux_Kernel_Development/0672327201/ch11lev1sec2.html on June 1st 2011.
- https://wiki.ubuntu.com/NattyNarwhal/ReleaseNotes as found on May 2nd 2011.
- http://ark.intel.com/Product.aspx?id=42801, as found on June 1st 2011.
- http://www.kernel.org/doc/gorman/html/understand/understand005.html found on June 1st 2011
- http://manpages.ubuntu.com/manpages/natty/man1/pagesize.1.html as found on May 31st 2011.
- http://manpages.ubuntu.com/manpages/dapper/en/man9/__kmalloc.9.html found on June 1st 2011.
- http://manpages.ubuntu.com/manpages/lucid/man2/mmap.2.html as found on June 2nd 2011.
- https://help.ubuntu.com/community/EnablingPAE found on June 3rd 2011.
- http://manpages.ubuntu.com/manpages/natty/man5/proc.5.html ofund on June 2nd 2011.
- Pro Ubuntu Server Administration, Sander van Vugt, 2009, Apress
Comments
Post a Comment