This utility program is used for monitoring the working of a real-time file system, registry and process or thread activity in a PC. It helps early fault-detection and immediate identification of out-of-limit variables for real-time process control, giving users the possibility to remedy potential failures before these occur. It can be used as a tool for system troubleshooting and malware hunting.
Operating system processes
When a process needs to store temporary data, it can request a memory block from the central memory pool by way of dynamic memory-allocation. However, the total available memory is limited at any point of time. If one process eats up all the free memory, then other processes will not be able to get their required memory. Implications of a memory-starved process can lead to shutdown or unexpected crash. Clearly, none of these results are desirable to a programmer, so the processes should never reach such a state or the system itself should not starve for memory-allocation.
It is the responsibility of each process to free dynamically-allocated memory when the process is completed. The memory, when freed, goes again to the central pool, where it can be reallocated to another process on demand. When a process dynamically allocates memory and does not free that memory after use, that process heads to memory leak.
Memory leaks add up over time and, if these are not cleaned, the system eventually runs out of memory. For example, we come across an out-of-memory situation on Linux machines when memory requests gets higher than the available free pool. This is typically because of the fact that the unused memory was not released over time or processes recursively request for more memory blocks.
Here comes the need to monitor dynamic memory requests of processes. Especially, on a development system, memory requests monitoring, as well as memory-usage pattern at different time quantum, becomes vital to capture leaks.
How a process monitor works
Processes use malloc, realloc and free function calls for dynamic memory requests. These functions are actually like a wrapper to real __libc__ memory function calls, which does the real allocation.
Since we want to monitor and log details of these requests like requested block size in bytes, allocated region (i.e., pointer address), etc, we have to hook the control flow in between these wrapper functions and the real __libc__ procedure call. To demonstrate this, we have a daemon (or Linux process), say, pmd, where we use malloc and realloc function calls for requesting memory blocks (and releasing these) periodically.
We have written a set of intermediate functions to hook the flow. From inside the hook function, we log details of the request. For logging purpose, we use a kernel module, pmm and a character device for communication. The kernel module becomes necessary when we want to understand and process the memory-usage footprint of a process at any point of time. Based on the memory footprint against time, the user can decide whether the process is a leaking memory or not. I leave this part of kernel programming open for readers to develop according to their requirements.
For implementation, the kernel module simply does the following:
1. Register a character device for the user space process to communicate
2. Use the device for logging memory-allocation requests from the user space daemon
3. Use /proc file system to display logs
Also Read:Interesting Electronics Projects
Software program
The program includes following packages: pmd.c (Linux user space daemon), pmm.c (Linux kernel module), pmm.h (common header file). and makefile (Linux kernel makefile).
User space process. The pmd is a user space process (or daemon) that runs an endless loop. The malloc and realloc are invoked for dynamic memory-allocation requests. We have added some functions for the purpose of hooking in-between. The hook functions, which we introduced, log memory request details by way of character device ioctl (input/output control), registered by our kernel module.
[stextbox id=”grey”]
/* This is our daemon’ main function */
int main( int argc, char *argv[]) {
char *chr;
daemonize();/* Makes this Linux process
as daemon */
openlogs(); /* This enables the hooking
of local memory request functions for
logging */
while (1) { /* Endless loop; daemon’s
main execution loop */
sleep(10);
chr = (char *)malloc(sizeof(char)*100);
/* malloc call */
sleep(10);
chr = (char *)realloc(chr,
sizeof(char)*200); /* realloc call */
sleep(10);
free(chr); /* free call */
}
closelogs();/* This disables the
hooking; the daemon exits */
return 0;
}
[/stextbox]
The hooks work with the following additional code:
[stextbox id=”grey”]
// Preprocessor defines
#define malloc(A)
malloc_(A, __FILE__, __FUNCTION__,
__LINE__)
#define realloc(A, B)
realloc_(A, B, __FILE__, __FUNCTION__,
__LINE__)
#define free(A)
free_(A, __FILE__, __FUNCTION__, __
LINE__)
voidopen logs() {
hooks_active = 1;
….
}
// Intermediate function which decides
to go via our malloc hook or directly
to __libc__ function
void *malloc_(size_t size, const char
*file, const char *fn, constintln) {
// Caller pointer
void *caller = __builtin_return_
address(0);
if (hooks_active) {
returnmy_malloc_hook(size, caller, file,
fn, ln);
}
return __libc_malloc(size);
}
// Malloc hook
void *my_malloc_hook(size_t size, void
*caller, const char *file, const char
*fn, constintln) {
void *result;
// Deactivate hooks for logging
hooks_active = 0;
// Call real libcmalloc
result = __libc_malloc(size);
// Do logging
printf(“Memhook-malloc: sz=%d,
caller=%u, result=%u, file=%s, func=%s,
line=%d\n”,
size, (unsigned int)caller, (unsigned
int)result, file, fn, ln); /* Prints on
the console */
….
// Reactivate hooks
hooks_active = 1;
return result;
}
[/stextbox]
When hooks_active = 1, the control flow will be: Malloc() ->malloc_() ->my_malloc_hook() -> __lib_malloc() (->pmm_ioctl() to kernel module).
When hooks_active variable is zero, it will be: Malloc() ->malloc_() -> __libc_malloc().
Kernel module. The pmm is used as a kernel module. The module init() registers the character device, say, pmmdev, and its fops (device open, ioctl, close) structure. The pmm_ioctl() is the main contact point for the module, which has following ioctls:
PMM_RECORD_LOG ioctl for recording the memory request from hook functions. On receipt of malloc or realloc logs, these will be added to the log list. And, on receipt of free log, it will be compared with already logged malloc logs and, when a match is found, both malloc and free logs will be removed from the log list. (Because, the matching pair of malloc and free will not increase the memory footprint of a process.)
PMM_SET_CONFIG ioctl for starting or stopping logging for a particular user process.
PMM_GET_CONFIG ioctl for getting current configuration (kernel thread-execution cycle, etc).
PMM_MONITORED_PIDS for getting the list of monitored user processes.
Initialise proc entry (say, /proc/pmm) for user interaction. Our proc has two entries.
Cfg. Displays the module’s configuration. This is used to start or stop scanning the kernel task structure by our kernel thread and modifying its frequency.
Pid. Used to display logs (memory-allocation logs sent from the user space daemon) to the user.
Start pmthread, a kernel thread. The thread runs over the kernel task structure to find out whether current monitoring user space daemons are in active or killed state.
Note. The kernel thread can be enhanced to process memory footprint at any point by accessing vm and rss sizes for understanding its overall increase in usage against the time quantum.
For further development
Check for other data members of kernel task_struct structure of user space processes for getting the complete picture of vm and rss sizes, shared library size, CS and DS segment sizes and much more. Using simple math, we can find out the overall memory consumption of processes that can be used for memory-leak alerting.
Hook functions can be separated as shared library so that these become a ready-made solution for developers seeking a programmable memory-leak logger infrastructure.
Download Source Code: click here
Testing procedure
Steps to test involve compilation and verification as a root user from the terminal. Follow the sequence as given below.
1. For compiling our user daemon, issue the following command:
[stextbox id=”grey”]# gcc pmd.c –o pmd –g –I .[/stextbox]
2. For compiling the kernel module (kernel makefile is included):
[stextbox id=”grey”]# make[/stextbox]
3. For installing the module, issue the following command:
[stextbox id=”grey”]# insmod pmm.ko[/stextbox]
4. For starting the daemon, issue the following command:
[stextbox id=”grey”]# ./pmd[/stextbox]
A typical output of this command run in Ubuntu system is shown in the screenshot above (Fig. 1).
5. For viewing memory request logs, issue the following command:
[stextbox id=”grey”]# cat /proc/pmm/pid[/stextbox]
Sample output
Some sample outputs are given below. Issue following commands on the terminal:
[stextbox id=”grey”]# cat /proc/pmm/pid[/stextbox]
The output will be something like this:
[stextbox id=”grey”][1] PID: 21113, Comp: pmd, PPID: 1,
State: 1, Log count: 2[/stextbox]
This means, the pmm proc is registered by the kernel module. It displays the format of pid file memory logs. (Here, PID of ‘pmd’ process is 21113.)
Issue following commands on the terminal:
[stextbox id=”grey”]# ./pmd[/stextbox]
The output will be something like this:
[stextbox id=”grey”]{MALLOC @06:57:11} Sz: 100B Pointer:
0x10012008 Caller: 0x10000e60 main()
207 pmd.c
{REALLOC @06:57:21} Sz: 200B
Prev Pointer: 0x10012008 Pointer:
0x10012008 Caller: 0x10000e84 main()
209 pmd.c[/stextbox]
The meanings of these outputs are given below.
type of log. {@}
Sz.
Pointer.
Caller.
Location. function-name, line-number and file-name.
The above details will give user the system status and health conditions.
To stop the process, issue following commands:
[stextbox id=”grey”]#ps -ef | grep pmd
root 20604 2 0 06:27 ? 00:00:00
[pmthread]
root 21113 1 0 06:30 ? 00:00:00 ./pmd[/stextbox]
Issue the following commands on the terminal:
[stextbox id=”grey”]# cat /proc/pmm/cfg[/stextbox]
The output will be something like this:
[stextbox id=”grey”]Configs:
[“INTERVAL” (Scan Interval) :=10000]
[“SCAN” (Scan on:1/off:0) := 1]
[“START_PID” (Start a monitor):=21113]
[“STOP_PID” (Stop a monitor) := 0][/stextbox]
This output displays configuration objects including interval, scanning and start-and-stop status.
For exanokem INTERVAL configuration is currently set to 10000ms and this can be changed using the following command:
[stextbox id=”grey”]
#Printf “INTERVAL 5000” >> /proc/pmm/cfg
[/stextbox]
To start monitoring on a PID, use the following command:
[stextbox id=”grey”]#Printf “START_PID >> /proc/
pmm/cfg[/stextbox]
L. Karthikeyan is B.E. (computer science) from AC Tech, Karaikudi. He is working as a technical lead, HCL Technologies, Chennai. He likes to read and write articles related to computer science and finance
