Computer Scientist

Monday, 16 January 2012

Terminal Colors

Here's a list of different colors:
30 black foreground
31 red foreground
32 green foreground
33 brown foreground
34 blue foreground
35 magenta (purple) foreground
36 cyan (light blue) foreground
37 gray foreground

40 black background
41 red background
42 green background
43 brown background
44 blue background
45 magenta background
46 cyan background
47 white background

Commands can also be combined using a semicolon, like so:


printf("\033[45;37mGrey on purple.\033[0m");



Finally here is a list of other neat commands that go at the end (where the '0' is):
0 reset all attributes to their defaults
1 set bold
5 set blink
7 set reverse video
22 set normal intensity
25 blink off
27 reverse video off


=====================================================
Another one:


\033[22;30m - black
\033[22;31m - red
\033[22;32m - green
\033[22;33m - brown
\033[22;34m - blue
\033[22;35m - magenta
\033[22;36m - cyan
\033[22;37m - gray
\033[01;30m - dark gray
\033[01;31m - light red
\033[01;32m - light green
\033[01;33m - yellow
\033[01;34m - light blue
\033[01;35m - light magenta
\033[01;36m - light cyan
\033[01;37m - white

Saturday, 19 November 2011

Use LD_PRELOAD to load another version library before running

LD_PRELOAD is a fantastic method to debug the code or library.

Basically, this just record other's work on the LD_PRELOAD and description. That's is fair enough, so I won't bored to modify that:


You Are Here

LD_PRELOAD fun

Here is a welcome digression from my previous Twitter oriented posts. I’m starting to play around with the LD_PRELOAD feature in the Linux dynamic linker. For those who might not know what this feature is, here is the description from ld.so (8).

LD_PRELOAD
              A whitespace-separated list of additional,  user-specified,  ELF
              shared  libraries  to  be loaded before all others.  This can be
              used  to  selectively  override  functions   in   other   shared
              libraries.   For  setuid/setgid  ELF binaries, only libraries in
              the standard search directories that are  also  setgid  will  be
              loaded.
So in pratical term, any libraries you specify in the LD_PRELOAD environment variable will loaded before any system libraries. This means that dynamic symbols in a loading program will be first searched in those libraries before being searched anywhere else. This means you can override any defined symbol you want in standard libraries.
Let’s start with a rather juvenile example. This will change the behavior of the read (2) function in order to make the user believe a file might have a different content.

 
ssize_t read(int fd, void *buf, size_t count) {
    static int done = 0;
    if (!done) {
        char silly_str[] = "Haha you got overriden.\n";
        size_t s = count > sizeof(silly_str) ? sizeof(silly_str) : count;
        memcpy(buf, silly_str, s);
        done = 1;
        return s;
    }
    else return 0;
}
If you compile this inside a library that is called, for example, libread.so, you can test this code by running:
> /bin/cat /etc/fstab
# /etc/fstab: static file system information.
#
...
> LD_LIBRARY_PATH=. LD_PRELOAD=libread.so /bin/cat /etc/fstab
Haha you got overriden.
That in itself is just a rather silly prank you can play on your friend’s computer if you happen to have access to it. Experienced programmer will start seeing potential uses for LD_PRELOAD. I am getting to that.
The subject of our next example will be the honorable ls (1). ls uses the opendir (3) function to open a directory and browse its files. It should react properly if it can’t open the directory. One way to test this is to make opendir() return NULL and observe how the caller reacts. You can do that using LD_PRELOAD.

DIR *opendir(const char *name) {
    return NULL;
}
 
> LD_LIBRARY_PATH=. LD_PRELOAD=libls1.so /bin/ls /tmp
/bin/ls: cannot open directory /tmp
What can you do now if you want to preserve part of the behavior of the function, or modify they result it returns? Your preloaded library will then need to use libdl to dynamically load the function it wants to modify the behavior.
The following example is a very simple override of the opendir (3) function which open a different directory than what the caller expects. I will explain more in detail the details of this function below.

DIR *opendir(const char *name) {
    DIR *(*libc_opendir)(const char *name);
    *(void **)(&libc_opendir) = dlsym(RTLD_NEXT, "opendir");
    return libc_opendir("/tmp");
}

libdl is fortunately very simple to use. The naive approach would be to use dlopen (3) to open the C library, then get the pointer to the function you are calling using dlsym (3). In theory, this technique is valid and working, but doing that circumvents the LD_PRELOAD mechanisme because preloaded libraries can be chained and calling directly into the C library prevents other caller to override our own function.
In practice, calling dlopen() on libc on an Ubuntu Karmic system made some program crash and burn for reasons I will not attempt to explain. The next technique should be preferred on Linux system, especially when dealing with the system C library.
dlsym() has an option that makes the Linux dynamic linker search for the right symbol to be override. This is the RTLD_NEXT flag, which is to be used just for the purpose of wrapper dynamic library functions.
libdl the task of returning the pointer to the right symbol. The RTLD_NEXT option to dlsym() returns the right symbol.
The next and final example of the use of LD_PRELOAD will still use the valiant ls. In time for Christmas, this will modify the output of ls by randomizing the d_type field returned in the dirent structure by readdir (3). If you use colorized ls output, and I believe most of you probably do, you should see a pretty display of color whenever you list a directory by preloading this function.

struct dirent64 *readdir64(DIR *dir) {
    static struct dirent64 *(* libc_readdir64)(DIR *dir) = NULL;
    struct dirent64 *dent;
    unsigned char rnd_dtype[7] = { DT_UNKNOWN, DT_REG,
                                   DT_DIR, DT_FIFO,
                                   DT_SOCK, DT_CHR,
                                   DT_BLK };
    if (libc_readdir64 == NULL) {
        *(void **)(&libc_readdir64) = dlsym(RTLD_NEXT, "readdir64");
        srand(time(NULL));
    }
    dent = libc_readdir64(dir);
    if (dent != NULL)
        dent->d_type = rnd_dtype[rand() % 7];
    return dent;
}
There is still a problem with this code on my new Ubuntu Hardy machine. The code from the preloaded library hangs before the program terminates. I do not understand why this happen and a search for this bug did not turn up anything. The problem doesn’t happen with Ubuntu Karmic.
There is nothing new about using LD_PRELOAD this way. Several very nice libraries have been built with the intention of modifying the behavior of typical libraries.
  • fakeroot: “fakeroot provides a fake root environment by means of LD_PRELOAD and SYSV IPC (or TCP) trickery.”
  • fakechroot: fakechroot provides a fake chroot environment to programs.
  • libtrash:“[...] the shared library which, when preloaded, implements a trash can under GNU/Linux”
  • cowdancer: cowdancer is an userland implementation of copy-on-write filesystem.
There are 29 projects matching LD_PRELOAD on freshmeat.net. You might have used some of them.
The code I have written for this demonstration is available on BitBucket.
Written by fdgonthier
January 11th, 2010 at 10:10 pm
Posted in Debian,Linux,Programming,Tips and Tricks
Tagged with , , , ,

reference:  http://www.lostwebsite.net/2010/01/ld_preload-fun/

Libcage Change Log

1. cage.*pp dump network CF_EXP for logging
2. udphandler.*pp dump network CF_EXP logging
3. dht.*pp dump dht storage CF_EXP logging
4. add elog into libcage to logging
5. add random library to get random number and poisson number.
6. add cf_exp, cf1_exp, cf2_exp for myself experiment.

TODO:
  • migrate to Cmake from Omake(It seems that hopeless)
  • using clock_gettime() to replace gettimeofday()

Timing problem in LInux

Gettimeofday???
Linux provides a 'gettimeofday()' function for users to check the epoch time. However, it gets the system's best guess at wall time. This can go backwards. This is why a 'timer_correct' function is embedded in the Libevent's event.c file. This function will be used when the "MONOTONIC" clock is not in used.


Monotonic clock
In Linux, another function 'clock_gettime(CLOCK_MONOTONIC)' is used to obtain the monotonic time, where the monotonic means that there is no possible to get a time backwards with this function. In this case, this is more reasonable used be used.

"POSIX.1-2008 marks gettimeofday() as obsolete, recommending the use of clock_gettime(2) instead."

Wednesday, 5 October 2011

Allocate memory in a function

I used to try to find some ways to allocate memory in a function. When other functions invoke this function with a null pointer as a parameter, this null pointer would be filled with some contents.

The most silly way to accomplish this task is like this:

void fill_function (void * pointer){
    pointer = (void *) malloc (certain length);

    pointer is filled with some contects;
}

void main () {
    int * file_content;
    fill_function(file_content);

    printf("", file_content);
}

Here, the most important thing which is ignored is the malloc function will allocate the memory in a certain position of the memory which is completely decided by malloc not the one on the left side of the equal symbol. If the file_content pointer is initialized with a number 5(assume that this is a memory address). The fill_function will not allocate the memory space from 5. The malloc will find another free place (for example 102, another memory address) and allocate the continuous memory space after 102, the local pointer variable pointer will be set to be 102. However, the invoking function main has no chance to catch this allocated address.

So a more reasonable way is like this:

int * fill_function (){
    int * pointer = (int *) malloc (certain length);

    return pointer;
}

void main(){
    int * file_content = fill_function();

    printf("", file_content);
}

Another problem is the garbage allocation collection problem. At this point, the boost::shared_ptr<> is encouraged to be used in this situation if you are programming by C++.

Tuesday, 28 June 2011

Sorting mechanism in Java (Refer to other's work)

Java Sorting: Comparator vs Comparable Tutorial

Java Comparators and Comparables? What are they? How do we use them? This is a question we received from one of our readers. This article will discuss the java.util.Comparator and java.lang.Comparable in details with aset of sample codes for further clarifications.

Prerequisites

  • Basic Java knowledge

System Requirements


What are Java Comparators and Comparables?

As both names suggest (and you may have guessed), these are used for comparing objects in Java. Using these concepts; Java objects can be
sorted according to a predefined order.

Two of these concepts can be explained as follows.

Comparable

A comparable object is capable of comparing itself with another object. The class itself must implements the java.lang.Comparable interface in order to be able to compare its instances.

Comparator

A comparator object is capable of comparing two different objects. The class is not comparing its instances, but some other class’s instances. This comparator class must implement the java.util.Comparator interface.

Do we need to compare objects?

The simplest answer is yes. When there is a list of objects, ordering these objects into different orders becomes a must in some situations. For example; think of displaying a list of employee objects in a web page. Generally employees may be displayed by sorting them using the employee id. Also there will be requirements to sort them according to the name or age as well. In these situations both these (above defined) concepts will become handy.

How to use these?

There are two interfaces in Java to support these concepts, and each of these has one method to be implemented by user.
Those are;

java.lang.Comparable: int compareTo(Object o1)
This method compares this object with o1 object. Returned int value has the following meanings.
  1. positive – this object is greater than o1
  2. zero – this object equals to o1
  3. negative – this object is less than o1

java.util.Comparator: int compare(Object o1, Objecto2)
This method compares o1 and o2 objects. Returned int value has the following meanings.
  1. positive – o1 is greater than o2
  2. zero – o1 equals to o2
  3. negative – o1 is less than o1

java.util.Collections.sort(List) and java.util.Arrays.sort(Object[]) methods can be used to sort using natural ordering of objects.
java.util.Collections.sort(List, Comparator) and java.util.Arrays.sort(Object[], Comparator) methods can be used if a Comparator is available for comparison.

The above explained Employee example is a good candidate for explaining these two concepts. First we’ll write a simple Java bean to represent the Employee.

public class Employee {
    private int empId;
    private String name;
    private int age;

    public Employee(int empId, String name, int age) {
        // set values on attributes
    }
    // getters & setters
}

Next we’ll create a list of Employees for using in different sorting requirements. Employees are added to a List without any specific order in the following class.

import java.util.*;

public class Util {
    
    public static List<Employee> getEmployees() {
        
        List<Employee> col = new ArrayList<Employee>();
        
        col.add(new Employee(5, "Frank", 28));
        col.add(new Employee(1, "Jorge", 19));
        col.add(new Employee(6, "Bill", 34));
        col.add(new Employee(3, "Michel", 10));
        col.add(new Employee(7, "Simpson", 8));
        col.add(new Employee(4, "Clerk",16 ));
        col.add(new Employee(8, "Lee", 40));
        col.add(new Employee(2, "Mark", 30));
        
        return col;
    }
}

Sorting in natural ordering

Employee’s natural ordering would be done according to the employee id. For that, above Employee class must be altered to add the comparing ability as follows.

public class Employee implements Comparable<Employee> {
    private int empId;
    private String name;
    private int age;
    
    /**
     * Compare a given Employee with this object.
     * If employee id of this object is 
     * greater than the received object,
     * then this object is greater than the other.
     */
    public int compareTo(Employee o) {
        return this.empId - o.empId ;
    }
    ….
}

The new compareTo() method does the trick of implementing the natural ordering of the instances. So if a collection of Employee objects is sorted using Collections.sort(List) method; sorting happens according to the ordering done inside this method.

We’ll write a class to test this natural ordering mechanism. Following class use the Collections.sort(List) method to sort the given list in natural order.

import java.util.*;

public class TestEmployeeSort {
    
    public static void main(String[] args) {     
        List coll = Util.getEmployees();
        Collections.sort(coll); // sort method
        printList(coll);
    }
    
    private static void printList(List<Employee> list) {
        System.out.println("EmpId\tName\tAge");
        for (Employee e: list) {
            System.out.println(e.getEmpId() + "\t" + e.getName() + "\t" + e.getAge());
        }
    }
}

Run the above class and examine the output. It will be as follows. As you can see, the list is sorted correctly using the employee id. As empId is an int value, the employee instances are ordered so that the int values ordered from 1 to 8.

EmpId Name Age
1 Jorge 19
2 Mark 30
3 Michel 10
4 Clerk 16
5 Frank 28
6 Bill 34
7 Simp 8
8 Lee 40

Sorting by other fields

If we need to sort using other fields of the employee, we’ll have to change the Employee class’s compareTo() method to use those fields. But then we’ll loose this empId based sorting mechanism. This is not a good alternative if we need to sort using different fields at different occasions. But no need to worry; Comparator is there to save us.

By writing a class that implements the java.util.Comparator interface, you can sort Employees using any field as you wish even without touching the Employee class itself; Employee class does not need to implement java.lang.Comparable or java.util.Comparator interface.

Sorting by name field

Following EmpSortByName class is used to sort Employee instances according to the name field. In this class, inside the compare() method sorting mechanism is implemented. In compare() method we get two Employee instances and we have to return which object is greater.

public class EmpSortByName implements Comparator<Employee>{

    public int compare(Employee o1, Employee o2) {
        return o1.getName().compareTo(o2.getName());
    }
}

Watch out: Here, String class’s compareTo() method is used in comparing the name fields (which are Strings).

Now to test this sorting mechanism, you must use the Collections.sort(List, Comparator) method instead of Collections.sort(List) method. Now change the TestEmployeeSort class as follows. See how the EmpSortByName comparator is used inside sort method.

import java.util.*;

public class TestEmployeeSort {
    
    public static void main(String[] args) {
        
        List coll = Util.getEmployees();
        //Collections.sort(coll);
        //use Comparator implementation
        Collections.sort(coll, new EmpSortByName());
        printList(coll);
    }
    
    private static void printList(List<Employee> list) {
        System.out.println("EmpId\tName\tAge");
        for (Employee e: list) {
            System.out.println(e.getEmpId() + "\t" + e.getName() + "\t" + e.getAge());
        }
    }
}

Now the result would be as follows. Check whether the employees are sorted correctly by the name String field. You’ll see that these are sorted alphabetically.

EmpId Name Age
6 Bill 34
4 Clerk 16
5 Frank 28
1 Jorge 19
8 Lee 40
2 Mark 30
3 Michel 10
7 Simp 8

Sorting by empId field

Even the ordering by empId (previously done using Comparable) can be implemented using Comparator; following class
does that.

public class EmpSortByEmpId implements Comparator<Employee>{

    public int compare(Employee o1, Employee o2) {
        return o1.getEmpId() - o2.getEmpId();
    }
}

Explore further

Do not stop here. Work on the followings by yourselves and sharpen knowledge on these concepts.
  1. Sort employees using name, age, empId in this order (ie: when names are equal, try age and then next empId)
  2. Explore how & why equals() method and compare()/compareTo() methods must be consistence.

If you have any issues on these concepts; please add those in the comments section and we’ll get back to you.


Reference : http://lkamal.blogspot.com/2008/07/java-sorting-comparator-vs-comparable.html

Monday, 13 June 2011

Double pointer's usages (on going)

The first usage of the double pointer is to allocate memory for a given pointer:

         nlp = &res.readdir_res_u.list;
while (d = readdir(dirp)) {
nl = *nlp = (namenode *)malloc(sizeof(namenode));
nl->name = strdup(d->d_name);
nlp = &nl->pNext;
}
*nlp = NULL;

This comes from "Power Programming with RPC P84"