In my experiment results, an random error happened quite frequently. But it is not show itself in each time's experiment. It makes me be really frustrated as it is shown in a experiment. The PROBLEM is: an additional extra number always suffix the real number. Due to it happened randomly, which means that no patterns can be found from the situations it is shown, it is difficult to spot the problem's root cause.
What can I do at this point is to skim through the code where it probably happened from. Because this code is written with pure C instead of C++, the most possible reason should be the wrong memory pointer.
After the searching of man docs of the Linux, the reason is found --- the using of 'strncpy'.
When we copy a group of characters to a destination C-type string, a null-termination will be added to the destination automatically as 'strcpy' (without the 'n') is used. There are no problems at all.
However, the 'strncpy' is used, we need to think about the null-termination problem. As the statement of warning in the man doc, if the first 'n' characters of source C-type string have not null-termination in it. This function will not append the '\0' termination to the end of destination string automatically. This is not related to the size of destination string. We only need to be bare in mind the size of n and the first n characters of source string.
SOLUTION: add the '\0' termination to the destination string manually.
Showing posts with label Experiment. Show all posts
Showing posts with label Experiment. Show all posts
Monday, 16 April 2012
Saturday, 19 November 2011
Libcage Change Log
1. cage.*pp dump network CF_EXP for logging
2. udphandler.*pp dump network CF_EXP logging
3. dht.*pp dump dht storage CF_EXP logging
4. add elog into libcage to logging
5. add random library to get random number and poisson number.
6. add cf_exp, cf1_exp, cf2_exp for myself experiment.
TODO:
2. udphandler.*pp dump network CF_EXP logging
3. dht.*pp dump dht storage CF_EXP logging
4. add elog into libcage to logging
5. add random library to get random number and poisson number.
6. add cf_exp, cf1_exp, cf2_exp for myself experiment.
TODO:
- migrate to Cmake from Omake(It seems that hopeless)
- using clock_gettime() to replace gettimeofday()
Friday, 18 February 2011
Experiment Q&A............
When I compiled the lookup program using MySQL connector C++ statically:
Q: what if I am confronted with "ld: can not found lm "?
A: The reason is that 'ld' can not find the static version of 'libm.a' within the default position.
So I need to install it in Fedora using yum: "sudo yum install glibc-static". This should solve the 'lm' problem.
Q: what if I am confronted with message: "undefined reference to 'pthread_self' and so on...."?
A: The problem is that the pthread library can not be found. Just direct the g++ the location of it by append "-lpthread" at the end.
Q: What is the command that I used to compile the program and link against the static library?
A: Like this: "g++ -static -o <dst> <src> /usr/local/lib/libmysqlcppconn-static.a /usr/local/mysql/lib/libmysqlclient.a -lpthread".
Q: When using the static library, sometimes there is a strange message pop out: can not found lc ?
A: Because the static version of glibc is not installed by default, we need to install by ourself:
sudo yum install glibc-static
Q: When using the static library, sometimes there is a strange message pop out: can not found lc ?
A: Because the static version of glibc is not installed by default, we need to install by ourself:
sudo yum install glibc-static
Monday, 7 February 2011
Touch files in a directory recursively
Using the following command is able to touch files recursively:
find . -print0 | xargs -r0 touch
where . is the current directory and the option r of xargs is specific for GNU xargs.
find . -print0 | xargs -r0 touch
where . is the current directory and the option r of xargs is specific for GNU xargs.
Saturday, 15 January 2011
Linux Network Security Issues
Until now, as far as I know, there are at least three different levels of network security mechanism that affects the running of network processes, TCP Wrapper, Iptables firewall, SELinux.
TCP Wrapper and SELinux are host-based mechanism. According to a response to a network packet, TCP Wrapper and SELinux will decide if this packet can be processed by the specific running process. They will not block any network access from other hosts, however, they will limit the running process in the host from processing network messages.
Iptables, on the other hand, provides a network-based security mechanism. It inspects every network packet whenever a packet going into a host or leaving a host. A great amount of distinct functional rules can be set up in order to filter some specific packets. By this way, unwanted packet is forbidden outside of the host.
In my recent experiment, several problems are suffered in these two issues. I'd like to record them here for future reference:
TCP Wrapper and SELinux are host-based mechanism. According to a response to a network packet, TCP Wrapper and SELinux will decide if this packet can be processed by the specific running process. They will not block any network access from other hosts, however, they will limit the running process in the host from processing network messages.
Iptables, on the other hand, provides a network-based security mechanism. It inspects every network packet whenever a packet going into a host or leaving a host. A great amount of distinct functional rules can be set up in order to filter some specific packets. By this way, unwanted packet is forbidden outside of the host.
In my recent experiment, several problems are suffered in these two issues. I'd like to record them here for future reference:
- In school's Fedora 11 system, snmp messages can not going out even if the corresponding port is opened by iptables firewall. When I was using tcpdump to inspect every packet of snmp protocol, I found that snmp request messages were able to go in the system. but there were never any packets coming out. I supposed that two possibilities: snmp crashed, or some other than iptables was keeping block the outgoing messages. Finally, after I searched almost the whole internet (joking, can I? but it is true that it is quite difficult to spot a specific rear problem on the internet.), I realised that I got half correct. There is something called TCP Wrapper which is used by Linux system to prevent some specific daemon processes from accessing from unwanted network hosts. In this case, all of the process other than several processes denoted in the file /etc/hosts.allow is allowed to be accessed in localhost host which means that only local access is allowed. This is right the reason why I was able to query snmp using localhost. In coming snmp messages from other hosts is not allowed to be processed by snmpd (so harsh!!). The solution is simple, just add the snmpd into hosts.allow file.
- The problem regarding to Iptables is a little foolish, but I learnt others when I modified the rules of Iptables. Actually, the reason of I can't transfer files to cspc020 is because I open the wrong port for tcp connection (don't believe the instructions on the webpage totally, this lessoned me). Just open tcp60000 is fine. During this process, i found that, always modify the iptables using iptables command line tools before modify the configuration file /etc/sysconfig/iptables, because the command line is temporary but effects at once. If there are some problems, I am able to resume it by restart computer and then the original configuration will be read. !!!! Good mechanism.
Thursday, 13 January 2011
Strange SNMP problem in Ubuntu or other platform
Description of the problem:
After I have installed SNMPd in Ubuntu 9.04, I was trying to snmpwalk system to test the correction of the installation and deployment. However, I got a Time out message even though I did not get any firewall installed in Ubuntu. At the beginning, I thought that it is the firewall problem like in Fedora. The truth of no firewall installed in Ubuntu by default makes me realise that this is not in that case.
Solution:
Ubuntu is different from Fedora, there is a file to keep the default snmpd's running options, which is:
/etc/default/snmpd. In it, the options denote that only local host is able to snmpwalk the SNMP agent. In this case, I will not only need to change the snmpd.conf configure file, but I will also need to change some lines in the snmpd configure file mentioned above to make it work. The changing is like this:
After I have installed SNMPd in Ubuntu 9.04, I was trying to snmpwalk system to test the correction of the installation and deployment. However, I got a Time out message even though I did not get any firewall installed in Ubuntu. At the beginning, I thought that it is the firewall problem like in Fedora. The truth of no firewall installed in Ubuntu by default makes me realise that this is not in that case.
Solution:
Ubuntu is different from Fedora, there is a file to keep the default snmpd's running options, which is:
/etc/default/snmpd. In it, the options denote that only local host is able to snmpwalk the SNMP agent. In this case, I will not only need to change the snmpd.conf configure file, but I will also need to change some lines in the snmpd configure file mentioned above to make it work. The changing is like this:
#SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid 127.0.0.1' SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid -c /etc/snmp/snmpd.conf
Monday, 13 December 2010
Coding works of the Experiment
Here is the TODO list for experiment coding: some of them have long time to achieve and may be implemented later on due to the current works.
* How to monitor the network bandwidth: the rough idea is to sum the number of packets during a period of time and then to calculate the amount of data per second during this period. At this point, how fine of the records should be concerned.
* How to write a daemon to let the program running background:
* How to stop the experiment program at anytime: Investigate how to used signal to tell the experiment process to terminate. Using IPC (Inter process call)??? The main issue is how to used Nagios to spread this instruction across all of the experiment machine.
* How to monitor the system resources usages:
* How to use automake to compile my own code together with the shared library of chord or sfslite. For this task, I have get some progresses on nagios snmp plugins. change the Makefile.am in ./src directory and run autoreconf --install in package directory and then automake (not sure if this is required). Then configure it and make it. PROBLEMs to be consider: 1. where is configure.ac? Is it optional for autoconf?
ANSWER: configure.ac was called configure.in before autoconf 2.50. It can still be found in Chord and nagios snmp plugins distributions. When you modified some places in Makefile.am files or Configure.ac files, don't do anything except make it. All of the .in files will be regenerated.
* How to monitor the network bandwidth: the rough idea is to sum the number of packets during a period of time and then to calculate the amount of data per second during this period. At this point, how fine of the records should be concerned.
* How to write a daemon to let the program running background:
* How to stop the experiment program at anytime: Investigate how to used signal to tell the experiment process to terminate. Using IPC (Inter process call)??? The main issue is how to used Nagios to spread this instruction across all of the experiment machine.
* How to monitor the system resources usages:
* How to use automake to compile my own code together with the shared library of chord or sfslite. For this task, I have get some progresses on nagios snmp plugins. change the Makefile.am in ./src directory and run autoreconf --install in package directory and then automake (not sure if this is required). Then configure it and make it. PROBLEMs to be consider: 1. where is configure.ac? Is it optional for autoconf?
ANSWER: configure.ac was called configure.in before autoconf 2.50. It can still be found in Chord and nagios snmp plugins distributions. When you modified some places in Makefile.am files or Configure.ac files, don't do anything except make it. All of the .in files will be regenerated.
Friday, 3 December 2010
Dive into more details of sfslite which is used by Chord official implementation
I decided to give up the usage of Openchord because it is quite difficult to use. I will back to Chord Official implementation.
:
One thing that is a little difficult to understand is in the Lesson 3: more than one callback are in the same procedure.
Line 27 and 28 can be merged into "delaycb (1, 0, wrap(docallback, wrap(hello)));"
I think I could understand it like this:
each wrap() will produce a callback version of a function (type of callback::ref
wrap(hello) provides a callback version of hello() method to function docallback( callcallback::ref ) ::ref)
wrap(docallback, wrap(hello)) provides a callback version of docallback(callback::ref) to delaycb and asks delaycb() to register docallback(callback::ref) in event queue.
========Bear in mind, the return value of wrap() is callback::ref=========
Here is the graphical expression of this mechanism:
Here I refresh some knowledge of sfslite and provide some useful web pages to understanding its code.
[Main Page]: http://www.okws.org/doku.php?id=sfslite
[Overview of sfslite]: http://www.okws.org/doku.php?id=sfslite:overview
[Some examples of usage]: http://www.okws.org/doku.php?id=sfslite:qhash
[async Tutorial]: http://pdos.csail.mit.edu/6.824-2004/async/
[Tips of using]: http://www.okws.org/doku.php?id=sfslite:tips
[Installation Guide]: http://www.okws.org/doku.php?id=sfslite:install
One thing that is a little difficult to understand is in the Lesson 3: more than one callback are in the same procedure.
Line 27 and 28 can be merged into "delaycb (1, 0, wrap(docallback, wrap(hello)));"
I think I could understand it like this:
each wrap() will produce a callback version of a function (type of callback
wrap(hello) provides a callback version of hello() method to function docallback( callcallback
wrap(docallback, wrap(hello)) provides a callback version of docallback(callback
========Bear in mind, the return value of wrap() is callback
Here is the graphical expression of this mechanism:
The usages of callback (callback::ref):
(1). it can be passed into a callback function like docallback as its parameter and be invoked in this callback function cb();
(2). it can be registered by a underlaying mechanism like delaycb.
Thursday, 13 May 2010
Hard working on Compilation
How to extract tar file in Linux with additional appendix.
tar -jxvf ***.tar.bz2
tar -zxvf ***.tar.gz
Today, I have to change my work to Fedora Linux due to the stuck on my Mac.
Firstly, I am using gmp 4.3.2 instead of 5.0.1.
Secondly, I am going to install sfslite. Let's try 0.8.17 firstly. and then 1.0.0.
0.8.17 still does not work like the situation in my Mac.
tar -jxvf ***.tar.bz2
tar -zxvf ***.tar.gz
Today, I have to change my work to Fedora Linux due to the stuck on my Mac.
Firstly, I am using gmp 4.3.2 instead of 5.0.1.
Secondly, I am going to install sfslite. Let's try 0.8.17 firstly. and then 1.0.0.
0.8.17 still does not work like the situation in my Mac.
Tuesday, 11 May 2010
<--Complete!!-->Concret Working Day
There are two works to do today:
1. Complete the experimental plan at least the single machine experiments on Chord as mentioned in yesterday's meeting.
2. Begin the experiment of Chord by construct the experimental environment.
A little reversal according to the above descriptions is that I am beginning with the experimental environment construction. I will record the whole process for future usage:
I am constructing the environment on my Mac laptop, this means that there are some risks that some problems might emerge when I working on the Fedora Linux where is the practice environment. Just keep this in mind is fine.
Source files are settled in ~/src/
Each package has its own build directory in ~/build/
From the very beginning, I have nothing installed on my mac. I have to CMM(configure-make-make_install) everything. There is a dependency across the serials of packages which is considered to be required and necessary.
Chord ---> sfslite (SFS) ---> GMP
Chord ---> Berkeley DB
Chord visualisation ---> vis ---> GTK+ (for Mac, Maybe not necessary for Linux)
Here I list the corresponding websites of these toolkits:
Chord: http://pdos.csail.mit.edu/chord/howto.html
sfslite: http://www.okws.org/doku.php?id=sfslite
GMP: http://gmplib.org/
vis: (build-in on Mac, so not sure if GTK+is necessary The build-in vis is totally different. Need to install the vis provided by Chord separately.)
GTK+ (for Mac): http://gtk-osx.sourceforge.net/
CMM process:
1. Build GMP:
no special, just CMM. make sure using "make check" to check the compile results after "make" it.
In the later build, I added a "--enable-cxx" options when I configure it.
2. Build sfslite:
need to append the "--with-sfsmisc" after "configure". "make check" is also available but nothing to check.
The version is changed from 1.2.7 to 0.8.16 because some problems emerging when configure chord.
There is a long story here. Choosing the right version is the most crucial step.
3. Build Berkeley DB 4.8.26
Not sure whether my settings are right or not. Just test it later on.
Configure process:
/dist/configure --enable-compat185 --enable-cxx
when I was trying to add --enable-java there is something wrong. a file can not be founded. So I "make realclean" to wipe out every thing.
Make process:
When I neglected --enable-java, everything worked smoothly.
Install process;
It is much less than I supposed. So I successfully installed berkeley DB.
3. Build Berkeley DB 4.4.20
It works much more smoothly than 4.8.26. One thing that I noticed is that 4.4.20 is released by sleepycat, while 4.8.26 is released by Oracle.
I used the quite same options that I used in 4.8.26 including --enable-java. It works good this time.
Maybe 4.8.26 require a rather newer version of JDK, because the Java in Mac is not be updated for ages.
After checking the configure.in file of Chord source, I found the accepted sfslite version is up to 4.6 from 3. <<CAUTION:>> So I changed the sfslite from version 4.4.20 to version 4.6.20 in experiments after 26/05/2013.
Dramatically, the error moved to SFSlite side. the configure tool require a latest version of SFSlite!!!
大哥！！source 文件不能乱考好不好？？！！再试试，全部重装。
4. Build Chord 1.0
Quite normal process. Follow the instructions on the website where the detailed description is stated.
Something should be noticed:
1. Choose the right version for each tool:
GMP 5.0.1
sfslite 0.8.17 (obtain from svn)
Berkeley DB 4.4.20
Chord 0.1 (lastest version)
When retrieve the source, do use CVS, SVN or Mercurial (this kind of tools) because the tar ball of sfslite 0.8.17 in the snapshot is not complete without "svc" directory.
2. Get ride of "cred" error when build sfslite.
Change ./async/sysconf.h according to the latest version of the file in sfslite 1.2.7. This problem is because the latest version of Linux has no "creb" definition.
1. Complete the experimental plan at least the single machine experiments on Chord as mentioned in yesterday's meeting.
2. Begin the experiment of Chord by construct the experimental environment.
A little reversal according to the above descriptions is that I am beginning with the experimental environment construction. I will record the whole process for future usage:
I am constructing the environment on my Mac laptop, this means that there are some risks that some problems might emerge when I working on the Fedora Linux where is the practice environment. Just keep this in mind is fine.
Source files are settled in ~/src/
Each package has its own build directory in ~/build/
From the very beginning, I have nothing installed on my mac. I have to CMM(configure-make-make_install) everything. There is a dependency across the serials of packages which is considered to be required and necessary.
Chord ---> sfslite (SFS)
Chord ---> Berkeley DB
Chord visualisation ---> vis ---> GTK+ (for Mac, Maybe not necessary for Linux)
Here I list the corresponding websites of these toolkits:
Chord: http://pdos.csail.mit.edu/chord/howto.html
sfslite: http://www.okws.org/doku.php?id=sfslite
GMP: http://gmplib.org/
vis: (build-in on Mac, so not sure if GTK+is necessary The build-in vis is totally different. Need to install the vis provided by Chord separately.)
GTK+ (for Mac): http://gtk-osx.sourceforge.net/
CMM process:
1. Build GMP:
no special, just CMM. make sure using "make check" to check the compile results after "make" it.
In the later build, I added a "--enable-cxx" options when I configure it.
2. Build sfslite:
need to append the "--with-sfsmisc" after "configure". "make check" is also available but nothing to check.
The version is changed from 1.2.7 to 0.8.16 because some problems emerging when configure chord.
There is a long story here. Choosing the right version is the most crucial step.
3. Build Berkeley DB 4.8.26
Not sure whether my settings are right or not. Just test it later on.
Configure process:
/dist/configure --enable-compat185 --enable-cxx
when I was trying to add --enable-java there is something wrong. a file can not be founded. So I "make realclean" to wipe out every thing.
Make process:
When I neglected --enable-java, everything worked smoothly.
Install process;
It is much less than I supposed. So I successfully installed berkeley DB.
3. Build Berkeley DB 4.4.20
It works much more smoothly than 4.8.26. One thing that I noticed is that 4.4.20 is released by sleepycat, while 4.8.26 is released by Oracle.
I used the quite same options that I used in 4.8.26 including --enable-java. It works good this time.
Maybe 4.8.26 require a rather newer version of JDK, because the Java in Mac is not be updated for ages.
After checking the configure.in file of Chord source, I found the accepted sfslite version is up to 4.6 from 3. <<CAUTION:>> So I changed the sfslite from version 4.4.20 to version 4.6.20 in experiments after 26/05/2013.
Dramatically, the error moved to SFSlite side. the configure tool require a latest version of SFSlite!!!
大哥！！source 文件不能乱考好不好？？！！再试试，全部重装。
4. Build Chord 1.0
Quite normal process. Follow the instructions on the website where the detailed description is stated.
Something should be noticed:
1. Choose the right version for each tool:
GMP 5.0.1
sfslite 0.8.17 (obtain from svn)
Berkeley DB 4.4.20
Chord 0.1 (lastest version)
When retrieve the source, do use CVS, SVN or Mercurial (this kind of tools) because the tar ball of sfslite 0.8.17 in the snapshot is not complete without "svc" directory.
2. Get ride of "cred" error when build sfslite.
Change ./async/sysconf.h according to the latest version of the file in sfslite 1.2.7. This problem is because the latest version of Linux has no "creb" definition.
Tuesday, 9 March 2010
Working on aMule Compilation
During most time today, I was compiling the aMule and trying to understand each option's meaning. I realized one thing, which is that I really need to learn how to use GNU make tool. A useful book is downloaded. I will go through it and get the main points regarding to my experiment from there.
Today's main progress is:
I have compiled aMule's source code (2.2.6) and installed the accomplishment successfully.
I have also found an aMule wiki page regarding "HOW TO compile aMule on Mac". I have to say that aMule community is so powerful and friendly. I suppose that there must be the manuel of compilation for aMule for all mainstream platform. Everything can be found!!!!
http://wiki.amule.org/index.php/HowTo_compile_on_Mac
Further works is:
Today's main progress is:
I have compiled aMule's source code (2.2.6) and installed the accomplishment successfully.
I have also found an aMule wiki page regarding "HOW TO compile aMule on Mac". I have to say that aMule community is so powerful and friendly. I suppose that there must be the manuel of compilation for aMule for all mainstream platform. Everything can be found!!!!
http://wiki.amule.org/index.php/HowTo_compile_on_Mac
Further works is:
- learn how to use GNU make tool.
- << Done! >
- It is time to write a story and find out some objectives instead of just doing computer geek stuff!! Try to think what should be involved. The previous papers regarding simulation would be helpful.
Subscribe to:
Posts (Atom)