[TCP/IP Stack] Socket not closed on one side causes FIN_WAIT_2, CLOSE_WAIT sockets for the program duration.
PRODUCT: TCP/IP Stacks
OP/SYS: All Operating Systems
COMPONENT: Java All versions
SOURCE: Philippe Vouters Fontainebleau/France
HIGH QUALITY MOBILES+TABLETS: http://android-land.fr
SYMPTOM(S) or PROBLEM(S): On Unix systems, the command: $ netstat -an | grep -i wait or its OpenVMS equivalent: $ @sys$manager:tcpip$define_commands $ pipe netstat -an | search sys$pipe: wait may reveal sockets in ever FIN_WAIT_2, CLOSE_WAIT states.
SOLUTION or HP RESPONSE: On the CLOSE_WAIT side: The CLOSE_WAIT state socket remains permanently allocated to the program until finished or aborted. Check for missing close statement on the CLOSE_WAIT side of the TCP/IP communication. If you run on a Unix or Linux operating system, you may use lsof from a root account using the command: # lsof | grep tcp | \ grep <first network port from netstat -an in CLOSE_WAIT state> This way you will know which program is missing a close statement. On OpenVMS systems, you may know if running HP TCP/IP for OpenVMS which program is missing a close statement. From the first network port information in netstat -an in a CLOSE_WAIT state, perform the following: $ TCPIP SHOW DEVICE/PORT=x This shows a BG device. Then: $ SHOW DEVICE BGxxxx:/FULL This gives the process owner of the device. For Java programs check for objets leaks, these objets refering to a TCP/IP communication which would not be garbage collected. These CLOSE_WAIT state sockets remain permanently allocated to the process until you abort it. They reflect a bug in the code. On the FIN_WAIT_2 side: You can make them disappear using sysctl (Linux), ndd (Solaris or HP-UX) by acting upon the FIN_WAIT_2 timeout. On OpenVMS and HP Tru64, you may also act upon the FIN_WAIT_2 timeout delay (refer to another article mentionned in the REFERENCE section below.)
ANALYSIS: On OpenVMS and HP-UX B.11.23, the problem is indeed visible using standard TCP/IP settings. The FIN_WAIT_2 awaits for a FIN ACK from the peer side which does not come due to the missing close statement triggering the FIN ACK. On HP-UX, it is possible to act upon the tcp_fin_wait_2_timeout ndd parameter to avoid too many sockets in FIN_WAIT_2 state. It manifests itself the following way on OpenVMS V8.3: HALLES > cc/define=(FD_SETSIZE=32768,FIN_WAIT_2_HANG) server HALLES > cc client HALLES > link server HALLES > link client HALLES > server :== $user3:[vouters]server HALLES > client :== $user3:[vouters]client HALLES > spawn/nowait server 50000 %DCL-S-SPAWNED, process VOUTERS_12853 spawned HALLES > Bind name to ls. Listen on ls for connections. Selecting ... HALLES > spawn/nowait client localhost 50000 %DCL-S-SPAWNED, process VOUTERS_39930 spawned HALLES > Gethostbyname for remote nodename. Open socket 1: AF_INET, SOCK_STREAM. Connect socket 1 to sock2_name. after : emask=0x0 rmask=0x8 ret=1 Accept connection from ls Selecting ... after : emask=0x0 rmask=0x10 ret=1 recv()==0 => peer disconnected... Selecting ... HALLES > pipe netstat -an | search sys$pipe: wait tcp 0 0 127.0.0.1.50000 127.0.0.1.49202 CLOSE_WAIT tcp 0 0 127.0.0.1.49202 127.0.0.1.50000 FIN_WAIT_2 On Linux Fedora 8, the problem is partly visible (no FIN_WAIT_2 state, but a CLOSE_WAIT state) because of TCP system (sysctl) tuning parameters. [philippe@victor ~]$ uname -a Linux victor.vouters.dyndns.org 2.6.24.4-64.fc8 #1 SMP i Sat Mar 29 09:02:14 EDT 2008 i686 i686 i386 GNU/Linux [philippe@victor ~]$ cc -DFIN_WAIT_2_HANG -o server server.c [philippe@victor ~]$ cc -o client client.c [philippe@victor ~]$ ./server 50000 & [1] 11034 [philippe@victor ~]$ Bind name to ls. Listen on ls for connections. Selecting ... [philippe@victor ~]$ ./client localhost 50000 & [2] 11036 Gethostbyname for remote nodename. [philippe@victor ~]$ Open socket 1: AF_INET, SOCK_STREAM. Connect socket 1 to sock2_name. after : emask=0x0 rmask=0x8 ret=1 Accept connection from ls Selecting ... after : emask=0x0 rmask=0x10 ret=1 recv()==0 => peer disconnected... Selecting ... [philippe@victor ~]$ netstat -an | grep -i wait tcp 0 0 127.0.0.1:50000 127.0.0.1:54105 CLOSE_WAIT [philippe@victor ~]$ /sbin/sysctl -a | grep -i wait net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120 net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60 net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
JAVA CONSIDERATION: It is very common in Java to give the role of closing the sockets to the Garbage Collector. If there is an object leak, this object refering to a socket on one side of the peer-to-peer communication, you ought to incur such a problem. The way to prove the role of closing sockets is indeed attributed to the Java Garbage Collector is to repeatedly issue the command: On Unix systems: $ netstat -an | grep -i established | wc -l On OpenVMS systems: $ @sys$manager:tcpip$define_commands $ pipe netstat -an | search/statistics sys$pipe: established |- search sys$pipe: "Records matched:" you should notice a number which grows until it reaches a peak and finally decreases somehow, moving from a low to a high limit.
REPRODUCERS: ************** * server.c ************** #include <errno.h> #include <sys/types.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <sys/socket.h> #include <signal.h> #ifdef sun #include <sys/sockio.h> #endif #include <netinet/in.h> #include <netdb.h> #include <arpa/inet.h> #include <sys/uio.h> #include <sys/ioctl.h> #include <unistd.h> /* * Functional Description * * This example creates a socket of type SOCK_STREAM (TCP), * binds and listens on the socket, receives a message * and closes the connection. * Error messages are printed to the screen. * * IPC calls used: * accept * bind * close * gethostbyname * listen * recv * shutdown * socket * * * Formal Parameters * The server program expects one parameter: * portnumber ... port number where it will listen * * * Routine Value * * Status */ #ifndef FD_SET #ifndef __DECC struct fd_set_struct {u_char fds_bits[64/8];}; typedef struct fd_set_struct fd_set; #endif #define NFDBITS sizeof(fd_set)/sizeof (u_char) #define FD_SETSIZE NFDBITS #define FD_SET(n, p) ((p)->fds_bits[(n)/NFDBITS] |= (1 << ((n) % NFDBITS))) #define FD_CLR(n, p) ((p)->fds_bits[(n)/NFDBITS] &= ~(1 << ((n) % NFDBITS))) #define FD_ISSET(n, p) ((p)->fds_bits[(n)/NFDBITS] & (1 << ((n) % NFDBITS))) #define FD_ZERO(p) memset((char *)(p), 0,sizeof(*(p))) #endif int ls; cleanup(int shut,int s) { int retval; /* * Shutdown and close sock1 completely. */ if (shut){ retval = shutdown(s,2); if (retval == -1) perror ("shutdown"); } retval = close (s); if (retval) perror ("close"); } /* end cleanup*/ void sighandler(int sig){ if (sig == SIGINT){ cleanup(0, ls); exit(EXIT_SUCCESS); } } void declsighandler(){ struct sigaction action; sigemptyset(&action.sa_mask); sigaddset(&action.sa_mask,SIGINT); action.sa_flags = 0; action.sa_handler = sighandler; sigaction(SIGINT,&action,NULL); } /*--------------------------------------------------------------------*/ main(int argc, char **argv) { int s; /* sockets */ char message[BUFSIZ]; struct sockaddr_in s_name; /* Address struct for s.*/ struct hostent *hp; /* Pointer to hostent data. */ char hostname[256]; /* Name of local host. */ int flag; int opt=1; int retval; /* helpful for debugging */ int status; #ifdef __VMS size_t namelength; #else int namelength; #endif int atmark; fd_set imask,rmask,wmask,emask; int ret; /* * Check input parameters. */ if (argc != 2 ){ printf("Usage: server portnumber.\n"); exit(EXIT_FAILURE); } /* * Open socket 2: AF_INET, SOCK_STREAM. */ if ((ls = socket (AF_INET, SOCK_STREAM, 0)) == -1){ perror( "socket"); exit(EXIT_FAILURE); } if (setsockopt(ls,SOL_SOCKET,SO_REUSEADDR,&opt,sizeof opt)<0){ printf ("setsockopt (SO_RESUSEADDR): %s\r\n",strerror(errno)); exit(EXIT_FAILURE); } /* * Get the host local name. */ retval = gethostname(hostname,sizeof hostname); if (retval) { perror ("gethostname"); cleanup (0,ls); exit(EXIT_FAILURE); } /* * Get pointer to network data structure for socket 2. */ if ((hp = gethostbyname (hostname)) == NULL) { perror( "gethostbyname"); cleanup(0, ls); exit(EXIT_FAILURE); } /* * Fill in the name & address structure for socket 2. */ s_name.sin_family = hp->h_addrtype; s_name.sin_port = htons(atoi(argv[1])); s_name.sin_addr.s_addr = htonl(INADDR_ANY); /* * Bind name to socket 2. */ printf(" \t Bind name to ls. \n"); retval = bind (ls,(struct sockaddr *)&s_name, sizeof s_name); if (retval) { perror("bind"); cleanup(0, ls); exit(EXIT_FAILURE); } /* * Listen on socket 2 for connections. */ printf(" \t Listen on ls for connections. \n"); retval = listen (ls, 5); if (retval) { perror("listen"); cleanup(0, ls); exit(EXIT_FAILURE); } declsighandler(); FD_ZERO(&imask); FD_SET(ls,&imask); for (;;){ loop:; memcpy(&rmask,&imask,sizeof imask); memcpy(&wmask,&imask,sizeof imask); memcpy(&emask,&imask,sizeof imask); printf(" Selecting ...\n"); ret=select(getdtablesize(),&rmask, NULL,&emask,NULL); printf(" after : emask=0x%x rmask=0x%x ret=%d\n\n", *((unsigned long *)&emask), *((unsigned long *)&rmask), ret); if (ret == -1){ perror ("Select"); exit(EXIT_FAILURE); } for (s=0;s<getdtablesize();s++){ if (FD_ISSET(s,&rmask)){ if (s == ls){ /* * Accept connection from socket 2: * accepted connection will be on socket 3. */ printf(" \t Accept connection from ls \n"); namelength = sizeof (s_name); s = accept (ls, (struct sockaddr *)&s_name, &namelength); if (s == -1) { perror ("accept"); cleanup(0, ls); exit(EXIT_FAILURE); } FD_SET(s,&imask); continue; } if (recv(s, message ,1,0)==0){ printf ("\t recv()==0 => peer disconnected...\n"); #ifndef FIN_WAIT_2_HANG /* * Forgetting to close the socket on only one side causes * the sockets to stay in CLOSE_WAIT/FIN_WAIT_2 as long as * the program is not exited or aborted. */ cleanup(1,s); #endif FD_CLR(s,&imask); continue; } else printf ("NORMAL message: %s\n",message); } if FD_ISSET(s,&emask){ retval = recv(s, message ,sizeof (message), MSG_OOB); printf(" \t -> retval = %d.\n",retval); /* ped */ if (retval == -1) { if (errno == EINVAL) { ioctl(s,SIOCATMARK,&atmark); printf("\t \t atmark = %d\n",atmark); printf ("\t \t PEEK ret = %d\n",recv(s, message ,1,MSG_PEEK)); printf (" \t PEEK message: %s\n", message); continue; } else { perror ("receive"); cleanup( 1, s); exit(EXIT_FAILURE); } } else printf (" \t OOB message: %s\n", message); } } } /* * Call cleanup to shutdown and close sockets. */ cleanup(0, ls); exit(EXIT_SUCCESS); } /* end main */ ************** * client.c ************** #include <stdio.h> #include <errno.h> #include <stdlib.h> #include <unistd.h> #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <netdb.h> #include <arpa/inet.h> #ifdef __VMS #include <unixio.h> #else #include <sys/uio.h> #endif /* * * MACRO DEFINITIONS * */ #ifndef vms #define TRUE 1 #define FALSE 0 #endif /* * Functional Description * * This example creates a socket of type SOCK_STREAM (TCP), * initiates a connection to the remote host, sends * a message to the remote host, and closes the connection. * Error messages are printed to the screen. * * IPC calls used: * close * connect * gethostbyname * send * shutdown * socket * * * Formal Parameters * The client program expects two parameters: * hostname ... name of remote host * portnumber ... port where remote host(server) is listening * * * Routine Value * * Status */ /*-----------------------------------------------------------*/ void cleanup(int shut, int s) { int retval; /* * Shutdown socket completely -- only if it was connected */ if (shut) { retval = shutdown(s,1); if (retval == -1) perror ("shutdown"); } /* * Close socket. */ retval = close (s); if (retval) perror ("close"); // exit(0); } /* end cleanup */ /*--------------------------------------------------------------------*/ main(int argc,char **argv){ int sock_1; /* socket */ static char message[] = "Hi there."; struct sockaddr_in sock2_name; /* Address struct for socket2.*/ struct hostent hostentstruct; /* Storage for hostent data. */ struct hostent *hp; /* Pointer to hostent data. */ static char hostname[256]; /* Name of local host. */ int flag; int retval; /* helpful for debugging */ int shut = 0; /* flag to cleanup */ int status; /* For return status - ped - */ /* * Check input parameters. */ if (argc != 3 ) { printf("Usage: client hostname portnumber.\n"); exit(1); } /* *Get pointer to network data structure for socket 2 (remote host). */ printf(" \t Gethostbyname for remote nodename. \n"); if ((hp = gethostbyname (argv[1])) == NULL){ perror( "gethostbyname"); exit(1); } /* * Copy hostent data to safe storage. */ hostentstruct = *hp; /* * Fill in the name & address structure for socket 2. */ sock2_name.sin_family = hostentstruct.h_addrtype; sock2_name.sin_port = htons(atoi(argv[2])); sock2_name.sin_addr = * ((struct in_addr *) hostentstruct.h_addr); while(1){ /* * Open socket 1: AF_INET, SOCK_STREAM. */ printf(" \t Open socket 1: AF_INET, SOCK_STREAM. \n"); if ((sock_1 = socket (AF_INET, SOCK_STREAM, 0)) == -1){ perror( "socket"); exit(1); } /* * Connect socket 1 to sock2_name. */ printf(" \t Connect socket 1 to sock2_name. \n"); retval = connect(sock_1, (struct sockaddr *)&sock2_name, sizeof (sock2_name)); if (retval){ perror("connect"); cleanup(shut, sock_1); } cleanup(FALSE, sock_1); pause(); }/* end while(1)*/ } /* end main */
REFERENCE(S): For TCP/IP states diagram, refer to the following URL: http://tangentsoft.net/wskfaq/articles/debugging-tcp.html For OpenVMS systems, a related document can be found at: ../tima/OpenVMS-TCPIP-FIN_WAIT_2_state_sockets.html