Codebase list nsntrace / upstream/4.5.g4d02e74
Import upstream version 4.5.g4d02e74 Debian Janitor 1 year, 11 months ago
11 changed file(s) with 294 addition(s) and 126 deletion(s). Raw diff Collapse all Expand all
0
01 # nsntrace
12 > Perform network trace of a single process by using network namespaces.
23
3 This application uses Linux network namespaces to perform network traces of a single application. The traces are saved as pcap files. And can later be analyzed by for instance Wireshark.
4 This application uses Linux network namespaces to perform a network trace of a single application. The trace is saved as a pcap file. And can later be analyzed by for instance Wireshark or tshark.
45
56 The nsntrace application is heavily inspired by the askbubuntu reply [here](https://askubuntu.com/a/499850).
67 And uses the same approach only confined to a single C program.
78
8 What the application does is use the clone syscall to create a new
9 network namespace (CLONE_NEWNET) and from that namespace launch the
10 requested process as well as start a trace using libpcap. This will ensure that all
11 the packets we trace come from the process.
9 What the application does is use the clone syscall to create a new network namespace (CLONE_NEWNET) and from that namespace launch the requested process as well as start a trace using libpcap. This will ensure that all the packets we trace come from the process.
1210
13 The problem we are left with is that the process is isolated in the
14 namespace and cannot reach any other network. We get around that by
15 creating virtual network interfaces. We keep one of them in the
16 root network namespace and but the other one in the newly created one where
17 our tracing takes place. We set the root namespaced one as the default gw
18 of the trace namespaced virtual device.
11 The problem we are left with is that the process is isolated in the namespace and cannot reach any other network. We get around that by creating virtual network interfaces. We keep one of them in the root network namespace and but the other one in the newly created one where our tracing takes place. We set the root namespaced one as the default gateway of the trace namespaced virtual device.
1912
20 And then to make sure we can reach our indented net, we use ip
21 tables and NAT to forward all traffic from the virtual device to our
22 default network interface.
13 And then to make sure we can reach our intended net, we use iptables and NAT to forward all traffic from the virtual device to our default network interface.
2314
24 This will allow us to capture the packets from a single process while
25 it is communicating with our default network. A limitation is that our
26 ip address will be the NAT one of the virtual device.
15 This will allow us to capture the packets from a single process while it is communicating with our default network. A limitation is that our ip address will be the NAT one of the virtual device.
2716
28 Another limitation is, that since we are using iptables and since
29 we are tracing raw sockets. This application needs to be run as root.
17 Another limitation is, that since we are using iptables and since we are tracing raw sockets. This application needs to be run as root.
3018
31 On many systems today the nameserver functionality is handled by an
32 application such as systemd-resolved or dnsmasq and the nameserver
33 address is a loopback address (like 127.0.0.53) where that application
34 listens for incoming DNS queries.
19 On many systems today the nameserver functionality is handled by an application such as systemd-resolved or dnsmasq and the nameserver address is a loopback address (like 127.0.0.53) where that application listens for incoming DNS queries.
3520
36 This will not work for us in this network namespace environment, since
37 we have our own namespaced loopback device. To work around this one can use the
38 `--use-public-dns` option to override resolv.conf in the namespace. Then nsntrace
39 will use nameservers from Quad9 (9.9.9.9), Cloudflare (1.1.1.1), Google (8.8.8.8) and
40 OpenDNS (208.67.222.222) to perform DNS queries.
21 This will not work for us in this network namespace environment, since we have our own namespaced loopback device. To work around this one can use the `--use-public-dns` option to override resolv.conf in the namespace. Then nsntrace will use nameservers from Quad9 (9.9.9.9), Cloudflare (1.1.1.1), Google (8.8.8.8) and OpenDNS (208.67.222.222) to perform DNS queries.
4122
4223 ## usage
4324 $ nsntrace
9778 13 0.113167 142.250.74.36 → 172.16.42.255 TCP 66 80 → 51088 [FIN, ACK] Seq=13815 Ack=143 Win=66816 Len=0 TSval=2846449554 TSecr=1362504583
9879 14 0.113190 172.16.42.255 → 142.250.74.36 TCP 66 51088 → 80 [ACK] Seq=143 Ack=13816 Win=64128 Len=0 TSval=1362504595 TSecr=2846449554
9980 ```
81
82 # building from source
83 To build nsntrace from source the following steps are needed:
84
85 ```
86 $ ./autogen.sh
87 $ ./configure
88 $ make
89 ```
90 And to install:
91 ```
92 $ sudo make install
93 ```
94
95 ## dependencies
96 The packages needed to build nsntrace are (Debian/Ubuntu style naming):
97 * automake
98 * docbook-xml
99 * docbook-xsl
100 * iptables
101 * libnl-route-3-dev
102 * lippcap-dev
103 * pkg-config
104 * xsltproc
5454 * to interface with the Netlink API and perform network configuration.
5555 */
5656
57 #define IP_BASE "172.16.42"
58 #define NS_IP IP_BASE ".255"
59 #define GW_IP IP_BASE ".254"
60 #define NS_IP_RANGE NS_IP "/31"
61 #define GW_IP_RANGE GW_IP "/31"
62
63 #define IF_BASE "nsntrace"
64 #define NS_IF IF_BASE "-netns"
65 #define GW_IF IF_BASE
57 #define NS_IF_BASE "nscap"
58 #define GW_IF_BASE "nsgw"
59
60
6661
6762 static struct nl_sock *
6863 _nsntrace_net_get_nl_socket() {
292287 _nsntrace_net_create_resolv_conf()
293288 {
294289 int fd;
290 char resolv_path[PATH_MAX] = { 0, };
295291 const char *resolv = "nameserver 9.9.9.9\n" /* Quad9 */
296292 "nameserver 1.1.1.1\n" /* Cloudflare */
297293 "nameserver 8.8.8.8\n" /* Google */
298294 "nameserver 208.67.222.222\n"; /* OpenDNS */
299295
300 if (mkdir(NSNTRACE_RUN_DIR, 0644) < 0) {
301 if (errno != EEXIST) {
302 perror("mkdir");
303 return;
304 }
305 }
306
307 char resolv_path[] = NSNTRACE_RUN_DIR "/resolv.confXXXXXX";
296
297 snprintf(resolv_path, PATH_MAX, "%s/%d/resolv.confXXXXXX",
298 NSNTRACE_RUN_DIR, getppid());
308299 if ((fd = mkstemp(resolv_path)) < 0) {
309300 perror("mkstemp");
310301 return;
393384 }
394385
395386 /*
387 * Attempt to find a free ip base (172.19.x) to use. We use randomization
388 * and check addresses that are already bound to our devices, this is to
389 * enable us to run multiple instances of nsntrace.
390 */
391 static const char *
392 _nsntrace_net_get_ip_base()
393 {
394 char buf[1024];
395 char ip[16];
396 struct ifconf ifc;
397 struct timeval tval;
398 int s;
399 int i;
400
401 s = socket(AF_INET, SOCK_DGRAM, 0);
402 if(s < 0) {
403 perror("socket");
404 return NULL;
405 }
406
407 gettimeofday(&tval, NULL);
408 srand(tval.tv_usec);
409 while (1) {
410 int digit = rand() % 255;
411 snprintf(ip, 16, "172.19.%d", digit);
412
413 /* query available interfaces. */
414 ifc.ifc_len = sizeof(buf);
415 ifc.ifc_buf = buf;
416 if(ioctl(s, SIOCGIFCONF, &ifc) < 0)
417 {
418 perror("ioctl");
419 return NULL;
420 }
421
422 /* iterate through the list of interfaces. */
423 int found = 0;
424 for(i = 0; i < (ifc.ifc_len / sizeof(struct ifreq)); i++)
425 {
426 struct ifreq *item = &ifc.ifc_req[i];
427 char *addr = inet_ntoa(((struct sockaddr_in *)&item->ifr_addr)->sin_addr);
428
429 if (!strncmp(addr, ip, strlen(ip))) {
430 found = 1;
431 break;
432 }
433 }
434 if (found == 0) {
435 return strdup(ip);
436 }
437 }
438 return NULL;
439 }
440
441 int
442 nsntrace_net_get_if_info(pid_t pid,
443 struct nsntrace_if_info *info)
444 {
445 const char *ip_base = _nsntrace_net_get_ip_base();
446 if (!ip_base) {
447 fprintf(stderr, "failed to allocate IP address");
448 return -1;
449 }
450
451 snprintf(info->ns_if, IFNAMSIZ, "%s%d", NS_IF_BASE, pid);
452 snprintf(info->gw_if, IFNAMSIZ, "%s%d", GW_IF_BASE, pid);
453 snprintf(info->ns_ip, IP_ADDR_LEN, "%s.255", ip_base);
454 snprintf(info->gw_ip, IP_ADDR_LEN, "%s.254", ip_base);
455 snprintf(info->ns_ip_range, IP_ADDR_LEN + 3, "%s/31", info->ns_ip);
456 snprintf(info->gw_ip_range, IP_ADDR_LEN + 3, "%s/31", info->gw_ip);
457
458 return 0;
459 }
460
461 /*
396462 * Set up the environment needed from the root network namespace point
397463 * of view. Create virtual ethernet interface (see above) and set our side
398464 * of it up and set address.
401467 */
402468 int
403469 nsntrace_net_init(pid_t ns_pid,
404 const char *device)
470 const char *device,
471 struct nsntrace_if_info *info)
405472 {
406473 int ret = 0;
407474
408 if ((ret = _nsntrace_net_create_veth(GW_IF, NS_IF, ns_pid)) < 0) {
409 return ret;
410 }
411
412 if ((ret = _nsntrace_net_iface_up(GW_IF, GW_IP_RANGE, NULL)) < 0) {
413 return ret;
414 }
415
416 if ((ret = _nsntrace_net_set_nat(GW_IP_RANGE, GW_IF, device, 1)) < 0) {
475 if ((ret = _nsntrace_net_create_veth(info->gw_if, info->ns_if, ns_pid)) < 0) {
476 return ret;
477 }
478
479 if ((ret = _nsntrace_net_iface_up(info->gw_if, info->gw_ip_range, NULL)) < 0) {
480 return ret;
481 }
482
483 if ((ret = _nsntrace_net_set_nat(info->gw_ip_range, info->gw_if, device, 1)) < 0) {
417484 return ret;
418485 }
419486
425492 * Teardown the temporary network trickery we created in init.
426493 */
427494 int
428 nsntrace_net_deinit(const char *device)
495 nsntrace_net_deinit(pid_t ns_pid,
496 const char *device,
497 struct nsntrace_if_info *info)
429498 {
430499 int ret = 0;
431500
432 if ((ret = _nsntrace_net_set_nat(GW_IP_RANGE, GW_IF, device, 0)) < 0) {
433 return ret;
434 }
435
436 if ((ret = _nsntrace_net_iface_delete(NS_IF)) < 0) {
437 return ret;
438 }
439
440 if ((ret = _nsntrace_net_iface_delete(GW_IF)) < 0) {
501 if ((ret = _nsntrace_net_set_nat(info->gw_ip_range, info->gw_if, device, 0)) < 0) {
502 return ret;
503 }
504
505 if ((ret = _nsntrace_net_iface_delete(info->ns_if)) < 0) {
506 return ret;
507 }
508
509 if ((ret = _nsntrace_net_iface_delete(info->gw_if)) < 0) {
441510 return ret;
442511 }
443512
448517 * Set up the namespaced net infrastructure needed.
449518 */
450519 int
451 nsntrace_net_ns_init(int use_public_dns)
520 nsntrace_net_ns_init(int use_public_dns,
521 struct nsntrace_if_info *info)
452522 {
453523 int ret = 0;
454524
455 if ((ret = _nsntrace_net_iface_up(NS_IF, NS_IP_RANGE, GW_IP)) < 0) {
525 if ((ret = _nsntrace_net_iface_up(info->ns_if, info->ns_ip_range, info->gw_ip)) < 0) {
456526 return EXIT_FAILURE;
457527 }
458528
490560
491561 return ch == '1';
492562 }
493
494 const char *
495 nsntrace_net_get_capture_interface()
496 {
497 return NS_IF;
498 }
499
500 const char *
501 nsntrace_net_get_capture_ip()
502 {
503 return NS_IP;
504 }
1515 #ifndef _NSNTRACE_NET_H_
1616 #define _NSNTRACE_NET_H
1717
18 #include <linux/if.h>
19
20 #define IP_ADDR_LEN 16 // 4 sets of 3 numbers each separater by a dot + '\0'
1821 #define NSNTRACE_RUN_DIR "/run/nsntrace"
1922
23 struct nsntrace_if_info {
24 char ns_if[IFNAMSIZ];
25 char gw_if[IFNAMSIZ];
26 char ns_ip[16];
27 char gw_ip[16];
28 char ns_ip_range[19];
29 char gw_ip_range[19];
30 };
31
2032 int nsntrace_net_init(pid_t ns_pid,
21 const char *device);
33 const char *device,
34 struct nsntrace_if_info *info);
2235
23 int nsntrace_net_deinit(const char *device);
36 int nsntrace_net_deinit(pid_t ns_pid,
37 const char *device,
38 struct nsntrace_if_info *info);
2439
25 int nsntrace_net_ns_init(int use_public_dns);
40 int nsntrace_net_ns_init(int use_public_dns,
41 struct nsntrace_if_info *info);
2642
2743 int nsntrace_net_ip_forward_enabled();
2844
29 const char *nsntrace_net_get_capture_ip();
30
31 const char *nsntrace_net_get_capture_interface();
45 int nsntrace_net_get_if_info(pid_t pid,
46 struct nsntrace_if_info *info);
3247
3348 #endif
8181 char * const *args;
8282 };
8383
84 struct nsntrace_common {
85 struct nsntrace_options *options;
86 struct nsntrace_if_info if_info;
87 };
88
8489 #define PUBLIC_DNS 1000
8590 static const char *short_opt = "+o:d:u:f:h";
8691 static struct option long_opt[] = {
145150 return rv;
146151 }
147152
153 static void
154 _nsntrace_remove_rundir(pid_t pid)
155 {
156 char path[PATH_MAX] = { 0, };
157
158 snprintf(path, PATH_MAX, "%s/%d", NSNTRACE_RUN_DIR, getppid());
159 nftw(path, _nsntrace_unlink_cb, 64, FTW_DEPTH | FTW_PHYS);
160 }
161
148162 static void _nsntrace_remove_ns()
149163 {
150164 char netns_path[PATH_MAX];
156170 }
157171
158172 /* clean up run-time files */
159 nftw(NSNTRACE_RUN_DIR, _nsntrace_unlink_cb, 64, FTW_DEPTH | FTW_PHYS);
173 _nsntrace_remove_rundir(getppid());
160174 }
161175
162176 static void
225239 }
226240
227241 static void
228 _nsntrace_start_tracer(struct nsntrace_options *options)
242 _nsntrace_start_tracer(struct nsntrace_options *options,
243 struct nsntrace_if_info *info)
229244 {
230245 int ret;
231 const char *ip;
232 const char *interface;
233
234246 /*
235247 * If outfile is a lone dash ("-"), the user wants us to output to STDOUT,
236248 * in that case we avoid printing anything else to STDOUT and use STDERR
249261 }
250262 }
251263
252 ip = nsntrace_net_get_capture_ip();
253 interface = nsntrace_net_get_capture_interface();
254
255264 fprintf(where, "Starting network trace of '%s' on interface %s.\n"
256265 "Your IP address in this trace is %s.\n"
257266 "Use ctrl-c to end at any time.\n\n",
258 options->args[0], options->device, ip);
259 ret = nsntrace_capture_start(interface, options->filter, fp);
267 options->args[0], options->device, info->ns_ip);
268 ret = nsntrace_capture_start(info->ns_if, options->filter, fp);
260269 if (ret != 0) {
261270 exit(ret);
262271 }
343352 netns_main(void *arg) {
344353 int status;
345354 int ret = EXIT_SUCCESS;
346 struct nsntrace_options *options = (struct nsntrace_options *) arg;
347
348 if (nsntrace_net_ns_init(options->use_public_dns) < 0) {
355 struct nsntrace_common *common = (struct nsntrace_common *) arg;
356 struct nsntrace_options *options = common->options;
357
358 if (nsntrace_net_ns_init(options->use_public_dns, &common->if_info) < 0) {
349359 fprintf(stderr, "failed to setup network environment\n");
350360 return EXIT_FAILURE;
351361 }
352362
353363 _nsntrace_handle_signals(_nsntrace_cleanup_ns_signal_callback);
354 _nsntrace_start_tracer(options);
364 _nsntrace_start_tracer(common->options, &common->if_info);
355365
356366 _nsntrace_set_name();
357367
383393 _nsntrace_cleanup_ns();
384394 exit(ret);
385395 } else { /* child - tracee */
386 _nsntrace_start_tracee(options);
396 _nsntrace_start_tracee(common->options);
387397 }
388398
389399 return ret;
462472 }
463473 }
464474
475 /*
476 * Create the pid based run dir for this nsntrace instance.
477 * This is for storing run-time files and to be able to cleanup
478 * easily without removing other nsntrace instances files.
479 */
480 static void
481 _nsntrace_mkrundir()
482 {
483 char path[PATH_MAX] = { 0, };
484
485 if (mkdir(NSNTRACE_RUN_DIR, 0644) < 0) {
486 if (errno != EEXIST) {
487 perror("mkdir");
488 }
489 }
490 snprintf(path, PATH_MAX, "%s/%d", NSNTRACE_RUN_DIR, getpid());
491 if (mkdir(path, 0644) < 0) {
492 if (errno != EEXIST) {
493 perror("mkdir");
494 }
495 }
496 }
497
465498 int
466499 main(int argc, char **argv)
467500 {
468501 struct nsntrace_options options = { 0 };
469 pid_t pid;
502 struct nsntrace_common common = { &options, };
503 pid_t pid = 0;
470504 int status;
471505 int ret = EXIT_SUCCESS;
472506
489523 exit(EXIT_FAILURE);
490524 }
491525
526 _nsntrace_mkrundir();
527
528 if (nsntrace_net_get_if_info(getpid(), &common.if_info) < 0) {
529 ret = EXIT_FAILURE;
530 goto out;
531 }
532
492533 /* here we create a new process in a new network namespace */
493534 pid = clone(netns_main, child_stack + STACK_SIZE,
494 CLONE_NEWNET | SIGCHLD, &options);
535 CLONE_NEWNET | SIGCHLD, &common);
495536 if (pid < 0) {
496 fprintf(stderr, "clone failed\n");
497 exit(EXIT_FAILURE);
537 perror("clone");
538 ret = EXIT_FAILURE;
539 goto out;
498540 }
499541
500542 _nsntrace_handle_signals(_nsntrace_cleanup);
501543
502 if ((ret = nsntrace_net_init(pid, options.device)) < 0 ||
503 (ret = nsntrace_capture_check_device(options.device))) {
544 if ((ret = nsntrace_net_init(pid, options.device, &common.if_info)) < 0 ||
545 (ret = nsntrace_capture_check_device(options.device))) {
504546 fprintf(stderr, "Failed to setup networking environment\n");
505547 kill(pid, SIGKILL);
506548 goto out;
515557 }
516558
517559 out:
518 nsntrace_net_deinit(options.device);
560 if (pid != 0) {
561 nsntrace_net_deinit(pid, options.device, &common.if_info);
562 }
563 _nsntrace_remove_rundir(getpid());
519564 return ret;
520565 }
00 #!/bin/sh
11
22 # HUP INT QUIT ABRT SEGV TERM
3 signals='1 2 3 6 11 15'
3 set -- 1 2 3 6 11 15
44 timeout=1
55 ip="172.16.42.254"
66 if=nsntrace
99 # sleep to allow for cleanup after signal
1010 sleep 1
1111
12 (sudo iptables -w -t nat -L | grep $ip) && {
12 (sudo iptables -w -t nat -L | grep "$ip") && {
1313 echo "Rules not cleaned up after signal $signal"
1414 exit 1
1515 }
1616
17 (sudo ip link | grep $if) && {
17 (sudo ip link | grep "$if") && {
1818 echo "Link not cleaned up after signal $signal"
1919 exit 1
2020 }
2121
22 ls /run/nsntrace > /dev/null 2>&1 && {
22 ls "/run/nsntrace/$1" > /dev/null 2>&1 && {
2323 echo "run-time files not cleaned up after signal $signal"
2424 exit 1
2525 }
2626
27 rm -rf *.pcap
27 rm -f -- *.pcap
2828 }
2929
3030 start_and_kill() {
31 local signal=$1
31 sig="$1"
3232
3333 sudo ../src/nsntrace ./test_program_dummy.sh &
34 sleep $timeout
34 sleep "$timeout"
3535 pid=$(pidof nsntrace)
36 sudo kill -$signal $pid
36 sudo kill "-$sig" "$pid"
3737
38 check_cleanup
38 check_cleanup "$pid"
39 unset sig
3940 }
4041
41 for signal in $signals; do
42 start_and_kill $signal
42 for signal ; do
43 start_and_kill "$signal"
4344 done
4445
46 sudo ../src/nsntrace ./test_program_dummy_ends.sh
47 check_cleanup "$!"
48
4549 exit 0
0 #!/bin/sh
1
2 num_packets=10
3
4 launch_nsntrace()
5 {
6 id="$1"
7 filter="icmp[icmptype]==icmp-echo"
8
9 sudo ../src/nsntrace -f "$filter" --use-public-dns -o "$id.pcap" ping -c "$num_packets" google.com > /dev/null 2>&1 &
10 sleep 1.0e-3
11
12 unset id filter
13 }
14
15 for i in $(seq 5); do
16 launch_nsntrace "$i"
17 done
18
19 sleep 20
20
21 for i in $(seq 5); do
22 captured=$(tshark -r "$i.pcap" | wc -l)
23 [ "$captured" = "$num_packets" ] || {
24 echo "failed to capture all packets"
25 exit 1
26 }
27 done
28
29 rm -f -- *.pcap
30 exit 0
00 #!/bin/sh
11
22 packets=99
3 cmd="./udp_send 1337 $packets"
43 filter="udp port 1337"
54
65 # start sending packages non-namespaced
76 ./udp_send 1337 -1 &
87
98 # stop the non-namespaced udp_send on exit
10 trap "kill $(pidof udp_send)" EXIT
9 pidof_udp_send=$(pidof udp_send)
10 trap 'kill "$pidof_udp_send"' EXIT
1111
1212 # make sure we only capture the packets from the namespaced version
13 sudo ../src/nsntrace -d lo -f "$filter" $cmd | grep "$packets packets" || {
13 sudo ../src/nsntrace -d lo -f "$filter" ./udp_send 1337 "$packets" | grep "$packets packets" || {
1414 echo "Did not capture $packets packets!"
1515 exit 1
1616 }
2626 check_return_code 1 -o /path/does/not/exist /bin/true
2727 check_return_code 1 -d invalid_device /bin/true
2828
29 rm -rf *.pcap
29 rm -f -- *.pcap
3030
3131 exit ${RET}
00 #!/bin/sh
11
2 while [ 1 ]; do
2 while true ; do
33 sleep 1
44 done
1010 int main(int argc, char **argv)
1111 {
1212 struct sockaddr_in addr = { 0 };
13 int port, num, s;
13 int port, num, s, interval = 0;
1414
15 if (argc != 3) {
16 fprintf(stderr, "usage: udp_send port num_packets\n");
15 if (argc < 3) {
16 fprintf(stderr, "usage: udp_send port num_packets [interval]\n");
1717 exit(EXIT_FAILURE);
1818 }
1919 port = atoi(argv[1]);
2020 num = atoi(argv[2]);
21
22 if (argc > 3) {
23 interval = atoi(argv[3]);
24 }
2125
2226 if ((s = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)) < 0) {
2327 fprintf(stderr, "failed to create socket\n");
3842 fprintf(stderr, "failed to send message\n");
3943 exit(EXIT_FAILURE);
4044 }
45
46 sleep(interval);
4147 }
4248
4349 return 0;