I have a machine with two ethernet cards, two class C networks and roughly 50 IP aliases on various devices.

The two class C networks are distinctly different, i.e., the MSB of the network address is different, and both use a 24 bit netmask.

for the sake of argument:

eth0 is setup on net 1: 192.168.98.0 eth1 is setup on net 2 192.168.99.0

The default route is to our router on eth0, the address is 192.168.98.10

So far, so good.

We bill our customers based on traffic usage and I wrote a libpcap based package to track network usage and calculate aggregates for 5 minute periods and flush this data to disk. I originally wrote this on a Sun machine running solaris 2.5.1.

This worked rather well and I was able to account for all traffic by walking through the ethernet and tcp/ip headers to find the data size.

I rewrote this package to run under 2.0.33 and now I have an odd problem: Packets sent to a particular address all use the same address on the return path.

If, from a different machine on our network at 192.168.98.15, I ping, with record route, to an address on the machine in question, I see:

PING 192.168.98.42 (192.168.98.42): 56 data bytes
64 bytes from 192.168.98.42: icmp_seq=0 ttl=64 time=1.3 ms
RR:     192.168.98.15
        192.168.98.42
        192.168.98.10
        192.168.98.15

And if I traceroute from the machine in question to another machine on our local network, and that address is on net 2, it still goes out over net 1:

traceroute -n 192.168.99.36
traceroute: Warning: Multiple interfaces found; using 192.168.98.10 @ eth0
traceroute to 192.168.99.36 (192.168.99.36), 30 hops max, 40 byte packets
 1  192.168.99.36  0.704 ms  0.606 ms  0.604 ms

So, the problem here is that I cannot track the traffic generated by by a particular website, since the source address of all outbound traffic is not the address of the website, but rather the primary address on eth0 (192.168.98.10)

I have checked /proc/sys/net/ipv4/ip_forwarding and the value is 0, so I am assuming ip_forwarding is turned off.

Is there a way to make this work properly, that is to say, if I request data from a particular address the address used on the sending is correct as well?

What else am I missing or should I be looking for?

TIA, Steve